We Keep Coding

Find specific node in parse tree depending on children it has

To automate a security review of C# code, I want to retrieve all methods from controllers that do have a [HttpPost] attribute, but do not have a [ValidateAntiForgeryToken] attribute. I am using ANTLR to get a ParseTree of the C# code. When I have that, what is the best way to obtain the nodes that have a HttpPost child but not a ValidateAntiForgeryToken child?
I have tried XPath, but it seems ANTLR only supports a subset of XPath. I am considering converting the parse tree to XML and use real XPath on it. Is there an easier way?
I am using the following code to parse the C# file:
import java.io.*;
import java.util.*;
import org.antlr.v4.runtime.*;
import org.antlr.v4.runtime.tree.*;
import org.antlr.v4.runtime.tree.xpath.*;
public class MyParser {
public static void main(String[] args) throws IOException {
CharStream input = CharStreams.fromFileName(args[0]);
Lexer lexer = new CSharpLexer(input);
TokenStream stream = new CommonTokenStream(lexer);
CSharpParser parser = new CSharpParser(stream);
ParseTree tree = parser.compilation_unit();
String xpath = "//class_member_declaration";
Collection<ParseTree> matches = XPath.findAll(tree, xpath, parser);
System.out.println(matches);
}
}
The tree looks like this:

Antlr4 does not support fancy matching on the ParseTree besides a subset of XPath. However, that is also probably the wrong way to solve this problem.
For most use cases, you should walk through the parse tree and collect the information you want. This can be done using a listener or a visitor. For example, the following code collects methods and attributes and prints methods that have certain attributes:
import java.util.*;
public class MyListener extends CSharpParserBaseListener {
String currentClass = null;
String currentMethod = null;
List<String> attributes;
boolean inClassMember = false;
#Override public void enterClass_definition(CSharpParser.Class_definitionContext ctx) {
this.currentClass = ctx.identifier().getText();
}
// Class member declaration. This thing holds both the attributes and the method declaration.
#Override public void enterClass_member_declaration(CSharpParser.Class_member_declarationContext ctx) {
this.attributes = new ArrayList<String>();
this.inClassMember = true;
}
#Override public void enterAttribute(CSharpParser.AttributeContext ctx) {
if (this.inClassMember) {
String attrName = ctx.namespace_or_type_name().identifier().get(0).getText();
this.attributes.add(attrName);
}
}
#Override public void enterMethod_declaration(CSharpParser.Method_declarationContext ctx) {
this.currentMethod = ctx.method_member_name().identifier().get(0).getText();
}
// In the exit we have collected our method name and attributes.
#Override public void exitClass_member_declaration(CSharpParser.Class_member_declarationContext ctx) {
if (this.attributes.contains("HttpPost") && !this.attributes.contains("ValidateAntiForgeryToken")) {
System.out.println(this.currentClass + "." + this.currentMethod);
}
this.attributes = null;
this.currentMethod = null;
this.inClassMember = false;
}
}
To make this more versatile, a better approach would be to convert the parse tree to another tree (i.e. abstract syntax tree) and search that tree for the information you want.

I am considering converting the parse tree to XML and use real XPath on it. Is there an easier way?
There is no need to convert a tree to actual XML in order to use XPath queries. The Apache Commons libary JXPath supports XPath queries on in-memory trees of Java objects.

Extending Eclipse Java Search

I'm developing an eclipse plugin where a user can search a java code given some text query, similar to the usual java search dialog in eclipse.
I'm using the following code to search for a text provided by user
SearchPattern pattern = SearchPattern.createPattern("<search_string>",
IJavaSearchConstants.TYPE, IJavaSearchConstants.PARAMETER_DECLARATION_TYPE_REFERENCE,
SearchPattern.R_EXACT_MATCH);
// step 2: Create search scope
// IJavaSearchScope scope = SearchEngine.createJavaSearchScope(packages);
IJavaSearchScope scope = SearchEngine.createWorkspaceScope();
// step3: define a result collector
SearchRequestor requestor = new SearchRequestor()
{
public void acceptSearchMatch(SearchMatch match)
{
System.out.println(match.getElement());
}
};
// step4: start searching
SearchEngine searchEngine = new SearchEngine();
try {
searchEngine.search(pattern, new SearchParticipant[] { SearchEngine
.getDefaultSearchParticipant() }, scope, requestor,
null);
} catch (CoreException e) {
e.printStackTrace();
}
Also I'm able to pass the query string from Search Dialogue to a class implementing ISearchPage.
public class QuerySearchPage extends DialogPage implements ISearchPage
{
...
public boolean performAction()
{
System.out.println(txtQuery.getText());
search();//search using the SearchEngine
SearchOperation so = new SearchOperation(iFileSet);
IRunnableWithProgress query = so;
try
{
container.getRunnableContext().run(true, true, query);
}
catch (InvocationTargetException | InterruptedException e)
{
e.printStackTrace();
}
return true;
}
}
Finally I got stuck at the point where I need to pass the search result to ISearchResultView. Basically, I have two questions:
Matched results are of type Object. How to pass these results to ISearchResultView which takes IFile as input?
How to get results in the below format?
I have already gone through the following links:
http://help.eclipse.org/mars/index.jsp?topic=%2Forg.eclipse.platform.doc.isv%2Fguide%2Fsearch_page.htm
http://agile.csc.ncsu.edu/SEMaterials/tutorials/plugin_dev/
http://grepcode.com/file/repository.grepcode.com/java/eclipse.org/3.5.2/org.eclipse/search/3.5.1/org/eclipse/search/ui/ISearchResult.java?av=f
http://codeandme.blogspot.de/2015/07/a-custom-search-provider.html
http://scg.unibe.ch/archive/projects/Bals10b-EclipsePlugins.pdf
How can I develop Eclipse search Plugin?
http://help.eclipse.org/juno/index.jsp?topic=%2Forg.eclipse.platform.doc.isv%2Freference%2Fapi%2Forg%2Feclipse%2Fsearch%2Fui%2FISearchResult.html
https://wiki.eclipse.org/FAQ_How_do_I_implement_a_search_operation%3F
Any help is highly welcomed.

Usually, you would implement a ISearchResultPage that is capable of displaying the search result. In its createControl() method you need to create a viewer that knows how to present the matches.
A commonly used abstract implementation of ISearchResultPage is AbstractTextSearchViewPage. This class uses a TableViewer or a TreeViewer to present the machtes, depending on whether they are hierarchical or not. In case you use the latter, implement its configureTreeViewer() and/or configureTableViewer() methods so that the viewers are equipped with label providers and content providers that know the specific type that represents a match, i.e. what you referred to as the 'Matched results of type Object'.
The AbstractTextSearchViewPage constructor must be told which kinds of layouts it supports: FLAG_LAYOUT_FLAT and/or FLAG_LAYOUT_TREE. The actual representation can be changed with setLayout().
To start with you could restrict the search view page to a flat layout and implement its configureTableViewer() like this:
viewer.setLabelProvider( new MyLabelProvider() );
viewer.setContentProvider( new MyContentProvider() );
The input for the content provider is your ISearchResult implementation. Hence the MyContentProvider could obtain the elements to be shown from the search result.
#Override
public void inputChanged( Viewer viewer, Object oldInput, Object newInput ) {
searchResult = ( MySearchResult )newInput;
}
#Override
public Object[] getElements( Object inputElement ) {
return searchResult.getElements();
}

Android speed issue with deserialization using SimpleXML

Raised a bounty as the only answer doesn't provide a good implementation for Android. Is there a speedier implementation compatible with Android? Or is SimpleXML the best performance I'll get?
I'm fairly novice to Java and Android development so don't know the proper procedure for deserializing an xml string to an object. I found a method that works in:
public static Object deserializeXMLToObject(String xmlFile,Object objClass) throws Exception
{
try
{
InputStream stream = new ByteArrayInputStream(xmlFile.getBytes("UTF-8"));
Serializer serializer = new Persister();
objClass = serializer.read(objClass, stream);
return objClass;
}
catch (Exception e)
{
return e;
}
}
Where xmlFile is the (misnamed) xml string, and objClass is an empty class of the class I want to deserialize to. This is generally a list of other objects.
Example class:
#Root(name="DepartmentList")
public class DepartmentList {
#ElementList(entry="Department", inline=true)
public List<Department> DepartmentList =new ArrayList<Department>();
public boolean FinishedPopulating = false;
}
Department class:
public class Department {
#Element(name="DeptID")
private String _DeptID ="";
public String DeptID()
{
return _DeptID;
}
public void DeptID(String Value)
{
_DeptID = Value;
}
#Element(name="DeptDescription")
private String _DeptDescription ="";
public String DeptDescription()
{
return _DeptDescription;
}
public void DeptDescription(String Value)
{
_DeptDescription = Value;
}
}
Example XML:
<DepartmentList>
<Department>
<DeptID>525</DeptID>
<DeptDescription>Dept 1</DeptDescription>
</Department>
<Department>
<DeptID>382</DeptID>
<DeptDescription>Dept 2</DeptDescription>
</Department>
</DepartmentList>
This has been working fine throughout the app, but I have come to a point where it needs to deserialise >300 objects in the list. This only takes approximately 5 secs, or close to a minute when debugging, but users are not happy with that performance and time wasted when debugging isn't desirable. Is there any way to speed this up? Or is there another way I should be doing this? Preferably only by changing the deserializeXMLToObject method.

I am sure someone will point to a better library that's out there, but according to one detailed answer, they are all slow on Android.
So here is my quick hack (yes I know its not very maintainable and is brittle to the XML not being formed exactly as specified) and some results:
private void doTest()
{
Thread t = new Thread()
{
public void run()
{
runOne(2000);
runOne(300);
runOne(20000);
}
private void runOne(int num)
{
String start = "<DepartmentList>";
String mid1 = "<Department>\n" +
"<DeptID>";
String mid2 = "</DeptID>\n" +
"<DeptDescription>Dept ";
String mid3 = "</DeptDescription></Department>";
String fin = "</DepartmentList>";
StringBuffer sb = new StringBuffer();
sb.append(start);
for (int i=0; i< num; i++)
{
sb.append(mid1);
sb.append(""+i);
sb.append(mid2);
sb.append(""+i);
sb.append(mid3);
}
sb.append(fin);
Pattern p = Pattern.compile(
"<Department\\s*>\\s*<DeptID\\s*>([^<]*)</DeptID>\\s*<DeptDescription\\s*>([^<]*)</DeptDescription>\\s*</Department>");
long startN = System.currentTimeMillis();
DepartmentList d = new DepartmentList();
List<Department> departments = d.DepartmentList;
Matcher m = p.matcher(sb);
while (m.find())
{
Department department = new Department();
department.DeptID(m.group(1));
department.DeptDescription(m.group(2));
departments.add(department);
}
long endN = System.currentTimeMillis();
Log.d("Departments", "parsed: " + departments.size() + " in " + (endN-startN) + " millis");
Log.d("Departments", "lastone: " + departments.get(departments.size() -1)._DeptID + " desc: " + departments.get(departments.size() -1)._DeptDescription);
}
};
t.start();
}
public class DepartmentList {
public List<Department> DepartmentList =new ArrayList<Department>();
public boolean FinishedPopulating = false;
}
public class Department {
private String _DeptID ="";
public String DeptID()
{
return _DeptID;
}
public void DeptID(String Value)
{
_DeptID = Value;
}
private String _DeptDescription ="";
public String DeptDescription()
{
return _DeptDescription;
}
public void DeptDescription(String Value)
{
_DeptDescription = Value;
}
}
I pasted this into an Android project and called it from the onCreate() method. Here is the results:
Platform num=300 num=2000 num=20000
=================================================
Nexus 7 5 38 355
Galaxy Y 29 430 1173
HTC Desire HD 19 189 539
Galaxy Nexus 14 75 379
All times are in milliseconds.

For my research this is the best way to optimize:
"Simple will dynamically build your object graph, this means that it will need to load classes that have not already been loaded, and build a schema for each class based on its annotations using reflection. So first use will always be by far the most expensive. Repeated use of the same persister instance will be many times faster. So try to avoid multiple persister instances, just use one if possible."
So refactoring your code to use the same Persister should improve you performance.
This and other tips I got from this question. In that case, this refactoring improved the performance, as stated by the author (from 15s to 3-5s).
Hope it helps

You can eliminate the intermediate (de)serialization steps by serializing directly to XML and deserializing directly from XML, using e.g. JAXB or XStream.
You may also be able to speed things up via multithreading. I'll assume that all of the XML strings you want to deserialize are in a ConcurrentLinkedQueue; alternatively, you can synchronize access to whatever non-threadsafe collection you're using. Use something like a ThreadPoolExecutor to minimize thread creation overhead.
public class DeserializeXML implements Runnable {
private final String xml;
private final ConcurrentLinkedQueue<Object> deserializedObjects;
public DeserializeXML(String xml, ConcurrentLinkedQueue deserializedObjects) {
this.xml = xml;
this.deserializedObjects = deserializedObjects;
}
public void run() {
deserializedObjects.offer(deserializeXMLToObject(xml, Object.class));
}
}
// ***
ConcurrentLinkedQueue<String> serializedObjects;
ConcurrentLinkedQueue<Object> deserializedObjects;
ThreadPoolExecutor executor = new ThreadPoolExecutor;
while(!serializedObjects.isEmpty()) {
executor.execute(new DeserializeXML(serializedObjects.poll(), deserializedObjects));
}

Got a similar issue with a SOAP Web Service times ago.
In the end I've changed the format of the XML, transforming the nodes in attributes.
example:
<node>
<attr1>value1</attr1>
<attr2>value2</attr2>
<attr3>value3</attr3>
<attr4>value4</attr4>
</node>
was transformed in
<node attr1='value1'attr2='value2' attr3='value3' attr4='value4' />
Maybe not the best solution theorically wise, but performance improvement was really good. Ofc if your XML is a bit more complex (repeteable nodes at various levels or multi level lists) things may be a bit complex.

Use a server proxy and transform your XML to JSON

File Output between multiple runtimes?

I am working on this project where the user enters some data which is written to an XML file.This part is working fine.
Now when the user runs the program he should be able to append to that file. Instead it creates a new file with just one entry!
A fileoutput stream is also not the solution.
Here is the code for serializing to XML
String medicine=medicfield.getText();
String doctor=dnamefield.getText();
int duration=Integer.parseInt(dodfield.getText());
int amount=Integer.parseInt(cyclefield.getText());
int inter=Integer.parseInt(intval.getText());
PrescripManager pm=new PrescripManager();
pm.setDcycle(amount);
pm.setDosage(duration);
pm.setInterval(inter);
pm.setmedName(medicine);
pm.setdocName(doctor);
try{
FileOutputStream file = new FileOutputStream("file.xml");
JAXBContext jaxbContext = JAXBContext.newInstance(PrescripManager.class);
Marshaller jaxbMarshaller = jaxbContext.createMarshaller();
// output pretty printed
jaxbMarshaller.marshal(pm, file);
}
catch(Exception ex)
{
erlbl.setText(ex.getMessage());
}
And the Class::
#XmlRootElement
public class PrescripManager {
private String medname,docname;
private int interval,dcycle,dosage;
private Date dt;
public String getmedName() {
return medname;
}
public void setmedName(String medname) {
this.medname = medname;
}
public String getdocName() {
return docname;
}
public void setdocName(String docname) {
this.docname = docname;
}
public int getInterval() {
return interval;
}
public void setInterval(int interval) {
this.interval = interval;
}
public int getDcycle() {
return dcycle;
}
public void setDcycle(int dcycle) {
this.dcycle = dcycle;
}
public int getDosage() {
return dosage;
}
public void setDosage(int dosage) {
this.dosage = dosage;
}
}

First of all, you are writing an XML file. You can not just append to an XML file, because that would mean you are writing after then closing top level tag, resulting in invalid XML file.
You have at least three choices:
read old file in, add to the actual data, then write entire XML-file back.
write multiple files, each a valid XML file, with sequence number or timestamp in file name.
do not use XML, use a format which can be appended to
As a side note, if you want to append to file, you can open it in append mode. That will make every write to it append (at least on Unix, when file is opened in append mode, and I presume it works the same in Windows).
How to open file in append mode in Java: http://docs.oracle.com/javase/6/docs/api/java/io/FileOutputStream.html#FileOutputStream(java.io.File, boolean)

You can use the "FileWriter" class which allows you to write at the end of a file. See
http://docs.oracle.com/javase/6/docs/api/java/io/FileWriter.html
This link should help you:
http://www.coderanch.com/t/276346//java/do-open-file-append-mode
I think it should work if you use
FilterWriter file = new FileWriter( "file.xml" , true );
instead of
FileOutputStream file = new FileOutputStream("file.xml");

You are using JAXB to process XML files, so it's better that you change your XML file format and Java class to support this.
You can add a new class as collections of PrescripManager class instances. Something like PrescripManagerList.
public class PrescripManagerList{
#XmlElementWrapper(name = "prescripManagers")
#XmlElement(name = "prescripManager")
private ArrayList<PrescripManager> prescripManagers;
}
When you running your code, try to read an existing XML file using JAXB unmarshaller to get a PrescripManagerList object, then add a new PrescripManager object to the ArrayList, then write the updated PrescripManagerList object to file using JAXB marshaller.

Why won't .NET deserialize my primitive array from a web service?

Help! I have an Axis web service that is being consumed by a C# application. Everything works great, except that arrays of long values always come across as [0,0,0,0] - the right length, but the values aren't deserialized. I have tried with other primitives (ints, doubles) and the same thing happens. What do I do? I don't want to change the semantics of my service.

Here's what I ended up with. I have never found another solution out there for this, so if you have something better, by all means, contribute.
First, the long array definition in the wsdl:types area:
<xsd:complexType name="ArrayOf_xsd_long">
<xsd:complexContent mixed="false">
<xsd:restriction base="soapenc:Array">
<xsd:attribute wsdl:arrayType="soapenc:long[]" ref="soapenc:arrayType" />
</xsd:restriction>
</xsd:complexContent>
</xsd:complexType>
Next, we create a SoapExtensionAttribute that will perform the fix. It seems that the problem was that .NET wasn't following the multiref id to the element containing the double value. So, we process the array item, go find the value, and then insert it the value into the element:
[AttributeUsage(AttributeTargets.Method)]
public class LongArrayHelperAttribute : SoapExtensionAttribute
{
private int priority = 0;
public override Type ExtensionType
{
get { return typeof (LongArrayHelper); }
}
public override int Priority
{
get { return priority; }
set { priority = value; }
}
}
public class LongArrayHelper : SoapExtension
{
private static ILog log = LogManager.GetLogger(typeof (LongArrayHelper));
public override object GetInitializer(LogicalMethodInfo methodInfo, SoapExtensionAttribute attribute)
{
return null;
}
public override object GetInitializer(Type serviceType)
{
return null;
}
public override void Initialize(object initializer)
{
}
private Stream originalStream;
private Stream newStream;
public override void ProcessMessage(SoapMessage m)
{
switch (m.Stage)
{
case SoapMessageStage.AfterSerialize:
newStream.Position = 0; //need to reset stream
CopyStream(newStream, originalStream);
break;
case SoapMessageStage.BeforeDeserialize:
XmlWriterSettings settings = new XmlWriterSettings();
settings.Indent = false;
settings.NewLineOnAttributes = false;
settings.NewLineHandling = NewLineHandling.None;
settings.NewLineChars = "";
XmlWriter writer = XmlWriter.Create(newStream, settings);
XmlDocument xmlDocument = new XmlDocument();
xmlDocument.Load(originalStream);
List<XmlElement> longArrayItems = new List<XmlElement>();
Dictionary<string, XmlElement> multiRefs = new Dictionary<string, XmlElement>();
FindImportantNodes(xmlDocument.DocumentElement, longArrayItems, multiRefs);
FixLongArrays(longArrayItems, multiRefs);
xmlDocument.Save(writer);
newStream.Position = 0;
break;
}
}
private static void FindImportantNodes(XmlElement element, List<XmlElement> longArrayItems,
Dictionary<string, XmlElement> multiRefs)
{
string val = element.GetAttribute("soapenc:arrayType");
if (val != null && val.Contains(":long["))
{
longArrayItems.Add(element);
}
if (element.Name == "multiRef")
{
multiRefs[element.GetAttribute("id")] = element;
}
foreach (XmlNode node in element.ChildNodes)
{
XmlElement child = node as XmlElement;
if (child != null)
{
FindImportantNodes(child, longArrayItems, multiRefs);
}
}
}
private static void FixLongArrays(List<XmlElement> longArrayItems, Dictionary<string, XmlElement> multiRefs)
{
foreach (XmlElement element in longArrayItems)
{
foreach (XmlNode node in element.ChildNodes)
{
XmlElement child = node as XmlElement;
if (child != null)
{
string href = child.GetAttribute("href");
if (href == null || href.Length == 0)
{
continue;
}
if (href.StartsWith("#"))
{
href = href.Remove(0, 1);
}
XmlElement multiRef = multiRefs[href];
if (multiRef == null)
{
continue;
}
child.RemoveAttribute("href");
child.InnerXml = multiRef.InnerXml;
if (log.IsDebugEnabled)
{
log.Debug("Replaced multiRef id '" + href + "' with value: " + multiRef.InnerXml);
}
}
}
}
}
public override Stream ChainStream(Stream s)
{
originalStream = s;
newStream = new MemoryStream();
return newStream;
}
private static void CopyStream(Stream from, Stream to)
{
TextReader reader = new StreamReader(from);
TextWriter writer = new StreamWriter(to);
writer.WriteLine(reader.ReadToEnd());
writer.Flush();
}
}
Finally, we tag all methods in the Reference.cs file that will be deserializing a long array with our attribute:
[SoapRpcMethod("", RequestNamespace="http://some.service.provider",
ResponseNamespace="http://some.service.provider")]
[return : SoapElement("getFooReturn")]
[LongArrayHelper]
public Foo getFoo()
{
object[] results = Invoke("getFoo", new object[0]);
return ((Foo) (results[0]));
}
This fix is long-specific, but it could probably be generalized to handle any primitive type having this problem.

Here's a more or less copy-pasted version of a blog post I wrote on the subject.
Executive summary: You can either change the way .NET deserializes the result set (see Chris's solution above), or you can reconfigure Axis to serialize its results in a way that's compatible with the .NET SOAP implementation.
If you go the latter route, here's how:
... the generated
classes look and appear to function
normally, but if you'll look at the
deserialized array on the client
(.NET/WCF) side you'll find that the
array has been deserialized
incorrectly, and all values in the
array are 0. You'll have to manually
look at the SOAP response returned by
Axis to figure out what's wrong;
here's a sample response (again,
edited for clarity):
<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv=http://schemas.xmlsoap.org/soap/envelope/>
<soapenv:Body>
<doSomethingResponse>
<doSomethingReturn>
<doSomethingReturn href="#id0"/>
<doSomethingReturn href="#id1"/>
<doSomethingReturn href="#id2"/>
<doSomethingReturn href="#id3"/>
<doSomethingReturn href="#id4"/>
</doSomethingReturn>
</doSomethingResponse>
<multiRef id="id4">5</multiRef>
<multiRef id="id3">4</multiRef>
<multiRef id="id2">3</multiRef>
<multiRef id="id1">2</multiRef>
<multiRef id="id0">1</multiRef>
</soapenv:Body>
</soapenv:Envelope>
You'll notice that Axis does not
generate values directly in the
returned element, but instead
references external elements for
values. This might make sense when
there are many references to
relatively few discrete values, but
whatever the case this is not properly
handled by the WCF basicHttpBinding
provider (and reportedly by gSOAP and
classic .NET web references as well).
It took me a while to find a solution:
edit your Axis deployment's
server-config.wsdd file and find the
following parameter:
<parameter name="sendMultiRefs" value="true"/>
Change it to false,
then redeploy via the command line,
which looks (under Windows) something
like this:
java -cp %AXISCLASSPATH% org.apache.axis.client.AdminClient server-config.wsdl
The web service's
response should now be deserializable
by your .NET client.

Found this link that may offer a better alternative: http://www.tomergabel.com/GettingWCFAndApacheAxisToBeFriendly.aspx