We Keep Coding

integrate call to a jar file | cut | awk and a java program into one unified process

I'm currently executing a rather complicated data pre-processing operation, this is:
cat large_file.txt \ | ./reverb -q | cut --fields=16,17,18 | awk -F\\t -vq="'" 'function quote(token) { gsub(q, "\\"q, token); return q token q } { print quote($2) "(" quote($3) ", " quote($1) ")." }' >> output.txt
As you can see, this is rather convoluted, first cat, then to that ./reverb, then to cut, and finally to awk.
Next I want to pass the output to a java program, i.e.:
public static void main(String[] args) throws IOException
{
Ontology ontology = new Ontology();
BufferedReader br = new BufferedReader(new FileReader("/home/matthias/Workbench/SUTD/2_January/Prolog/horn_data_test.pl"));
Pattern p = Pattern.compile("'(.*?)'\\('(.*?)','(.*?)'\\)\\.");
String line;
while ((line = br.readLine()) != null)
{
Matcher m = p.matcher(line);
if( m.matches() )
{
String verb = m.group(1);
String object = m.group(2);
String subject = m.group(3);
ontology.addSentence( new Sentence( verb, object, subject ) );
}
}
for( String joint: ontology.getJoints() )
{
for( Integer subind: ontology.getSubjectIndices( joint ) )
{
Sentence xaS = ontology.getSentence( subind );
for( Integer obind: ontology.getObjectIndices( joint ) )
{
Sentence yOb = ontology.getSentence( obind );
Sentence s = new Sentence( xaS.getVerb(),
xaS.getObject(),
yOb.getSubject() );
System.out.println( s );
}
}
}
}
What would be the best way to synthesize this process into one coherent operation? Ideally I'd like to just specify the input file and the output file and run it once. As it stands the entire process is quite discombobulated.
Maybe I can just put all these calls into a bash script? Is that feasible?
The input initially contains English language sentences, one per line, this is:
Oranges are delicious and contain vitamin c.
Brilliant scientists learned that we can prevent scurvy by imbibing vitamin c.
Colorless green ideas sleep furiously.
...
The pre-processing makes it look like this:
'contain'('vitamin c','oranges').
'prevent'('scurvy','vitamin c').
'sleep'('furiously','ideas').
...
The java program is for learning "rules" by inference, so if the processed data yields 'contain'('vitamin c','oranges'). & 'prevent'('scurvy','vitamin c'). then the java code will emit 'prevent'('scurvy','oranges').

I looked at the source code for reverb and I think it's very easy to adapt it to produce the output you want. If you look at the reverb class CommandLineReverb.java, it has the following two methods:
private void extractFromSentReader(ChunkedSentenceReader reader)
throws ExtractorException {
long start;
ChunkedSentenceIterator sentenceIt = reader.iterator();
while (sentenceIt.hasNext()) {
// get the next chunked sentence
ChunkedSentence sent = sentenceIt.next();
chunkTime += sentenceIt.getLastComputeTime();
numSents++;
// make the extractions
start = System.nanoTime();
Iterable<ChunkedBinaryExtraction> extractions = extractor
.extract(sent);
extractTime += System.nanoTime() - start;
for (ChunkedBinaryExtraction extr : extractions) {
numExtrs++;
// run the confidence function
start = System.nanoTime();
double conf = getConf(extr);
confTime += System.nanoTime() - start;
NormalizedBinaryExtraction extrNorm = normalizer
.normalize(extr);
printExtr(extrNorm, conf);
}
if (numSents % messageEvery == 0)
summary();
}
}
private void printExtr(NormalizedBinaryExtraction extr, double conf) {
String arg1 = extr.getArgument1().toString();
String rel = extr.getRelation().toString();
String arg2 = extr.getArgument2().toString();
ChunkedSentence sent = extr.getSentence();
String toks = sent.getTokensAsString();
String pos = sent.getPosTagsAsString();
String chunks = sent.getChunkTagsAsString();
String arg1Norm = extr.getArgument1Norm().toString();
String relNorm = extr.getRelationNorm().toString();
String arg2Norm = extr.getArgument2Norm().toString();
Range arg1Range = extr.getArgument1().getRange();
Range relRange = extr.getRelation().getRange();
Range arg2Range = extr.getArgument2().getRange();
String a1s = String.valueOf(arg1Range.getStart());
String a1e = String.valueOf(arg1Range.getEnd());
String rs = String.valueOf(relRange.getStart());
String re = String.valueOf(relRange.getEnd());
String a2s = String.valueOf(arg2Range.getStart());
String a2e = String.valueOf(arg2Range.getEnd());
String row = Joiner.on("\t").join(
new String[] { currentFile, String.valueOf(numSents), arg1,
rel, arg2, a1s, a1e, rs, re, a2s, a2e,
String.valueOf(conf), toks, pos, chunks, arg1Norm,
relNorm, arg2Norm });
System.out.println(row);
}
The first method is called per sentence and does the extraction. Then it calls the second method to print the tab separated values to the output stream. I guess all you have to do is implement your own version of the second method 'printExtr()'.

Javassist - How to add line number to method

I am a newbie to java bytecode and javassist. I created a new class file with using javassist. Although I added fields and methods, I couldn't achieve to add line number to method. Result of my research, I understand that I need to add linenumberattribute to codeattribute of method info. Moreover, linenumberattribute consists of linenumbertable. I don't know how can I create a new linenumberattribute with javassist.

I am writing a compiler that produces JVM code. I need line numbers in the output. I do it this way.
I build up a list of objects similar to this:
public class MyLineNum {
public final short pc;
public final short lineNum;
}
Then I add the line number table:
final ClassFile classFile = ...;
final ConstPool constPool = classFile.getConstPool();
final MethodInfo minfo = new MethodInfo( ... );
final Bytecode code = new Bytecode( constPool );
... code that writes to 'code'
final List<MyLineNum> lineNums = new ArrayList<>();
... code that adds to 'lineNums'
final CodeAttribute codeAttr = code.toCodeAttribute();
if ( !lineNums.isEmpty() ) {
// JVM spec describes method line number table thus:
// u2 line_number_table_length;
// { u2 start_pc;
// u2 line_number;
// } line_number_table[ line_number_table_length ];
final int numLineNums = lineNums.size();
final byte[] lineNumTbl = new byte[ ( numLineNums * 4 ) + 2 ];
// Write line_number_table_length.
int byteIx = 0;
ByteArray.write16bit( numLineNums, lineNumTbl, byteIx );
byteIx += 2;
// Write the individual line number entries.
for ( final MyLineNum ln : lineNums) {
// start_pc
ByteArray.write16bit( ln.pc, lineNumTbl, byteIx );
byteIx += 2;
// line_number
ByteArray.write16bit( ln.lineNum, lineNumTbl, byteIx );
byteIx += 2;
}
// Add the line number table to the CodeAttribute.
#SuppressWarnings("unchecked")
final List<AttributeInfo> codeAttrAttrs = codeAttr.getAttributes();
codeAttrAttrs.removeIf( ( ai ) -> ai.getName().equals( "LineNumberTable" ) ); // remove if already present
codeAttrAttrs.add( new AttributeInfo( constPool, "LineNumberTable", lineNumTbl ) );
}
// Attach the CodeAttribute to the MethodInfo.
minfo.setCodeAttribute( codeAttr );
// Attach the MethodInfo to the ClassFile.
try {
classFile.addMethod( minfo );
}
catch ( final DuplicateMemberException ex ) {
throw new AssertionError( "Caught " + ex, ex );
}

Is it possible to resolve mathematical function dynamically

Is it possible, to resolve mathematical functions dynamically, e.g by the use of a given API?
Given there is a function
a = b + c
is there a way to write something comparable to:
Function func = new Function("a=b+c");
Result result = func.resolve("a=1", "b=?", "c=a");
System.out.println(result.getValue());
Ideally, resolve(Object... args) should accept further classes of type Function.
EDIT 1: The API should be includable into a Java EE environment such as jBossAS.
EDIT 2: Actually I want to solve equations with one unknown variable, but I want to avoid hard coded functions. Thus, for the function
a+b=c
I don't want to write the functions hard coded
getA(int b, int c){...}
getB(int a, int c){...}
getC(int a, int b){...}
Instead, as #Berylium says, I want to evaluate the expression dynamically.
EDIT 3: I'm trying symja right now and I think I'm getting closer, but I have troubles with the syntax.
try {
F.initSymbols(null);
EvalUtilities util = new EvalUtilities();
StringBufferWriter buf = new StringBufferWriter();
String input = "....";
IExpr result = util.evaluate(input);
OutputFormFactory.get().convert(buf, result);
String output = buf.toString();
System.out.println("Evaluation for " + input + " is " + output);
} catch (final Exception e) {
Can you help me with the input syntax?
EDIT 4: Got it :-) For input
String input = "Solve[{a==10,c==20,a+b==c},{a,b,c}]";
the output is
Evaluation for Solve[{a==10,c==20,a+b==c},{a,b,c}] is {{a->10,c->20,b->10}}

Any embeddable Java scripting engine will do:
For example, using BeanShell:
final Interpreter interp = new Interpreter();
try {
interp.eval("System.out.println(\"Hello, world\");");
final String s = "Hello, world (2)";
interp.set("test", s);
interp.eval("System.out.println(test);");
System.out.println("3+4=" + interp.eval("3+4"));
interp.set("a", 4);
interp.set("b", 5);
System.out.println("a + b = " + interp.eval("a + b"));
final String script1 =
"public int f(int a) { return a * a; }; System.out.println(f(4));";
interp.eval(script1);
final String script2 =
"public int f(int a) { return a * a; }; f(4)";
System.out.println(interp.eval(script2));
final String script3 =
"import java.util.Date; Date date = new Date(); date";
System.out.println(interp.eval(script3));
final String script4 =
"class X { public int a; } X x = new X(); x.a = 5; x.a";
System.out.println(interp.eval(script4));
} catch (EvalError e) {
e.printStackTrace();
}
One advantage is that BeanShell uses Java syntax which is quite close to Java. So there is no need to learn/use another language/syntax.

After adding symja JAR to the build path, the following code prints the output below:
Code:
public static void main(String[] args) {
try {
F.initSymbols(null);
EvalUtilities util = new EvalUtilities();
StringBufferWriter buf = new StringBufferWriter();
String input = "Solve[{a==10,c==20,a+b==c},{a,b,c}]";
IExpr result = util.evaluate(input);
OutputFormFactory.get().convert(buf, result);
String output = buf.toString();
System.out.println("Evaluation for " + input + " is " + output);
} catch (final Exception e) {
e.printStackTrace();
} finally {
// Call terminate() only one time at the end of the program
ComputerThreads.terminate();
}
}
Output:
Evaluation for Solve[{a==10,c==20,a+b==c},{a,b,c}] is {{a->10,c->20,b->10}}

Querying Prolog variables with JPL

I want to make a query to use Prolog in java through JPL, I read the documentation (http://www.swi-prolog.org/packages/jpl/java_api/getting_started.html)
The prolog predicates are these:
child_of(joe, ralf).
child_of(mary, joe).
child_of(steve, joe).
child_of(steve, ralf).
descendent_of(X, Y) :-
child_of(X, Y).
descendent_of(X, Y) :-
child_of(Z, Y),
descendent_of(X, Z).
My code looks like this
Variable X = new Variable();
Query q4 =
new Query(
"descendent_of",
new Term[] {X,new Atom("joe")}
);
java.util.Hashtable solution;
while ( q4.hasMoreSolutions() ){
solution = q4.nextSolution();
System.out.println( "X = " + solution.get(X));
}
According to my prolog predicates, my java code should retrieve 'mary' and 'steve', but I get this:
X = null
X = null
What I'm doing wrong? thanks in advance
EDIT: this is my entire testing
Query q1 =
new Query(
"consult",
new Term[] {new Atom("C:\\Users\\cardozo\\Documents\\fer\\info2\\lore\\test.pl")}
);
return q1;
System.out.println( "consult " + (q.query() ? "succeeded" : "failed"));
Query q2 =
new Query(
"child_of",
new Term[] {new Atom("joe"),new Atom("X")}
);
Boolean resp= q2.query();
System.out.println("child_of(joe,X) is " + resp.toString()
);
Query q3 =
new Query(
"descendent_of",
new Term[] {new Atom("steve"),new Atom("ralf")}
);
System.out.println(
"descendent_of(joe,ralf) is " +
( q3.query() ? "provable" : "not provable" )
);
Variable X = new Variable();
Query q4 =
new Query(
"descendent_of",
new Term[] {X,new Atom("joe")}
);
java.util.Hashtable solution;
q4.query();
while ( q4.hasMoreSolutions() ){
solution = q4.nextSolution();
System.out.println( "X = " + solution.get("X"));
}
And this is what I get in my java console as result
run:
% C:\Users\cardozo\Documents\fer\info2\lore\test.pl compiled 0.00 sec, 8 clauses
consult succeeded
child_of(joe,X) is false
descendent_of(joe,ralf) is provable
X = null
X = null
BUILD SUCCESSFUL (total time: 0 seconds)

I found the solution, I have to use the class Compound (included in jpl) like this
Query q4 = new Query(new Compound("descendent_of", new Term[] { new Variable("X"), new Atom("joe")}));
while ( q4.hasMoreSolutions() ){
solution = q4.nextSolution();
System.out.println( "X = " + solution.get("X"));
}
And I get the solution
X = mary
X = steve

I would try to get variable by name:
solution.get("X")
edit
with a literal query like
Query q4 = new Query("descendent_of(X,joe)")

Java: Get last element after split

I am using the String split method and I want to have the last element.
The size of the Array can change.
Example:
String one = "Düsseldorf - Zentrum - Günnewig Uebachs"
String two = "Düsseldorf - Madison"
I want to split the above Strings and get the last item:
lastone = one.split("-")[here the last item] // <- how?
lasttwo = two.split("-")[here the last item] // <- how?
I don't know the sizes of the arrays at runtime :(

You could use lastIndexOf() method on String
String last = string.substring(string.lastIndexOf('-') + 1);

Save the array in a local variable and use the array's length field to find its length. Subtract one to account for it being 0-based:
String[] bits = one.split("-");
String lastOne = bits[bits.length-1];
Caveat emptor: if the original string is composed of only the separator, for example "-" or "---", bits.length will be 0 and this will throw an ArrayIndexOutOfBoundsException. Example: https://onlinegdb.com/r1M-TJkZ8

You can use the StringUtils class in Apache Commons:
StringUtils.substringAfterLast(one, "-");

using a simple, yet generic, helper method like this:
public static <T> T last(T[] array) {
return array[array.length - 1];
}
you can rewrite:
lastone = one.split("-")[..];
as:
lastone = last(one.split("-"));

String str = "www.anywebsite.com/folder/subfolder/directory";
int index = str.lastIndexOf('/');
String lastString = str.substring(index +1);
Now lastString has the value "directory"

Gathered all possible ways together!!
By using lastIndexOf() & substring() methods of Java.lang.String
// int firstIndex = str.indexOf( separator );
int lastIndexOf = str.lastIndexOf( separator );
String begningPortion = str.substring( 0, lastIndexOf );
String endPortion = str.substring( lastIndexOf + 1 );
System.out.println("First Portion : " + begningPortion );
System.out.println("Last Portion : " + endPortion );
split()Java SE 1.4. Splits the provided text into an array.
String[] split = str.split( Pattern.quote( separator ) );
String lastOne = split[split.length-1];
System.out.println("Split Array : "+ lastOne);
How to split String before first comma?
Java 8 sequential ordered stream from an array.
String firstItem = Stream.of( split )
.reduce( (first,last) -> first ).get();
String lastItem = Stream.of( split )
.reduce( (first,last) -> last ).get();
System.out.println("First Item : "+ firstItem);
System.out.println("Last Item : "+ lastItem);
Apache Commons Langjar « org.apache.commons.lang3.StringUtils
String afterLast = StringUtils.substringAfterLast(str, separator);
System.out.println("StringUtils AfterLast : "+ afterLast);
String beforeLast = StringUtils.substringBeforeLast(str, separator);
System.out.println("StringUtils BeforeLast : "+ beforeLast);
String open = "[", close = "]";
String[] groups = StringUtils.substringsBetween("Yash[777]Sam[7]", open, close);
System.out.println("String that is nested in between two Strings "+ groups[0]);
Guava: Google Core Libraries for Java. « com.google.common.base.Splitter
Splitter splitter = Splitter.on( separator ).trimResults();
Iterable<String> iterable = splitter.split( str );
String first_Iterable = Iterables.getFirst(iterable, "");
String last_Iterable = Iterables.getLast( iterable );
System.out.println(" Guava FirstElement : "+ first_Iterable);
System.out.println(" Guava LastElement : "+ last_Iterable);
Scripting for the Java Platform « Run Javascript on the JVM with Rhino/Nashorn
Rhino « Rhino is an open-source implementation of JavaScript written entirely in Java. It is typically embedded into Java applications to provide scripting to end users. It is embedded in J2SE 6 as the default Java scripting engine.
Nashorn is a JavaScript engine developed in the Java programming language by Oracle. It is based on the Da Vinci Machine and has been released with Java 8.
Java Scripting Programmer's Guide
public class SplitOperations {
public static void main(String[] args) {
String str = "my.file.png.jpeg", separator = ".";
javascript_Split(str, separator);
}
public static void javascript_Split( String str, String separator ) {
ScriptEngineManager manager = new ScriptEngineManager();
ScriptEngine engine = manager.getEngineByName("JavaScript");
// Script Variables « expose java objects as variable to script.
engine.put("strJS", str);
// JavaScript code from file
File file = new File("E:/StringSplit.js");
// expose File object as variable to script
engine.put("file", file);
try {
engine.eval("print('Script Variables « expose java objects as variable to script.', strJS)");
// javax.script.Invocable is an optional interface.
Invocable inv = (Invocable) engine;
// JavaScript code in a String
String functions = "function functionName( functionParam ) { print('Hello, ' + functionParam); }";
engine.eval(functions);
// invoke the global function named "functionName"
inv.invokeFunction("functionName", "function Param value!!" );
// evaluate a script string. The script accesses "file" variable and calls method on it
engine.eval("print(file.getAbsolutePath())");
// evaluate JavaScript code from given file - specified by first argument
engine.eval( new java.io.FileReader( file ) );
String[] typedArray = (String[]) inv.invokeFunction("splitasJavaArray", str );
System.out.println("File : Function returns an array : "+ typedArray[1] );
ScriptObjectMirror scriptObject = (ScriptObjectMirror) inv.invokeFunction("splitasJavaScriptArray", str, separator );
System.out.println("File : Function return script obj : "+ convert( scriptObject ) );
Object eval = engine.eval("(function() {return ['a', 'b'];})()");
Object result = convert(eval);
System.out.println("Result: {}"+ result);
// JavaScript code in a String. This code defines a script object 'obj' with one method called 'hello'.
String objectFunction = "var obj = new Object(); obj.hello = function(name) { print('Hello, ' + name); }";
engine.eval(objectFunction);
// get script object on which we want to call the method
Object object = engine.get("obj");
inv.invokeMethod(object, "hello", "Yash !!" );
Object fileObjectFunction = engine.get("objfile");
inv.invokeMethod(fileObjectFunction, "hello", "Yashwanth !!" );
} catch (ScriptException e) {
e.printStackTrace();
} catch (NoSuchMethodException e) {
e.printStackTrace();
} catch (FileNotFoundException e) {
e.printStackTrace();
}
}
public static Object convert(final Object obj) {
System.out.println("\tJAVASCRIPT OBJECT: {}"+ obj.getClass());
if (obj instanceof Bindings) {
try {
final Class<?> cls = Class.forName("jdk.nashorn.api.scripting.ScriptObjectMirror");
System.out.println("\tNashorn detected");
if (cls.isAssignableFrom(obj.getClass())) {
final Method isArray = cls.getMethod("isArray");
final Object result = isArray.invoke(obj);
if (result != null && result.equals(true)) {
final Method values = cls.getMethod("values");
final Object vals = values.invoke(obj);
System.err.println( vals );
if (vals instanceof Collection<?>) {
final Collection<?> coll = (Collection<?>) vals;
Object[] array = coll.toArray(new Object[0]);
return array;
}
}
}
} catch (ClassNotFoundException | NoSuchMethodException | SecurityException
| IllegalAccessException | IllegalArgumentException | InvocationTargetException e) {
}
}
if (obj instanceof List<?>) {
final List<?> list = (List<?>) obj;
Object[] array = list.toArray(new Object[0]);
return array;
}
return obj;
}
}
JavaScript file « StringSplit.js
// var str = 'angular.1.5.6.js', separator = ".";
function splitasJavaArray( str ) {
var result = str.replace(/\.([^.]+)$/, ':$1').split(':');
print('Regex Split : ', result);
var JavaArray = Java.to(result, "java.lang.String[]");
return JavaArray;
// return result;
}
function splitasJavaScriptArray( str, separator) {
var arr = str.split( separator ); // Split the string using dot as separator
var lastVal = arr.pop(); // remove from the end
var firstVal = arr.shift(); // remove from the front
var middleVal = arr.join( separator ); // Re-join the remaining substrings
var mainArr = new Array();
mainArr.push( firstVal ); // add to the end
mainArr.push( middleVal );
mainArr.push( lastVal );
return mainArr;
}
var objfile = new Object();
objfile.hello = function(name) { print('File : Hello, ' + name); }
JavaScript Array constructor or array literal.

With Guava:
final Splitter splitter = Splitter.on("-").trimResults();
assertEquals("Günnewig Uebachs", Iterables.getLast(splitter.split(one)));
assertEquals("Madison", Iterables.getLast(splitter.split(two)));
Splitter, Iterables

Since he was asking to do it all in the same line using split so i suggest this:
lastone = one.split("-")[(one.split("-")).length -1]
I always avoid defining new variables as far as I can, and I find it a very good practice

You mean you don't know the sizes of the arrays at compile-time? At run-time they could be found by the value of lastone.length and lastwo.length .

Also you can use java.util.ArrayDeque
String last = new ArrayDeque<>(Arrays.asList("1-2".split("-"))).getLast();

In java 8
String lastItem = Stream.of(str.split("-")).reduce((first,last)->last).get();

I guess you want to do this in i line. It is possible (a bit of juggling though =^)
new StringBuilder(new StringBuilder("Düsseldorf - Zentrum - Günnewig Uebachs").reverse().toString().split(" - ")[0]).reverse()
tadaa, one line -> the result you want (if you split on " - " (space minus space) instead of only "-" (minus) you will loose the annoying space before the partition too =^) so "Günnewig Uebachs" instead of " Günnewig Uebachs" (with a space as first character)
Nice extra -> no need for extra JAR files in the lib folder so you can keep your application light weight.