We Keep Coding

How to pass data from java to python script as parameter

I wanted to run a script from java class.
I was able to do it with ProcessBuilder.
Now I am struggling to pass data as an argument to the script. I can pass a string, but I am having trouble passing list of data (data read from a CSV file).
One that I tried was using StringBuffer and passing the data as Bytes
StringBuffer sb = new StringBuffer();
ArrayList argsList = new ArrayList();
// append the sb from record.
//convert it to bytes
//sb.toString() contains my data with a delimiter ';'
// sb.toString().getBytes("UTF-8").toString()
argsList.add(0,"Continuum/anaconda3/envs/sth/python");
argsList.add(1,"python/test.py");
argsList.add(2,"python/test.py");
argsList.add(3,sb.toString().getBytes("UTF-8").toString())
ProcessBuilder builder = new ProcessBuilder(argsList);
Process p = builder.start();
I can not pass sb.toString().getBytes("UTF-8") to the process builder.
I get java.lang.ArrayStoreException
On the other hand, if I pass sb.toString(), I get
java.io.IOException: Cannot run program Continuum/anaconda3/envs/sth/python, CreateProcess error=206, The filename or extension is too long
add it to list of commands to process builder.
How can I read what I passed in python?
Is this even the right way to pass data ( row X column)?
Thanks

You can try below code ::
import java.io.*;
class RunPythonCommand {
public static void main(String [] args) {
try {
String prg = "import sys\nprint int(sys.argv[1])+int(sys.argv[2])\n";
BufferedWriter out = new BufferedWriter(new FileWriter("sample_code.py"));
out.write(prg);
out.close();
int param1 = 10;
int param2 = 32;
ProcessBuilder pb = new ProcessBuilder("python", "sample_code.py", "" + param2, "" + param1);
Process p = pb.start();
BufferedReader in = new BufferedReader(new InputStreamReader(p.getInputStream()));
int ret = new Integer(in.readLine()).intValue();
System.out.println("value is : " + ret);
} catch (Exception e) {
System.out.println(e);
}
}
}
Hope it will help you

You can pass args as array.:
final String PYTHON = "python";
final String SCRIPT_PATH = new File("Pyscript.py").getPath();
final String ACTION = "-aexport";
final String[] CMD_ARRAY = {PYTHON, SCRIPT_PATH, ACTION};
ProcessBuilder processBuilder = new ProcessBuilder(CMD_ARRAY);

Java read process output unbuffered / realtime

I want to read the stdout of a process right as it is generated.
The process will send information for a progress indicator, so it doesn't make sense that I get the information all at once, which I do and which is the problem. I tried to use Scanner class as suggested in a post, but I still get the output only after the process has finished.
I realize this question has been asked before, but it hasn't been answered.
You will probably want to look at class StreamGobblerOutput first.
public List<String> executeCall(String fileName)
{
StringBuilder sbOutput = new StringBuilder();
StringBuilder sbError = new StringBuilder();
File file = new File(fileName);
try ( BufferedReader br = new BufferedReader(new FileReader(file)) ) {
String line;
while ((line = br.readLine()) != null) {
String [] parts = line.split("\\s");
if(parts.length<2) {
sbError.append("Command too short for call: " + parts[0]);
continue;
}
List<String> args = new ArrayList<String>();
args.add ("sfb.exe");
for(int i = 1; i <parts.length; ++i) {
args.add (parts[i]);
}
args.add (sfbPassword);
ProcessBuilder pb = new ProcessBuilder (args);
pb.directory(new File(Support.getJustThePathFromFile(file)));
Map<String, String> envs = pb.environment();
String path = envs.get("Path");
envs.put("Path", Paths.get(".").toAbsolutePath().normalize().toString() + ";" +path);
//pb.redirectOutput(new Redirect() {});
Process p = pb.start();
String outputPathPrefix = pb.directory().getCanonicalPath();
// any output?
StreamGobblerOutput outputGobbler = new StreamGobblerOutput(p.getInputStream(), outputPathPrefix);
outputGobbler.start();
// any errors?
StreamGobblerError errorGobbler = new StreamGobblerError(p.getErrorStream());
errorGobbler.start();
try
{
p.waitFor();
}
catch (InterruptedException e)
{
// TODO Auto-generated catch block
e.printStackTrace();
}
sbOutput = outputGobbler.getOutput();
sbError = errorGobbler.getErrors();
String rootPath= Support.getJustThePathFromFile(new File(fileName));
File rootFile = new File(rootPath + "/..");
String rootFolder = rootFile.getCanonicalFile().getName();
System.err.println("rootFolder: " + rootFolder);
mainApp.addModifiedFiles(outputGobbler.getModifiedFileNames(), rootFolder);
}
} catch ( IOException ex) {
sbError.append(ex.getMessage());
}
mainApp.addOutput(sbOutput.toString());
mainApp.addError(sbError.toString());
return;
}
private class StreamGobblerOutput extends Thread {
private InputStream is;
private String outputPathPrefix;
private StringBuilder sbOutput;
private List<String> modifiedFileNames;
private Scanner scanner;
private StreamGobblerOutput(InputStream is, String outputPathPrefix) {
this.is = is;
this.outputPathPrefix = outputPathPrefix;
sbOutput = new StringBuilder();
modifiedFileNames = new ArrayList<String>();
scanner = new Scanner(is);
}
public StringBuilder getOutput() {
return sbOutput;
}
public List<String> getModifiedFileNames() {
return modifiedFileNames;
}
#Override
public void run() {
//create pattern
Pattern patternProgress = Pattern.compile("\\((\\d+)%\\)");
//InputStreamReader isr = new InputStreamReader(is);
//BufferedReader br = new BufferedReader(isr);
String ligne = null;
while (scanner.hasNextLine()) {
ligne = scanner.nextLine();
sbOutput.append(ligne);
sbOutput.append("\r\n");
//bw.write("\r\n");
Matcher mProgress = patternProgress.matcher(ligne);
if (mProgress.find()) {
int percentage = Integer.parseInt(mProgress.group(1));
System.err.println("percentage=" + percentage);
mainApp.mainWindowController.setProgressExecute(percentage/100.0);
}
}
mainApp.mainWindowController.setProgressExecute(1.0);
if (scanner != null) {
scanner.close();
}
}
}
private class StreamGobblerError extends Thread {
private InputStream is;
private StringBuilder sbError;
private Scanner scanner;
private StreamGobblerError(InputStream is) {
this.is = is;
sbError = new StringBuilder();
scanner = new Scanner(is);
}
public StringBuilder getErrors() {
return sbError;
}
#Override
public void run() {
//InputStreamReader isr = new InputStreamReader(is);
//BufferedReader br = new BufferedReader(isr);
String ligne = null;
while (scanner.hasNextLine()) {
ligne = scanner.nextLine();
sbError.append(ligne);
sbError.append("\r\n");
}
if (scanner != null) {
scanner.close();
}
}
}
Update: I tried redirecting the output to a file and reading from it, but it appears this runs into the same buffering problem as the previous implementation: I get only two data points.
As a workaround, I will have to ask the creator of the .exe to include 4100 extra characters in each line showing the progress.

If your external process is C/C++ (stdio) based, than this is most likely a block buffering issue:
stdio-based programs as a rule are line buffered if they are running interactively in a terminal and block buffered when their stdout is redirected to a pipe. In the latter case, you won't see new lines until the buffer overflows or flushed.
see this answer for more details, and some possible workarounds.
Please also note that according to this, line buffering is not an option on Win32:
_IOLBF
For some systems, this provides line buffering. However, for Win32, the behavior is the same as _IOFBF - Full Buffering.
so if you choose to modify the "exe" program to set a proper output mode with setvbuf, you would have to use:
_IONBF No buffer
instead.

From the Javadocs
Optionally, a PrintStream can be created so as to flush automatically;
this means that the flush method is automatically invoked after a byte
array is written, one of the println methods is invoked, or a newline
character or byte ('\n') is written.
Reference: http://docs.oracle.com/javase/8/docs/api/java/io/PrintStream.html
One way to do this is to have output stream flush() after every write.
System.out.flush()
You could also define your own flushable PrintStream and use that.

Multiple using of Java ProcessBuilder for agrep.exe

My Java program needs to launch agrep.exe with parameters for all pairs of elements in a big matrix and get number of matching errors of two stings. I've wrote a code, but it runs very slowly. Can I speed up this part of code? Or, maybe, you can suggest me some java implementation of agrep function?
public static double getSignatureDistance(String one, String two) throws IOException, InterruptedException {
String strReprOne = one.replace(".*","").replace("\\.",".");
String strReprTwo = two.replace(".*","").replace("\\.",".");
PrintWriter writer = new PrintWriter("tmp.txt", "UTF-8");
writer.print(strReprTwo);
writer.close();
List<String> cmd = new ArrayList<>();
cmd.add("agrep.exe");
cmd.add("-B");
cmd.add(one);
cmd.add("tmp.txt");
ProcessBuilder pb = new ProcessBuilder(cmd);
pb.redirectErrorStream(true);
Process proc = pb.start();
BufferedReader in = new BufferedReader(new InputStreamReader(proc.getInputStream()));
StringBuilder lineBuilder = new StringBuilder();
String line = "";
char[] buf = new char[2];
while (in.read(buf) == 2) {
lineBuilder.append(buf);
}
line = lineBuilder.toString();
Pattern p = Pattern.compile("(\\d+)\\serror");
Matcher m = p.matcher(line);
double agrep = 0;
if(m.find()) {
agrep = Double.valueOf(m.group(1));
}
in.close();
proc.destroy();
double length = strReprOne.length();
return agrep/length;
}
Can I use FREJ library for this purpose? For example, perform match of strings, get match result and multiply it by length of matched region?

Nobody knows, so I've used FREJ library.

Collect Linux command output

I am now on a linux machine. I have a Java program which would run some linux command, for example ps, top, list or free -m.
The way to run a command in Java is as follows:
Process p = Runtime.getRuntime().exec("free -m");
How could I collect the output by Java program? I need to process the data in the output.

Use Process.getInputStream() to get an InputStream that represents the stdout of the newly created process.
Note that starting/running external processes from Java can be very tricky and has quite a few pitfalls.
They are described in this excellent article, which also describes ways around them.

To collect the output you could do something like
Process p = Runtime.getRuntime().exec("my terminal command");
p.waitFor();
BufferedReader buf = new BufferedReader(new InputStreamReader(
p.getInputStream()));
String line = "";
String output = "";
while ((line = buf.readLine()) != null) {
output += line + "\n";
}
System.out.println(output);
This would run your script and then collect the output from the script into a variable. The link in Joachim Sauer's answer has additional examples of doing this.

As for some command need to wait for a while, add p.waitFor(); if necessary.
public static void main(String[] args) {
CommandLineHelper obj = new CommandLineHelper();
String domainName = "google.com";
//in mac oxs
String command = "ping -c 3 " + domainName;
String output = obj.executeCommand(command);
System.out.println(output);
}
private String executeCommand(String command) {
StringBuffer output = new StringBuffer();
Process p;
try {
p = Runtime.getRuntime().exec(command);
p.waitFor();
BufferedReader reader =
new BufferedReader(new InputStreamReader(p.getInputStream()));
String line = "";
while ((line = reader.readLine())!= null) {
output.append(line + "\n");
}
} catch (Exception e) {
e.printStackTrace();
}
return output.toString();
}

The technicalities of calling an external process are quite involved. The jproc library helps abstracting over these by automatically consuming the output of the command and providing the result as a string. The example above would be written like this:
String result = ProcBuilder.run("free", "-m");
It also allows to set a timeout, so that your application isn't blocked by an external command that is not terminating.

public String RunLinuxGrepCommand(String command) {
String line = null;
String strstatus = "";
try {
String[] cmd = { "/bin/sh", "-c", command };
Process p = Runtime.getRuntime().exec(cmd);
BufferedReader in = new BufferedReader(new InputStreamReader(p.getInputStream()));
while ((line = in.readLine()) != null) {
strstatus = line;
}
in.close();
} catch (Exception e) {
StringWriter sw = new StringWriter();
PrintWriter pw = new PrintWriter(sw);
e.printStackTrace(pw);
pw.flush();
String stackTrace = sw.toString();
int lenoferrorstr = stackTrace.length();
if (lenoferrorstr > 500) {
strstatus = "Error:" + stackTrace.substring(0, 500);
} else {
strstatus = "Error:" + stackTrace.substring(0, lenoferrorstr - 1);
}
}
return strstatus;
}
This functioin will give result of any linux command

Android Reading from an Input stream efficiently

I am making an HTTP get request to a website for an android application I am making.
I am using a DefaultHttpClient and using HttpGet to issue the request. I get the entity response and from this obtain an InputStream object for getting the html of the page.
I then cycle through the reply doing as follows:
BufferedReader r = new BufferedReader(new InputStreamReader(inputStream));
String x = "";
x = r.readLine();
String total = "";
while(x!= null){
total += x;
x = r.readLine();
}
However this is horrendously slow.
Is this inefficient? I'm not loading a big web page - www.cokezone.co.uk so the file size is not big. Is there a better way to do this?
Thanks
Andy

The problem in your code is that it's creating lots of heavy String objects, copying their contents and performing operations on them. Instead, you should use StringBuilder to avoid creating new String objects on each append and to avoid copying the char arrays. The implementation for your case would be something like this:
BufferedReader r = new BufferedReader(new InputStreamReader(inputStream));
StringBuilder total = new StringBuilder();
for (String line; (line = r.readLine()) != null; ) {
total.append(line).append('\n');
}
You can now use total without converting it to String, but if you need the result as a String, simply add:
String result = total.toString();
I'll try to explain it better...
a += b (or a = a + b), where a and b are Strings, copies the contents of both a and b to a new object (note that you are also copying a, which contains the accumulated String), and you are doing those copies on each iteration.
a.append(b), where a is a StringBuilder, directly appends b contents to a, so you don't copy the accumulated string at each iteration.

Have you tried the built in method to convert a stream to a string? It's part of the Apache Commons library (org.apache.commons.io.IOUtils).
Then your code would be this one line:
String total = IOUtils.toString(inputStream);
The documentation for it can be found here:
http://commons.apache.org/io/api-1.4/org/apache/commons/io/IOUtils.html#toString%28java.io.InputStream%29
The Apache Commons IO library can be downloaded from here:
http://commons.apache.org/io/download_io.cgi

Another possibility with Guava:
dependency: compile 'com.google.guava:guava:11.0.2'
import com.google.common.io.ByteStreams;
...
String total = new String(ByteStreams.toByteArray(inputStream ));

I believe this is efficient enough... To get a String from an InputStream, I'd call the following method:
public static String getStringFromInputStream(InputStream stream) throws IOException
{
int n = 0;
char[] buffer = new char[1024 * 4];
InputStreamReader reader = new InputStreamReader(stream, "UTF8");
StringWriter writer = new StringWriter();
while (-1 != (n = reader.read(buffer))) writer.write(buffer, 0, n);
return writer.toString();
}
I always use UTF-8. You could, of course, set charset as an argument, besides InputStream.

What about this. Seems to give better performance.
byte[] bytes = new byte[1000];
StringBuilder x = new StringBuilder();
int numRead = 0;
while ((numRead = is.read(bytes)) >= 0) {
x.append(new String(bytes, 0, numRead));
}
Edit: Actually this sort of encompasses both steelbytes and Maurice Perry's

Possibly somewhat faster than Jaime Soriano's answer, and without the multi-byte encoding problems of Adrian's answer, I suggest:
File file = new File("/tmp/myfile");
try {
FileInputStream stream = new FileInputStream(file);
int count;
byte[] buffer = new byte[1024];
ByteArrayOutputStream byteStream =
new ByteArrayOutputStream(stream.available());
while (true) {
count = stream.read(buffer);
if (count <= 0)
break;
byteStream.write(buffer, 0, count);
}
String string = byteStream.toString();
System.out.format("%d bytes: \"%s\"%n", string.length(), string);
} catch (IOException e) {
e.printStackTrace();
}

Maybe rather then read 'one line at a time' and join the strings, try 'read all available' so as to avoid the scanning for end of line, and to also avoid string joins.
ie, InputStream.available() and InputStream.read(byte[] b), int offset, int length)

Reading one line of text at a time, and appending said line to a string individually is time-consuming both in extracting each line and the overhead of so many method invocations.
I was able to get better performance by allocating a decent-sized byte array to hold the stream data, and which is iteratively replaced with a larger array when needed, and trying to read as much as the array could hold.
For some reason, Android repeatedly failed to download the entire file when the code used the InputStream returned by HTTPUrlConnection, so I had to resort to using both a BufferedReader and a hand-rolled timeout mechanism to ensure I would either get the whole file or cancel the transfer.
private static final int kBufferExpansionSize = 32 * 1024;
private static final int kBufferInitialSize = kBufferExpansionSize;
private static final int kMillisecondsFactor = 1000;
private static final int kNetworkActionPeriod = 12 * kMillisecondsFactor;
private String loadContentsOfReader(Reader aReader)
{
BufferedReader br = null;
char[] array = new char[kBufferInitialSize];
int bytesRead;
int totalLength = 0;
String resourceContent = "";
long stopTime;
long nowTime;
try
{
br = new BufferedReader(aReader);
nowTime = System.nanoTime();
stopTime = nowTime + ((long)kNetworkActionPeriod * kMillisecondsFactor * kMillisecondsFactor);
while(((bytesRead = br.read(array, totalLength, array.length - totalLength)) != -1)
&& (nowTime < stopTime))
{
totalLength += bytesRead;
if(totalLength == array.length)
array = Arrays.copyOf(array, array.length + kBufferExpansionSize);
nowTime = System.nanoTime();
}
if(bytesRead == -1)
resourceContent = new String(array, 0, totalLength);
}
catch(Exception e)
{
e.printStackTrace();
}
try
{
if(br != null)
br.close();
}
catch(IOException e)
{
// TODO Auto-generated catch block
e.printStackTrace();
}
}
EDIT: It turns out that if you don't need to have the content re-encoded (ie, you want the content AS IS) you shouldn't use any of the Reader subclasses. Just use the appropriate Stream subclass.
Replace the beginning of the preceding method with the corresponding lines of the following to speed it up an extra 2 to 3 times.
String loadContentsFromStream(Stream aStream)
{
BufferedInputStream br = null;
byte[] array;
int bytesRead;
int totalLength = 0;
String resourceContent;
long stopTime;
long nowTime;
resourceContent = "";
try
{
br = new BufferedInputStream(aStream);
array = new byte[kBufferInitialSize];

If the file is long, you can optimize your code by appending to a StringBuilder instead of using a String concatenation for each line.

byte[] buffer = new byte[1024]; // buffer store for the stream
int bytes; // bytes returned from read()
// Keep listening to the InputStream until an exception occurs
while (true) {
try {
// Read from the InputStream
bytes = mmInStream.read(buffer);
String TOKEN_ = new String(buffer, "UTF-8");
String xx = TOKEN_.substring(0, bytes);

To convert the InputStream to String we use the
BufferedReader.readLine() method. We iterate until the BufferedReader return null which means there's no more data to read. Each line will appended to a StringBuilder and returned as String.
public static String convertStreamToString(InputStream is) {
BufferedReader reader = new BufferedReader(new InputStreamReader(is));
StringBuilder sb = new StringBuilder();
String line = null;
try {
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
is.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return sb.toString();
}
}`
And finally from any class where you want to convert call the function
String dataString = Utils.convertStreamToString(in);
complete

I am use to read full data:
// inputStream is one instance InputStream
byte[] data = new byte[inputStream.available()];
inputStream.read(data);
String dataString = new String(data);
Note that this applies to files stored on disk and not to streams with no default size.