We Keep Coding

Java: Write to and read from same process multiple times

I've gone through so many related StackOverflow questions for this that I'm getting lost in them, and I've coded this multiple ways, but none seem to solve this problem in a way that works for me: How can I send output to the same command and process multiple times while at the same time receiving input from this same process?
(See Input various strings to same process in Java for a similar question, but this ended with only a theoretical answer.)
The command (command line, from a C++ executable) loads a large file, and then I want to send input to it very quickly, get back the answer, do other stuff in between, then send different input and get the corresponding answer. Multiply this by thousands or millions of times.
One implementation, with threads:
ProcessBuilder pb = new ProcessBuilder(command.split(" "));
kenLMProcess = pb.start();
KenLMInThread lmInput = new KenLMInThread(kenLMProcess.getInputStream());
KenLMInThread lmError = new KenLMInThread(kenLMProcess.getErrorStream());
KenLMOutThread lmOutput = new KenLMOutThread(kenLMProcess.getOutputStream());
lmOutput.inStr = "Test . \n";
lmInput.start();
lmOutput.start();
lmError.start();
lmOutput.join();
lmInput.join();
lmError.join();
outStr = lmInput.newStr;
But join waits until the thread ends. What if I don't want to wait for it to end? I can't seem to figure out how to use wait() for that purpose. For one I'd prefer to not have to keep opening and closing a new output stream and input stream every time I query the command. But at least that's better than starting a new ProcessBuilder every time.
Here's what run() looks like for KenLMOutThread:
public void run() {
try {
pw.write(inStr+"\n");
pw.write('\n');
} catch (Exception e) {
System.out.println("Error while inputting to KenLM.");
e.printStackTrace();
} finally {
pw.flush();
try {
pw.flush();
bw.close();
} catch (IOException e) {
e.printStackTrace();
}
}
Supposedly flush is supposed to let it move on, and "\n" at the end is supposed to help, but it just hangs unless I use close. And if I use close, I can't use the OutputStream anymore. I'm also then unable to make a new OutputStream from the Process.
If it helps, here's a more simple implementation with everything together (taken from How to send EOF to a process in Java?):
Note that close() is used, and using flush() without close() causes the program to hang.
public static String pipe(String str, String command2) throws IOException, InterruptedException {
Process p2 = Runtime.getRuntime().exec(command2);
OutputStream out = p2.getOutputStream();
out.write(str.getBytes());
out.close();
p2.waitFor();
BufferedReader reader
= new BufferedReader(new InputStreamReader(p2.getInputStream()));
StringBuilder sb = new StringBuilder();
String line;
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
return sb.toString();
}
Other things I've tried:
Using exec(): Process kenLMProcess=Runtime.getRuntime().exec(command);
Putting the command process in its own thread: KenLMProcessThread procThread = new KenLMProcessThread(pb.start());

If the target process is hanging unless you close the output stream, the problem is at that end: it is reading until end of stream before doing anything. Nothing you can do about that at the sending end.

Best programming way to read streaming data

I am reading a streaming data from an TCP streaming software. I'm currently using while loop to read continuously. But I am not sure that if this is the best technique to read streaming data.
Following is the code i'm currently using:
Socket client=new Socket("169.254.99.2",1234);
System.out.println("Client connected ");
//getting the o/p stream of that connection
PrintStream out=new PrintStream(client.getOutputStream());
out.print("Hello from client\n");
out.flush();
//reading the response using input stream
BufferedReader in= new BufferedReader(new InputStreamReader(client.getInputStream()));
int a = 1;
int b= 1;
//
while(a==b){
// I'm just printing it out.
System.out.println("Response" + in.read());
}
Suggestions plz???

That loop would be the same as while(true), which is continuous. Also, I suggest running this in a thread.
After you init your socket and streams, I suggest calling a method like this:
Thread messageThread;
public void chatWithServer() {
messageThread = new Thread(new Runnable() {
public void run() {
String serverInput;
while((serverInput = in.readLine()) != null) {
//do code here
}
}
};
messageThread.start();
}
We put it in a thread so the loop doesn't hold up the rest of the client's code. (does not progress after loop)
The while loop initilizes serverInput within the parameters, so each time it loops, it re-inits serverInput so it doesn't constantly loop with the very first sent piece of data.
You gotta put it in parenthesis, because of course, while loops only accept boolean parameters (true/false). So, in pseudocode, if the InputStream always returns something, continue with the new recieved data.

I'm currently using while loop to read continuously.
That is the best technique for reading streaming data. However your loop must test for end of stream, which is signalled by read() retuning -1 in Java. Your 'a==b' test is pointless. There are several possible loop tests:
while (true) // with a break when you detect EOS
Or
while ((c = in.read()) != -1)
where 'c' is an 'int'.
But I am not sure that if this is the best technique to read streaming data.
Why not?

alternative to readLine() method in BufferedReader?

I am new to java Network programming.I was googling the code for a TCP client in java.I came across the following example.
import java.lang.*;
import java.io.*;
import java.net.*;
class Client {
public static void main(String args[]) {
try {
Socket skt = new Socket("localhost", 1234);
BufferedReader in = new BufferedReader(new
InputStreamReader(skt.getInputStream()));
System.out.print("Received string: '");
while (!in.ready()) {}
System.out.println(in.readLine()); // Read one line and output it
System.out.print("'\n");
in.close();
}
catch(Exception e) {
System.out.print("Whoops! It didn't work!\n");
}
}
}
The client seems to read out the data one "line" at a time?. I am connecting to a server that is streaming OpenFlow packets.A wireshark screenshot of OpenFlow packets is given below.
[http://www.openflow.org/downloads/screenshot-openflow-dissector-2008-07-15-2103.jpg][1]
Once I recieve the complete packets I want to dump that to a file and then later read it using wireshark for example.In the above code they are using calss BufferedReader to read the data in "lines"? At least that is how I understand it.Is there someway in which I can get full packets and then write it to the file?

Readers are for working with text data. If you are working with binary data (it's not entirely clear from that screenshot), you should be working with some type of Stream (either InputStream or possibly DataInputStream). Don't just look for random examples on online, try to find ones that actually apply to what you are interested in doing.
also, don't ever use InputStream.available, it's pretty much useless. as is any example code using it.
also, a simple google search for "OpenFlow java" had some interesting hits. are you sure you need to write something from scratch?

No, but there are libraries that provides such functions. See for example Guava
http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/io/ByteStreams.html
If you don't want to (or can't) use libraries you shoud consume a stream like this
List<String> lst = new ArrayList<String>();
String line;
while ((line = in.readLine()) != null) {
lst.add(line);
}
or
String str = "";
String line;
while ((line = in.readLine()) != null) {
str += line + "\n";
}
Note that the BufferedReader.readLine() method will give you a new line on linebreaks ('\n'). If the InputStream is binary you should work with bytes instead.

How can I print .exe printf() messages from java program

I have one application that prints messages from Test.exe in console .My java program creates one process by executing this Test.exe.
This application prints messages by reading from input-stream of that process.
The problem, that I am facing is,
I have two scenarios:
1) When I double click test.exe, messages("Printing : %d") are printing for every second.
2)But when I run my java application,whole messages are printing at last(not for every second) before terminating Test.exe.If .exe has a very huge messages to print,then it will print those messages(I think whenever buffer becomes full)and flushing will be done.
But how can I print messages same as 1st case.
Help from anyone would be appreciated. :)
Here is the code for this Test.exe.
#include <stdio.h>
#include <windows.h>
void main(void)
{
int i=0;
while (1)
{
Sleep(500);
printf("\nPrinting : %d",i);
i++;
if (i==10)
//if(i==100)
{
return 0;
}
}
}
And my Java application is below:
public class MainClass {
public static void main(String[] args) {
String str = "G:\\Charan\\Test\\Debug\\Test.exe";
try {
Process testProcess = Runtime.getRuntime().exec(str);
InputStream inputStream = new BufferedInputStream(
testProcess.getInputStream());
int read = 0;
byte[] bytes = new byte[1000];
String text;
while (read >= 0) {
if (inputStream.available() > 0 ) {
read = inputStream.read(bytes);
if (read > 0) {
text = new String(bytes, 0, read);
System.out.println(text);
}
}
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
Is it possible in reverse order.If I input some text from console,Java should read and pass that String to .exe(or testProcess).How .exe scan something from Java program.
Could anyone help me..

Given that you're trying to print stdout from that process line by line, I would created a BufferedReader object using the process' input stream and use the readLine() method on that. You can get a BufferedReader object using the following chain of constructors:
BufferedReader testProcessReader = new BufferedReader(new InputStreamReader(testProcess.getInputStream()));
And to read line by line:
String line;
while ((line = testProcessReader.readLine()) != null) {
System.out.println(line);
}
The assumption here is that Test.exe is flushing its output, which is required by any read from the Java side. You can flush the output from C by calling fflush(stdout) after every call to printf().
If you don't flush, the data only lives in a buffer. When considering performance, it's a trade-off, how often you want the data to be written vs. how many writes / flush operations you want to save. If performance is critical, you can consider looking into a more efficient inter-process communication mechanism to pass data between the processes instead of stdout. Since you are on Windows, the first step might be to take a look at the Microsoft IPC help page.

Seems to have something to do with not flushing. I guess it's on both sides - The C library you use seems to only automatically flush output when writing to a terminal. Flush manually after calling printf.
On the Java side, try reading from a non-buffered stream.

Receiving mixed media over Java socket. Yours better?

I'm about to give a programming exercice in Java and I'd like my students to discover the intrinsics of HTTP themselves rather than having URLConnection doing all the job for them. In order to estimate the complexity, I came up with the following snippet, which parses the reply (imho, one of the hardest part of the job), which will return e.g. "HTTP/1.1 200 OK", push things like "Server: makato" and "content-length: 1337" in the headers vector and leave the InputStream at the first byte of the content, so that a DataInputStream or a InputStreamReader can later be built on top of it safely.
I'm curious to know if someone with more experience of the Java classes could suggest more elegant alternatives. One thing I'm not pleased with is that each individual is.read() will inevitably generate an additional system call (assuming that Socket.getInputStream() is used to feed is argument).
public static String recvHttpHeaders(InputStream is, Vector<String> headers)
throws Exception {
byte line[] = new byte[512];
String pending=null;
String status=null;
boolean complete=false, CR=false;
int n=0;
while (!complete) {
int x = is.read();
switch(x) {
case -1: throw new Exception("something went wrong");
case '\r':
if (CR) throw new Exception("encoding mismatch CRCR");
CR=true;
break;
case '\n': // bare LF are accepted silently.
String ln = new String(line,0,n,"ASCII");
if (pending!=null) ln = pending + ln;
if (status==null) status = ln;
else headers.add(ln);
complete = ln.length()==0;
pending = null;
n=0; CR=false;
break;
default:
if (CR) throw new Exception("encoding mismatch ?CR");
if (n>=512) {
String part = new String(line, "ASCII");
if (pending!=null) pending += part;
else pending = part;
n=0;
}
line[n++]=(byte)x;
break;
}
}
return status;
}
edit: admittedly, one would love to use xxx.readline() here to avoid messing up with lines reconstruction. BufferedReader (or any other *Reader, actually) converts bytes into chars according to one charset. That means I'm no longer free to chose that charset for the content if I used that feature in the header parsing. I haven't found any byte-level classes that has readline ability built-in.
performance solution: Thanks for pointing out BufferedInputStream. I made a few additional tests, and indeed, invoking as
BufferedInputStream bis = new BufferedInputStream(socket.getInputStream());
String status = recvHttpHeaders(bis, headers);
rawCopy(bis, output);
indeed reduce the amount of system calls performed and still allow me to properly receive binary content unmodified.

You should rather use BufferedReader to read texts. Wrap your input stream:
BufferedReder br = new BufferedReader(new InputStreamReader(is));
Then use readLine() to read stuff line by line:
String line = null;
while((line = br.readLine()) != null) {
// deal with the line
}

Following comments of Sripathi Krishnan and Adam Paynter, the way to improve it is to use a BufferedInputStream, so that performance remains acceptable and no charset transformation happens.