How to finish the work correctly at the output end of the pipe? I need the writing thread to terminate or do some other work, while the reading thread reads all written data up to end.
Should I close the pipe at the writing end or what?
UPDATE 1
I want to clarify... According to given answers, am I correct thinking that by-design pipes behavior does not suppose any graceful termination?
I.e. once opened, the only way to stop piping is to break the pipe?
Conventional streams expect end of the stream signal, when read() method returns -1. Am right thinking that this never happens with piped streams?
Yes, closing the PipedOutputStream results in a -1 on the PipedInputStream.
Looks pretty graceful to me! Here's my SSCCE:
import java.io.*;
import java.nio.charset.*;
public class SOPipe
{
public static void main(String[] args) throws Exception
{
PipedOutputStream os = new PipedOutputStream();
PipedInputStream is = new PipedInputStream(os);
ReaderThread readerThread = new ReaderThread(is);
WriterThread writerThread = new WriterThread(os);
readerThread.start();
writerThread.start();
readerThread.join();
writerThread.join();
System.out.println("Both Reader and Writer completed.");
System.out.println("Main method returning normally now.");
}
private static final Charset LATIN1 = Charset.forName("latin1");
public static class WriterThread extends Thread
{
private final PipedOutputStream _os;
public WriterThread(PipedOutputStream os)
{
_os = os;
}
public void run()
{
try
{
String msg = "Ceci n'est pas une pipe";
byte[] msgBytes = msg.getBytes(LATIN1);
System.out.println("WriterThread sending message: " + msg);
for(int i = 0; i < msgBytes.length; i++)
{
_os.write(msgBytes, i, 1);
System.out.println("WriterThread wrote a byte!");
_os.flush();
}
_os.close();
System.out.println("[COMPLETED] WriterThread");
}
catch (Exception e)
{
e.printStackTrace();
}
}
}
public static class ReaderThread extends Thread
{
private final PipedInputStream _is;
public ReaderThread(PipedInputStream is)
{
_is = is;
}
public void run()
{
try
{
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buffer = new byte[1];
int read;
while ((read = _is.read(buffer, 0, 1)) != -1)
{
System.out.println("ReaderThread read a byte!");
baos.write(buffer, 0, read);
}
System.out.println("[COMPLETED] ReaderThread; received: "
+ new String(baos.toByteArray(), LATIN1));
_is.close();
}
catch (Exception e)
{
e.printStackTrace();
}
}
}
}
It is enough just to close output stream.
You can extend PipedOutputStream class & override its write() method to add your custom logic, after it has written all the bytes to the piped output stream.
public class CustomPipedOutput extends PipedOutputStream {
#Override
public void write(byte[] byteArray, int offset, int length){
super.write(byteArray, offset, length);
//-- Code to be executed after writing bytes
}
#Override
public void close(){
super.close();
//-- Code to be executed after closing piped input stream
}
}
Similarly, can extend other methods if required accordingly.
Related
I want to communicate as a TCP Server on Port 2000 and 2001 with my TCP Client (Machine which sends Bytestreams).
Therefore I programmed a Spring Boot Application in Java.
This Question is only for Port 2001:
I use Camunda as BPMN-Engine for executing and orchestrating.
I start Threads like this:
package com.example.workflow;
import org.camunda.bpm.engine.delegate.DelegateExecution;
import org.camunda.bpm.engine.delegate.JavaDelegate;
public class StartTCPServersDelegate implements JavaDelegate {
#Override
public void execute(DelegateExecution delegateExecution) throws Exception {
Runnable serverZyklisch = new ServerZyklisch();
Runnable serverAzyklisch = new ServerAzyklisch((String) delegateExecution.getVariable("param"));
Thread t1 = new Thread(serverZyklisch);
t1.start();
System.out.println("Thread Zyklisch gestartet");
Thread t2 = new Thread(serverAzyklisch);
t2.start();
System.out.println("Thread Azyk. gestartet");
String val1 = (String) delegateExecution.getVariable("param");
int valueParam = Integer.parseInt(val1);
System.out.println("Param ist: "+valueParam);
}
}
This is my ServerAzyklisch Class:
public class ServerAzyklisch implements Runnable, JavaDelegate {
private ServerSocket ssocket;
String param;
HexToByteConverter hexToByteConverter = new HexToByteConverter();
public ServerAzyklisch(String Pparam) throws IOException {
ssocket = new ServerSocket(2000);
param = Pparam;
}
public void run() {
System.out.println(param+"Paraaam");
InputStream in;
OutputStream out = null;
Socket socket;
while(true){
try {
socket = ssocket.accept();
in = socket.getInputStream();
out = socket.getOutputStream();
byte []data = new byte[132];
int numBytes = 0;
byte[]durch = hexToByteConverter.hexStringToByteArray("333333330041006400040000000400000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000");
byte[]durchlauf = hexToByteConverter.hexStringToByteArray("333333330041006400040000000400000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000");
byte[]Pressen1hexdump111 = hexToByteConverter.hexStringToByteArray("33333333003d0064000600000004004001c9c78900010000006f00000000000000000000000000010000000000140000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000005");
byte[]Pressen1hexdump110 = hexToByteConverter.hexStringToByteArray("33333333003d0064000600000004004001c9c78900010000006e0000000000000000000000000001000000000014000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"+param);
byte[]Pressen2hexdump = hexToByteConverter.hexStringToByteArray("3333333300400065000a00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000");
byte[]Pressen3hexdump = hexToByteConverter.hexStringToByteArray("3333333300400065001400000000003d01c9c7890001000000c9000000000000000000000000000100000000001e00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000");
byte[]Pressen3hexdumpNextBohrer = hexToByteConverter.hexStringToByteArray("3333333300400065001400000000003f01c9c789000100000078000000000000000000000000000100000000001e00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002");
byte[]Pressen4hexdumpNextRSCIDBohrer = hexToByteConverter.hexStringToByteArray("33333333003f0065001400000000003d01c9c78900010000007a000000000000000000000000000100000000001e00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000");
//gleichen Stream zurückschicken, der angekommen ist, für Durchlauf
while((numBytes = in.read(data)) != -1){
System.out.println(Arrays.toString(data));
out.write(Pressen1hexdump110);
out.write(Pressen2hexdump);
out.write(Pressen3hexdumpNextBohrer);
//out.write(durchlauf);
}
} catch (SocketException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
}
#Override
public void execute(DelegateExecution delegateExecution) throws IOException {
}
}
I get everytime a different Result to my Client, so the behaviour is always another. But I want to send once all three bytearrays to my Client. I think something is wrong with my while loop.
Do you have any idea ?
By the comments, the communication is based on request-response pairs. You need to read 3 messages from the client, and return a response for each message. To do this, replace the while loop with:
readMessage(in, data);
out.write(Pressen1hexdump110);
readMessage(in, data);
out.write(Pressen2hexdump);
readMessage(in, data);
out.write(Pressen3hexdumpNextBohrer);
where the readMessage method is a new method you must add, that reads a complete request from the client.
If the client requests are always 128 bytes, there is a convenient method in DataInputStream that you can use:
void readMessage(InputStream in, byte[] buffer) throws IOException {
new DataInputStream(in).readFully(buffer, 0, 128);
}
In the general case the readMessage method would have to look something like this in pseudo-code:
void readMessage(InputStream in, byte[] buffer) {
// Read a message
while message is not complete:
read from "in" into "buffer"
if "in" was closed: throw an exception because the connection was closed mid-request
else: incorporate newly read data from "buffer" in message
done
}
I am trying to write the data using the pipe input streams. But from thread dump it looks like there is a lock on pipe input stream.
PipedOutputStream pos = new PipedOutputStream();
PipedInputStream pis = new PipedInputStream(pos);
FileInputStream fis = null;
GZIPOutputStream gos = null;
byte[] buffer = new byte[1024];
try {
fis = new FileInputStream(file);
gos = new GZIPOutputStream(pos);
int length;
while ((length = fis.read(buffer, 0, 1024)) != -1)
gos.write(buffer, 0, length);
} catch(Exception e){
print("Could not read the file");
}
finally {
try {
fis.close();
gos.close();
}catch (Exception ie){
printException(ie);
}
}
writeObject(pis);
pos.close();
writeobj method will simply read from the stream and but read method gets locked.
The thread dumps indicate some wait on pipe input stream.
main" prio=10 tid=0x08066000 nid=0x48d2 in Object.wait() [0xb7fd2000..0xb7fd31e8]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xa5c28be8> (a java.io.PipedInputStream)
at java.io.PipedInputStream.awaitSpace(PipedInputStream.java:257)
at java.io.PipedInputStream.receive(PipedInputStream.java:215)
- locked <0xa5c28be8> (a java.io.PipedInputStream)
at java.io.PipedOutputStream.write(PipedOutputStream.java:132)
at java.util.zip.GZIPOutputStream.finish(GZIPOutputStream.java:95)
at java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:146)
Locked ownable synchronizers:
- None
I am not really sure who is locking it up. Read docs to figure out the locking calls. But could not figure out what is going wrong and how to overcome it.
Working with PipedInputStream and PipedOutputStream must be in separate threads.
Read the Javadoc carefully:
http://docs.oracle.com/javase/6/docs/api/java/io/PipedInputStream.html
Typically, data is read from a PipedInputStream object by one thread and data is written to the corresponding PipedOutputStream by some other thread. Attempting to use both objects from a single thread is not recommended, as it may deadlock the thread.
PipedInputStream has a small non-expanding buffer. Once the buffer is full, writes to the PipedOutputStream block until the buffered input is read by a different thread. You cannot use the two from the same thread, because the write will be waiting for a read that cannot happen.
In your case, you are not reading any of the data until you have written all of it, so the solution is to use a ByteArrayOutputStream and ByteArrayInputStream instead:
Write all the data to a ByteArrayOutputStream.
When finished, call toByteArray() on the stream to retrieve the byte data.
(Optional) Create a ByteArrayInputStream with the byte data to read from it as an InputStream.
I needed a filter to intercept slow connections where I need to close DB connections ASAP so I initially used Java pipes but when looked closer at their implementation, it is all synchronized so I ended up creating my own QueueInputStream using a small buffer and Blocking queue to put the buffer in the queue once was full, it is lock free except when for the lock conditions used at LinkedBlockingQueue which with the aid of the small buffer it should be cheap, this class is only intended to be used for a single producer and consumer per instance:
import java.io.IOException;
import java.io.OutputStream;
import java.util.concurrent.*;
public class QueueOutputStream extends OutputStream
{
private static final int DEFAULT_BUFFER_SIZE=1024;
private static final byte[] END_SIGNAL=new byte[]{};
private final BlockingQueue<byte[]> queue=new LinkedBlockingDeque<>();
private final byte[] buffer;
private boolean closed=false;
private int count=0;
public QueueOutputStream()
{
this(DEFAULT_BUFFER_SIZE);
}
public QueueOutputStream(final int bufferSize)
{
if(bufferSize<=0){
throw new IllegalArgumentException("Buffer size <= 0");
}
this.buffer=new byte[bufferSize];
}
private synchronized void flushBuffer()
{
if(count>0){
final byte[] copy=new byte[count];
System.arraycopy(buffer,0,copy,0,count);
queue.offer(copy);
count=0;
}
}
#Override
public synchronized void write(final int b) throws IOException
{
if(closed){
throw new IllegalStateException("Stream is closed");
}
if(count>=buffer.length){
flushBuffer();
}
buffer[count++]=(byte)b;
}
#Override
public synchronized void write(final byte[] b, final int off, final int len) throws IOException
{
super.write(b,off,len);
}
#Override
public synchronized void close() throws IOException
{
flushBuffer();
queue.offer(END_SIGNAL);
closed=true;
}
public Future<Void> asyncSendToOutputStream(final ExecutorService executor, final OutputStream outputStream)
{
return executor.submit(
new Callable<Void>()
{
#Override
public Void call() throws Exception
{
try{
byte[] buffer=queue.take();
while(buffer!=END_SIGNAL){
outputStream.write(buffer);
buffer=queue.take();
}
outputStream.flush();
} catch(Exception e){
close();
throw e;
} finally{
outputStream.close();
}
return null;
}
}
);
}
This is a follow up to this question. The answer suggested there is
to copy the Process out, err, and input streams to the System versions
with IOUtils.copy as follows (after fixing various compilation errors):
import org.apache.commons.io.IOUtils;
import java.io.IOException;
public class Test {
public static void main(String[] args)
throws IOException, InterruptedException {
final Process process = Runtime.getRuntime().exec("/bin/sh -i");
new Thread(new Runnable() {public void run() {
try {
IOUtils.copy(process.getInputStream(), System.out);
} catch (IOException e) {}
} } ).start();
new Thread(new Runnable() {public void run() {
try {
IOUtils.copy(process.getErrorStream(), System.err);
} catch (IOException e) {}
} } ).start();
new Thread(new Runnable() {public void run() {
try {
IOUtils.copy(System.in, process.getOutputStream());
} catch (IOException e) {}
} } ).start();
process.waitFor();
}
}
However, the resulting code doesn't work for interactive processes like the one executing sh -i command. In the latter case there is no response to any of the sh commands.
So my question is: could you suggest an alternative to copy the streams that will work with interactive processes?
The problem is that IOUtil.copy() is running while there is data in the InputStream to be copied. Since your process only produces data from time to time, IOUtil.copy() exits as it thinks there is no data to be copied.
Just copy data by hand and use a boolean to stop the thread form outside:
byte[] buf = new byte[1024];
int len;
while (threadRunning) { // threadRunning is a boolean set outside of your thread
if((len = input.read(buf)) > 0){
output.write(buf, 0, len);
}
}
This reads in chunks as many bytes as there are available on inputStream and copies all of them to output. Internally InputStream puts thread so wait() and then wakes it when data is available.
So it's as efficient as you can have it in this situation.
Process.getOutputStream() returns a BufferedOutputStream, so if you want your input to actually get to the subprocess you have to call flush() after every write().
You can also rewrite your example to do everything on one thread (although it uses polling to read both System.in and the process' stdout at the same time):
import java.io.*;
public class TestProcessIO {
public static boolean isAlive(Process p) {
try {
p.exitValue();
return false;
}
catch (IllegalThreadStateException e) {
return true;
}
}
public static void main(String[] args) throws IOException {
ProcessBuilder builder = new ProcessBuilder("bash", "-i");
builder.redirectErrorStream(true); // so we can ignore the error stream
Process process = builder.start();
InputStream out = process.getInputStream();
OutputStream in = process.getOutputStream();
byte[] buffer = new byte[4000];
while (isAlive(process)) {
int no = out.available();
if (no > 0) {
int n = out.read(buffer, 0, Math.min(no, buffer.length));
System.out.println(new String(buffer, 0, n));
}
int ni = System.in.available();
if (ni > 0) {
int n = System.in.read(buffer, 0, Math.min(ni, buffer.length));
in.write(buffer, 0, n);
in.flush();
}
try {
Thread.sleep(10);
}
catch (InterruptedException e) {
}
}
System.out.println(process.exitValue());
}
}
You should instead use the ProcessBuilder.redirectOutput method & friends. Read more here
I use Runtime exec() method to create a subprocess in Java. However, since the subprocess is an interactive program, I need to provide input to it as and when required by it. Also I need to show the output of the subprocess. How can I do this in the simplest possible way?
I was using a StreamGobbler to show the program output using process.getInputStream(). I, however, do not know how to identify when the program is waiting for input and when to provide it input using proc.getOutputStream. How can I do this?
You need to copy the input and output between the subprocess' streams and System streams (System.in, System.out and System.err). This is related to my recent quesion. The best solution I have found so far is:
import java.io.FileInputStream;
import java.io.FilterInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.lang.reflect.Field;
import java.nio.ByteBuffer;
import java.nio.channels.AsynchronousCloseException;
import java.nio.channels.FileChannel;
class StreamCopier implements Runnable {
private InputStream in;
private OutputStream out;
public StreamCopier(InputStream in, OutputStream out) {
this.in = in;
this.out = out;
}
public void run() {
try {
int n;
byte[] buffer = new byte[4096];
while ((n = in.read(buffer)) != -1) {
out.write(buffer, 0, n);
out.flush();
}
}
catch (IOException e) {
System.out.println(e);
}
}
}
class InputCopier implements Runnable {
private FileChannel in;
private OutputStream out;
public InputCopier(FileChannel in, OutputStream out) {
this.in = in;
this.out = out;
}
public void run() {
try {
int n;
ByteBuffer buffer = ByteBuffer.allocate(4096);
while ((n = in.read(buffer)) != -1) {
out.write(buffer.array(), 0, n);
out.flush();
}
out.close();
}
catch (AsynchronousCloseException e) {}
catch (IOException e) {
System.out.println(e);
}
}
}
public class Test {
private static FileChannel getChannel(InputStream in)
throws NoSuchFieldException, IllegalAccessException {
Field f = FilterInputStream.class.getDeclaredField("in");
f.setAccessible(true);
while (in instanceof FilterInputStream)
in = (InputStream)f.get((FilterInputStream)in);
return ((FileInputStream)in).getChannel();
}
public static void main(String[] args)
throws IOException, InterruptedException,
NoSuchFieldException, IllegalAccessException {
Process process = Runtime.getRuntime().exec("sh -i +m");
Thread outThread = new Thread(new StreamCopier(
process.getInputStream(), System.out));
outThread.start();
Thread errThread = new Thread(new StreamCopier(
process.getErrorStream(), System.err));
errThread.start();
Thread inThread = new Thread(new InputCopier(
getChannel(System.in), process.getOutputStream()));
inThread.start();
process.waitFor();
System.in.close();
outThread.join();
errThread.join();
inThread.join();
}
}
The tricky part here is to extract a channel from System.in. Without this you will not be able to interrupt the thread that reads input when the subprocess terminates.
This approach has a serious drawback: after closing System.in you can no longer read from it. The workaround that I'm currently using is to have a single input redirecting thread used for all subprocesses.
Ask yourself "How do I know when the program wants input when I run it from the command line"? You see what it prompts and enter data based on that prompt. The principle will be the same, except your code will need to interpret the program's output and provide the correct input.
To avoid reinventing the wheel, take a look at ExpectJ and/or Expect4J, which are Java implementations of the venerable *nix Expect tool, which is designed to handle this kind of programmatic interaction.
When launching a process from Java, both stderr and stdout can block on output if I don't read from the pipes. Currently I have a thread that pro-actively reads from one and the main thread blocks on the other.
Is there an easy way to join the two streams or otherwise cause the subprocess to continue while not losing the data in stderr?
Set the redirectErrorStream property on ProcessBuilder to send stderr output to stdout:
ProcessBuilder builder = new ProcessBuilder(command);
builder.redirectErrorStream(true);
You should then create a thread to deal with the process stream, something like the following:
Process p = builder.start();
InputHandler outHandler = new InputHandler(p.getInputStream());
Where InputHandler is defined as:
private static class InputHandler extends Thread {
private final InputStream is;
private final ByteArrayOutputStream os;
public InputHandler(InputStream input) {
this.is = input;
this.os = new ByteArrayOutputStream();
}
public void run() {
try {
int c;
while ((c = is.read()) != -1) {
os.write(c);
}
} catch (Throwable t) {
throw new IllegalStateException(t);
}
}
public String getOutput() {
try {
os.flush();
} catch (Throwable t) {
throw new IllegalStateException(t);
}
return os.toString();
}
}
Alternatively, just create two InputHandlers for the InputStream and ErrorStream. Knowing that the program will block if you don't read them is 90% of the battle :)
Just have two threads, one reading from stdout, one from stderr?