We Keep Coding

Read from ByteArrayOutputStream while it's being written to

I have a class that is constantly producing data and writing it to a ByteArrayOutputStream on its own thread. I have a 2nd thread that gets a reference to this ByteArrayOutputStream. I want the 2nd thread to read any data (and empty) the ByteArrayOutputStream and then stop when it doesn't get any bytes and sleep. After the sleep, I want it to try to get more data and empty it again.
The examples I see online say to use PipedOutputStream. If my first thread is making the ByteArrayOutputStream available to the outside world from a separate reusable library, I don't see how to hook up the inputStream to it.
How would one setup the PipedInputStream to connect it to the ByteArrayOutputStream to read from it as above? Also, when reading the last block from the ByteArrayOutputStream, will I see bytesRead == -1, indicating when the outputStream is closed from the first thread?
Many thanks,
Mike

Write to the PipedOutputStream directly (that is, don't use a ByteArrayOutputStream at all). They both extend OutputStream and so have the same interface.
There are connect methods in both PipedOutputStream and PipedInputStream that are used to wire two pipes together, or you can use one of the constructors to create a pair.
Writes to the PipedOutputStream will block when the buffer in the PipedInputStream fills up, and reads from the PipedInputStream will block when the buffer is empty, so the producer thread will sleep (block) if it gets "ahead" of the consumer and vice versa.
After blocking the threads wait for 1000ms before rechecking the buffer, so it's good practice to flush the output after writes complete (this will wake the reader if it is sleeping).
Your input stream will see the EOF (bytesRead == -1) when you close the output stream in the producer thread.
import java.io.*;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public class PipeTest {
public static void main(String[] args) throws IOException {
PipedOutputStream out = new PipedOutputStream();
// Wire an input stream to the output stream, and use a buffer of 2048 bytes
PipedInputStream in = new PipedInputStream(out, 2048);
ExecutorService executor = Executors.newCachedThreadPool();
// Producer thread.
executor.execute(() -> {
try {
for (int i = 0; i < 10240; i++) {
out.write(0);
// flush to wake the reader
out.flush();
}
out.close();
} catch (IOException e) {
throw new UncheckedIOException(e);
}
});
// Consumer thread.
executor.execute(() -> {
try {
int b, read = 0;
while ((b = in.read()) != -1) {
read++;
}
System.out.println("Read " + read + " bytes.");
} catch (IOException e) {
throw new UncheckedIOException(e);
}
});
executor.shutdown();
}
}

java CountDownLatch in a thread and race condition

I have a thread that must read the stdin and the stdout from a console application. This thread must to read continually until the end of the java program (the java program starts the console application and read and write with him).
Now I have this question: the tread, when reads from console application, if reads a determined thing must to set a flag variable that runs an exception. I solved putting a latch CountDown. But when the thread doesn't read this thing, the barrier latch.await() locks the continuos of the java program and I don't know how to resolve this trouble.
This is the part of code:
p = Runtime.getRuntime().exec("testpad -i -c"+can+" -n"+pad+" "+pathFile);
final InputStream inStream = p.getInputStream();
Thread uiThread = new Thread("UIHandler") {
#Override
public void run() {
InputStreamReader reader = new InputStreamReader(inStream);
Scanner scan = new Scanner(reader);
String prec=null;
while (scan.hasNextLine()) {
prec=scan.nextLine();
System.out.println(prec);
if(err.equals(prec)){
flag[0] = 1;
latch.countDown();
}
}
}
};
uiThread.start();
latch.await();
if(flag[0]!=1){
this.dispose();
new menu().setVisible(true);
}
else{
Exception e = new Exception("Error!");
Component f = null;
JOptionPane.showMessageDialog(f, err, e.getMessage(), JOptionPane.WARNING_MESSAGE);
this.dispose();
new inizio().setVisible(true);
}
In the if in the thread there is the part that sets the flag variable. But if the thread doesn't enter there, the java program blocks for the barriel latch.await(). Is there a method to do this thing and never blocks the program? Thanks at all.

latch.await() will block until the latch finishes so you should move that line to the end.

Java Process.waitFor() and IO streams

I have the following code;
String[] cmd = { "bash", "-c", "~/path/to/script.sh" };
Process p = Runtime.getRuntime().exec(cmd);
PipeThread a = new PipeThread(p.getInputStream(), System.out);
PipeThread b = new PipeThread(p.getErrorStream(), System.err);
p.waitFor();
a.die();
b.die();
The PipeThread class is quite simple so I will include it in full;
public class PipeThread implements Runnable {
private BufferedInputStream in;
private BufferedOutputStream out;
public Thread thread;
private boolean die = false;
public PipeThread(InputStream i, OutputStream o) {
in = new BufferedInputStream(i);
out = new BufferedOutputStream(o);
thread = new Thread(this);
thread.start();
}
public void die() { die = true; }
public void run() {
try {
byte[] b = new byte[1024];
while(!die) {
int x = in.read(b, 0, 1024);
if(x > 0) out.write(b, 0, x);
else die();
out.flush();
}
}
catch(Exception e) { e.printStackTrace(); }
try {
in.close();
out.close();
}
catch(Exception e) { }
}
}
My problem is this; p.waitFor() blocks endlessly, even after the subprocess has terminated. If I do not create the pair of PipeThread instances, then p.waitFor() works perfectly. What is it about the piping of io streams that is causing p.waitFor() to continue blocking?
I'm confused as I thought the IO streams would be passive, unable to keep a process alive, or to make Java think the process is still alive.

In your PipeThread code you are looping forever until !die - but you call PipeThread.die() after p.waitFor() - what's exactly stopping the PipeThread threads?

So after much head-scratching, I realised what is happening. p.waitFor() actually isn't blocking indefinitely, rather my method of checking it is failing; a System.out.println() statement after p.waitFor().
PipeThread is a class that I had lying around from a previous project, and I have often used it to pipe one stream onto another in a separate thread.
This must be the first time I have used it with System.out. PipeThread closes whatever streams it is passed upon reading EOF. Both streams in this case included my standard output, so debugging with System.out.println becomes impossible... :(
Weirdly, no IOException was thrown by System.out.println(), I will investigate.

GZIPOutputStream that does its compression in a separate thread

Is there an implemetation of GZIPOutputStream that would do the heavy lifting (compressing + writing to disk) in a separate thread?
We are continuously writing huge amounts of GZIP-compressed data. I am looking for a drop-in replacement that could be used instead of GZIPOutputStream.

You can write to a PipedOutputStream and have a thread which reads the PipedInputStream and copies it to any stream you like.
This is a generic implementation. You give it an OutputStream to write to and it returns an OutputStream for you to write to.
public static OutputStream asyncOutputStream(final OutputStream out) throws IOException {
PipedOutputStream pos = new PipedOutputStream();
final PipedInputStream pis = new PipedInputStream(pos);
new Thread(new Runnable() {
#Override
public void run() {
try {
byte[] bytes = new byte[8192];
for(int len; (len = pis.read(bytes)) > 0;)
out.write(bytes, 0, len);
} catch(IOException ioe) {
ioe.printStackTrace();
} finally {
close(pis);
close(out);
}
}
}, "async-output-stream").start();
return pos;
}
static void close(Closeable closeable) {
if (closeable != null) try {
closeable.close();
} catch (IOException ignored) {
}
}

I published some code that does exactly what you are looking for. It has always frustrated me that Java doesn't automatically pipeline calls like this across multiple threads, in order to overlap computation, compression, and disk I/O:
https://github.com/lukehutch/PipelinedOutputStream
This class splits writing to an OutputStream into separate producer and consumer threads (actually, starts a new thread for the consumer), and inserts a blocking bounded buffer between them. There is some data copying between buffers, but this is done as efficiently as possible.
You can even layer this twice to do the disk writing in a separate thread from the gzip compression, as shown in README.md.

Flaws with PipedInputStream/PipedOutputStream

I've seen two answers on SO that claim that the PipedInputStream and PipedOutputStream classes provided by Java are flawed. But they did not elaborate on what was wrong with them. Are they really flawed, and if so in what way? I'm currently writing some code that uses them, so I'd like to know whether I'm taking a wrong turn.
One answer said:
PipedInputStream and PipedOutputStream are broken (with regards to threading). They assume each instance is bound to a particular thread. This is bizarre.
To me that seems neither bizarre nor broken. Perhaps the author also had some other flaws in mind?
Another answer said:
In practice they are best avoided. I've used them once in 13 years and I wish I hadn't.
But that author could not recall what the problem was.
As with all classes, and especially classes used in multiple threads, you will have problems if you misuse them. So I do not consider the unpredictable "write end dead" IOException that PipedInputStream can throw to be a flaw (failing to close() the connected PipedOutputStream is a bug; see the article Whats this? IOException: Write end dead, by Daniel Ferbers, for more information). What other claimed flaws are there?

They are not flawed.
As with all classes, and especially classes used in multiple threads, you will have problems if you misuse them. The unpredictable "write end dead" IOException that PipedInputStream can throw is not a flaw (failing to close() the connected PipedOutputStream is a bug; see the article Whats this? IOException: Write end dead, by Daniel Ferbers, for more information).

I have used them nicely in my project and they are invaluable for modifying streams on the fly and passing them around. The only drawback seemed to be that PipedInputStream had a short buffer (around 1024) and my outputstream was pumping in around 8KBs.
There is no defect with it and it works perfectly well.
-------- Example in groovy
public class Runner{
final PipedOutputStream source = new PipedOutputStream();
PipedInputStream sink = new PipedInputStream();
public static void main(String[] args) {
new Runner().doit()
println "Finished main thread"
}
public void doit() {
sink.connect(source)
(new Producer(source)).start()
BufferedInputStream buffer = new BufferedInputStream(sink)
(new Consumer(buffer)).start()
}
}
class Producer extends Thread {
OutputStream source
Producer(OutputStream source) {
this.source=source
}
#Override
public void run() {
byte[] data = new byte[1024];
println "Running the Producer..."
FileInputStream fout = new FileInputStream("/Users/ganesh/temp/www/README")
int amount=0
while((amount=fout.read(data))>0)
{
String s = new String(data, 0, amount);
source.write(s.getBytes())
synchronized (this) {
wait(5);
}
}
source.close()
}
}
class Consumer extends Thread{
InputStream ins
Consumer(InputStream ins)
{
this.ins = ins
}
public void run()
{
println "Consumer running"
int amount;
byte[] data = new byte[1024];
while ((amount = ins.read(data)) >= 0) {
String s = new String(data, 0, amount);
println "< $s"
synchronized (this) {
wait(5);
}
}
}
}

One flaw might be that there is not clear way for the writer to indicate to the reader that it encountered a problem:
PipedOutputStream out = new PipedOutputStream();
PipedInputStream in = new PipedInputStream(out);
new Thread(() -> {
try {
writeToOut(out);
out.close();
}
catch (SomeDataProviderException e) {
// Have to notify the reading side, but how?
}
}).start();
readFromIn(in);
The writer could close out, but maybe the reader misinterprets that as end of data. To handle this correctly additional logic is needed. It would be easier if functionality to manually break the pipe was provided.
There is now JDK-8222924 which requests a way to manually break the pipe.

From my point of view there is a flaw. More precisely there is a high risk of a deadlock if the Thread which should pump data into the PipedOutputStream dies prematurely before it actually writes a single byte into the stream. The problem in such a situation is that the implementation of the piped streams is not able to detect the broken pipe. Consequently the thread reading from PipedInputStream will wait forever (i.e. deadlock) in it's first call to read().
Broken pipe detection actually relies on the first call to write() as the implementation will than lazily initialize the write side thread and only from that point in time broken pipe detection will work.
The following code reproduces the situation:
import java.io.IOException;
import java.io.PipedInputStream;
import java.io.PipedOutputStream;
import org.junit.Test;
public class PipeTest
{
#Test
public void test() throws IOException
{
final PipedOutputStream pout = new PipedOutputStream();
PipedInputStream pin = new PipedInputStream();
pout.connect(pin);
Thread t = new Thread(new Runnable()
{
public void run()
{
try
{
if(true)
{
throw new IOException("asd");
}
pout.write(0); // first byte which never get's written
pout.close();
}
catch(IOException e)
{
throw new RuntimeException(e);
}
}
});
t.start();
pin.read(); // wait's forever, e.g. deadlocks
}
}

The flaws that I see with the JDK implementation are:
1) No timeouts, reader or writer can block infinitely.
2) Suboptimal control over when data is transferred (should be done only with flush, or when circular buffer is full)
So I created my own to address the above, (timeout value passed via a ThreadLocal):
PipedOutputStream
How to use:
PiedOutputStreamTest
Hope it helps...