I am trying to write the data using the pipe input streams. But from thread dump it looks like there is a lock on pipe input stream.
PipedOutputStream pos = new PipedOutputStream();
PipedInputStream pis = new PipedInputStream(pos);
FileInputStream fis = null;
GZIPOutputStream gos = null;
byte[] buffer = new byte[1024];
try {
fis = new FileInputStream(file);
gos = new GZIPOutputStream(pos);
int length;
while ((length = fis.read(buffer, 0, 1024)) != -1)
gos.write(buffer, 0, length);
} catch(Exception e){
print("Could not read the file");
}
finally {
try {
fis.close();
gos.close();
}catch (Exception ie){
printException(ie);
}
}
writeObject(pis);
pos.close();
writeobj method will simply read from the stream and but read method gets locked.
The thread dumps indicate some wait on pipe input stream.
main" prio=10 tid=0x08066000 nid=0x48d2 in Object.wait() [0xb7fd2000..0xb7fd31e8]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xa5c28be8> (a java.io.PipedInputStream)
at java.io.PipedInputStream.awaitSpace(PipedInputStream.java:257)
at java.io.PipedInputStream.receive(PipedInputStream.java:215)
- locked <0xa5c28be8> (a java.io.PipedInputStream)
at java.io.PipedOutputStream.write(PipedOutputStream.java:132)
at java.util.zip.GZIPOutputStream.finish(GZIPOutputStream.java:95)
at java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:146)
Locked ownable synchronizers:
- None
I am not really sure who is locking it up. Read docs to figure out the locking calls. But could not figure out what is going wrong and how to overcome it.
Working with PipedInputStream and PipedOutputStream must be in separate threads.
Read the Javadoc carefully:
http://docs.oracle.com/javase/6/docs/api/java/io/PipedInputStream.html
Typically, data is read from a PipedInputStream object by one thread and data is written to the corresponding PipedOutputStream by some other thread. Attempting to use both objects from a single thread is not recommended, as it may deadlock the thread.
PipedInputStream has a small non-expanding buffer. Once the buffer is full, writes to the PipedOutputStream block until the buffered input is read by a different thread. You cannot use the two from the same thread, because the write will be waiting for a read that cannot happen.
In your case, you are not reading any of the data until you have written all of it, so the solution is to use a ByteArrayOutputStream and ByteArrayInputStream instead:
Write all the data to a ByteArrayOutputStream.
When finished, call toByteArray() on the stream to retrieve the byte data.
(Optional) Create a ByteArrayInputStream with the byte data to read from it as an InputStream.
I needed a filter to intercept slow connections where I need to close DB connections ASAP so I initially used Java pipes but when looked closer at their implementation, it is all synchronized so I ended up creating my own QueueInputStream using a small buffer and Blocking queue to put the buffer in the queue once was full, it is lock free except when for the lock conditions used at LinkedBlockingQueue which with the aid of the small buffer it should be cheap, this class is only intended to be used for a single producer and consumer per instance:
import java.io.IOException;
import java.io.OutputStream;
import java.util.concurrent.*;
public class QueueOutputStream extends OutputStream
{
private static final int DEFAULT_BUFFER_SIZE=1024;
private static final byte[] END_SIGNAL=new byte[]{};
private final BlockingQueue<byte[]> queue=new LinkedBlockingDeque<>();
private final byte[] buffer;
private boolean closed=false;
private int count=0;
public QueueOutputStream()
{
this(DEFAULT_BUFFER_SIZE);
}
public QueueOutputStream(final int bufferSize)
{
if(bufferSize<=0){
throw new IllegalArgumentException("Buffer size <= 0");
}
this.buffer=new byte[bufferSize];
}
private synchronized void flushBuffer()
{
if(count>0){
final byte[] copy=new byte[count];
System.arraycopy(buffer,0,copy,0,count);
queue.offer(copy);
count=0;
}
}
#Override
public synchronized void write(final int b) throws IOException
{
if(closed){
throw new IllegalStateException("Stream is closed");
}
if(count>=buffer.length){
flushBuffer();
}
buffer[count++]=(byte)b;
}
#Override
public synchronized void write(final byte[] b, final int off, final int len) throws IOException
{
super.write(b,off,len);
}
#Override
public synchronized void close() throws IOException
{
flushBuffer();
queue.offer(END_SIGNAL);
closed=true;
}
public Future<Void> asyncSendToOutputStream(final ExecutorService executor, final OutputStream outputStream)
{
return executor.submit(
new Callable<Void>()
{
#Override
public Void call() throws Exception
{
try{
byte[] buffer=queue.take();
while(buffer!=END_SIGNAL){
outputStream.write(buffer);
buffer=queue.take();
}
outputStream.flush();
} catch(Exception e){
close();
throw e;
} finally{
outputStream.close();
}
return null;
}
}
);
}
Related
I am planning a function that creates and returns an InputStream that in turn reads from another InputStream because the initialization of that InputStream is not trivial and I would like to use it in multiple places. Consider this simple example:
private static InputStream openStream() throws IOException {
Path path = Paths.get("/etc/passwd");
InputStream inputStream = Files.newInputStream(path);
return new BufferedInputStream(inputStream);
}
I will use this function as follows:
public static void main(String[] args) {
try (InputStream stream = openStream()) {
byte[] buffer = new byte[1024];
int numBytes;
while ((numBytes = stream.read(buffer, 0, buffer.length)) > 0) {
System.out.printf("Just read %d bytes from stream!%n", numBytes);
}
} catch (IOException e) {
e.printStackTrace();
}
}
However, I am concerned that closing the BufferedInputStream in this example will not close the InputStream inside it. Will this lead to orphaned file handles and memory leaks if called multiple times? What is a better solution for this?
A simple solution I could think of is to define a closable container class and put both input streams into that class. When calling close(), this class would simply close all its open handles.
class StreamContainer implements Closeable {
private final InputStream[] inputStreams;
public StreamContainer(InputStream... inputStreams) {
this.inputStreams = inputStreams;
}
#Override
public void close() throws IOException {
for (InputStream inputStream : this.inputStreams) {
inputStream.close();
}
}
}
But I suppose, there might be a better solution, built-in mechanic or development pattern. Or maybe these constructs should be avoided?
In this cases you should read the code source of the BufferedInputStream, this is the close definition
public void close() throws IOException {
while(true) {
byte[] buffer;
if ((buffer = this.buf) != null) {
if (!U.compareAndSetObject(this, BUF_OFFSET, buffer, (Object)null)) {
continue;
}
InputStream input = this.in;
this.in = null;
if (input != null) {
input.close();
}
return;
}
return;
}
}
As you can see when closing the BufferedInputStream, the underlying input stream is closed as well.
And this is the documentation of close:
public void close() throws IOException
Closes this input stream and releases any system resources associated with the stream. Once the
stream has been closed, further read(), available(), reset(), or
skip() invocations will throw an IOException. Closing a previously
closed stream has no effect.
I have a data producer that runs in a separate thread and pushes generated data into PipedOutputStream which is connected to PipedInputStream. A reference of this input stream is exposed via public API so that any client can use it. The PipedInputStream contains a limited buffer which, if full, blocks the data producer. Basically, as the client reads data from the input stream, new data is generated by the data producer.
The problem is that the data producer may fail and throw an exception. But as the consumer is running in a separate thread, there is no nice way to get the exception to the client.
What I do is that I catch that exception and close the input stream. That will result in a IOException with message "Pipe closed" on the client side but I would really like to give the client the real reason behind that.
This is a rough code of my API:
public InputStream getData() {
final PipedInputStream inputStream = new PipedInputStream(config.getPipeBufferSize());
final PipedOutputStream outputStream = new PipedOutputStream(inputStream);
Thread thread = new Thread(() -> {
try {
// Start producing the data and push it into output stream.
// The production my fail and throw an Exception with the reason
} catch (Exception e) {
try {
// What to do here?
outputStream.close();
inputStream.close();
} catch (IOException e1) {
}
}
});
thread.start();
return inputStream;
}
I have two ideas how to fix that:
Store the exception in the parent object and expose it to the client via API. I. e. if the reading fails with an IOException, the client could ask the API for the reason.
Extend / re-implement the piped streams so that I could pass a reason to the close() method. Then the IOException thrown by the stream could contain that reason as a message.
Any better ideas?
Coincidentally I just wrote similar code to allow GZip compression of a stream. You don't need to extend PipedInputStream, just FilterInputStream will do and return a wrapped version, e.g.
final PipedInputStream in = new PipedInputStream();
final InputStreamWithFinalExceptionCheck inWithException = new InputStreamWithFinalExceptionCheck(in);
final PipedOutputStream out = new PipedOutputStream(in);
Thread thread = new Thread(() -> {
try {
// Start producing the data and push it into output stream.
// The production my fail and throw an Exception with the reason
} catch (final IOException e) {
inWithException.fail(e);
} finally {
inWithException.countDown();
}
});
thread.start();
return inWithException;
And then InputStreamWithFinalExceptionCheck is just
private static final class InputStreamWithFinalExceptionCheck extends FilterInputStream {
private final AtomicReference<IOException> exception = new AtomicReference<>(null);
private final CountDownLatch complete = new CountDownLatch(1);
public InputStreamWithFinalExceptionCheck(final InputStream stream) {
super(stream);
}
#Override
public void close() throws IOException {
try {
complete.await();
final IOException e = exception.get();
if (e != null) {
throw e;
}
} catch (final InterruptedException e) {
throw new IOException("Interrupted while waiting for synchronised closure");
} finally {
stream.close();
}
}
public void fail(final IOException e) {
exception.set(Preconditions.checkNotNull(e));
}
public void countDown() {complete.countDown();}
}
This is my implementation, taken from above accepted answer https://stackoverflow.com/a/33698661/5165540 , where I don't use the CountDownLatch complete.await() as it would cause a deadlock if the InputStream gets abruptly closed before the writer has finished writing the full content.
I still set the exception caught when PipedOutpuStream is being used, and I create the PipedOutputStream in the spawn thread, using a try-finally-resource pattern to ensure it gets closed, waiting in the Supplier until the 2 streams are piped.
Supplier<InputStream> streamSupplier = new Supplier<InputStream>() {
#Override
public InputStream get() {
final AtomicReference<IOException> osException = new AtomicReference<>();
final CountDownLatch piped = new CountDownLatch(1);
final PipedInputStream is = new PipedInputStream();
FilterInputStream fis = new FilterInputStream(is) {
#Override
public void close() throws IOException {
try {
IOException e = osException.get();
if (e != null) {
//Exception thrown by the write will bubble up to InputStream reader
throw new IOException("IOException in writer", e);
}
} finally {
super.close();
}
};
};
Thread t = new Thread(() -> {
try (PipedOutputStream os = new PipedOutputStream(is)) {
piped.countDown();
writeIozToStream(os, projectFile, dataFolder);
} catch (final IOException e) {
osException.set(e);
}
});
t.start();
try {
piped.await();
} catch (InterruptedException e) {
t.cancel();
Thread.currentThread().interrupt();
}
return fis;
}
};
Calling code is something like
try (InputStream is = streamSupplier.getInputStream()) {
//Read stream in full
}
So when is InputStream is closed this will be signaled in the PipedOutputStream causing eventually a "Pipe closed" IOException, ignored at that point.
If I keep instead the complete.await() line in the FilterInputStreamclose() I could suffer from deadlock (PipedInputStream trying to close, waiting on complete.await(), while PipedOutputStream is waiting forever on PipedInputStreamawaitSpace )
I met a problem with BufferedWriter when I write data to a single file with some threads.
I set the buffer size of the BufferedWriter, but no matter what number I set, it flushes the data to disk when the buffer is 8192 (the default buffer size), not the size I set (here is 16384). Is there a problem with my code?
This is how I'm constructing the BufferedWriter:
new BufferedWriter(new FileWriter(fileName, true), 16384);
This is the full code:
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
public class Test1 {
public static void main(String[] args) throws IOException {
for(int i =0;i<10;i++){
MyThread r = new MyThread();
Thread t = new Thread(r);
t.start();
}
}
}
class MyThread implements Runnable {
public void run() {
String s = "{addffffffkkkljlkj2015dd}\n";
BufferedWriter bw = null;
try {
bw = new BufferedWriter(new FileWriter(
"/Users/liaoliuqing/Downloads/1.txt", true),16384);
} catch (IOException e) {
e.printStackTrace();
}
for(int i =0 ; i<1000; i++){
try {
bw.write(String.format("%03d", i)+s);
//bw.flush();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
Is there a problem with my code?
A few. Mainly: potential IO and concurrency errors. File buffer size might be a lesser concern (and one you can't effectively deal with).
Trying to open already opened file. All your threads are trying to write into the same file (1.txt). That might be an issue. FileWriter documentation says:
Some platforms, in particular, allow a file to be opened for writing by only one FileWriter (or other file-writing object) at a time. In such situations the constructors in this class will fail if the file involved is already open.
Lines might be cut and mixed. If you have several threads with their respective buffers flushing at some point into the same output, you might not even need weird race-conditions or threads stopped right of the middle or a write operation to see your output corrupted.
As I solution (If your threads must share the same output) you can use a shared object with synchronized access to take care of actual writing. I implemented SafeAppender in my example, but probably there are better alternatives out there.
No flushing and closing buffers will mean (the tail of) your data will be lost (like tears in the rain). A finally block is usually good to take care of that.
Also, as stated by other users, BufferedWriter buffer size does not affect the buffer size in FileOutputStream (and so FileWriter). And it looks the java.io and java.nio APIs dont offer any way to mess with that. If you look at the Java library sources you might notice BufferedWriter buffer size just means the amount of chars you store before actually writing into the delegate output. The default size (8192) is optimal for most cases, and increasing it might mean more trouble (potentially losing more data) than benefits.
This is my code, if it serves you:
// http://stackoverflow.com/questions/32451526/how-to-set-the-buffer-size-on-a-bufferedwriter-over-a-filewriter
public class TestWriter {
public static class SafeAppender {
private BufferedWriter bw;
private int users = 0;
public SafeAppender(File f) throws IOException {
bw = new BufferedWriter(new FileWriter(f));
}
public synchronized void append(String s) throws IOException {
bw.write(s);
}
public synchronized void incrUsers() {
users ++;
}
public synchronized void decrUsers() {
if (--users <= 0) {
try {
bw.flush();
System.err.println("INFO-appender-flush()");
} catch (Throwable whatever) { /* log-if-you-care*/}
}
}
// Might be called by GC, or not
#Override protected void finalize() throws Throwable {
try {
bw.close();
System.err.println("INFO-appender-close()");
} catch (Throwable whatever) { /* log-if-you-care */}
super.finalize();
}
}
private static class MyRunnable implements Runnable {
final static String S = "{addffffffkkkljlkj2015dd}";
SafeAppender appender;
String threadId;
public MyRunnable (SafeAppender a, String tid) {
appender = a; threadId = tid;
}
public void run() {
appender.incrUsers();
try {
for(int i =0 ; i<1000; i++){
// NOTE: Not a good idea to printStackTrace if each line fails. Let thread fail
String line = String.format("%s-%03d-%s\n", threadId, i, S);
appender.append(line);
}
} catch (IOException e) {
System.err.printf("ERROR-%s-%s\n", threadId, e.toString());
} finally {
appender.decrUsers();
}
}
}
public static void main(String[] args) {
try {
File f = File.createTempFile("TestWriter", ".txt");
System.err.printf("INFO-main-Writing into %s\n", f.getCanonicalPath());
SafeAppender appender = new SafeAppender (f);
for(int i =0;i<10;i++){
MyRunnable r = new MyRunnable(appender, ""+i);
Thread t = new Thread(r);
t.start();
}
} catch (Throwable e) {
e.printStackTrace(System.err);
}
}
}
FileWriter actually uses its own fixed-size 1024 byte buffer. The BufferedWriter on the other hand, show that it uses and 8192 byte buffer size (default), which can be configured by the user to any other desired size.
And to further muddy the waters, the Java 6 implementation of OutputStreamWriter actually delegates to a StreamEncoder, which uses its own buffer with a default size of 8192 bytes. And the StreamEncoder buffer is user-configurable, although there is no way to access it directly through the enclosing OutputStreamWriter.
I solve the problem by using OutputStream, not writer, here is the code:
bw = new BufferedOutputStream(
new FileOutputStream(new File("/Users/liaoliuqing/Downloads/1.txt"),true),165537);
What you are seeing is not the size of the buffer BufferedWriter, but the size of the buffer used internally by FileWriter. Quoting from the Java Documentation (http://docs.oracle.com/javase/7/docs/api/java/io/FileWriter.html)
The constructors of this class assume that the default character encoding and the default byte-buffer size are acceptable. To specify these values yourself, construct an OutputStreamWriter on a FileOutputStream.
So if you wanted to have a fine grain control on when the data is actually written to the disk you should instantiate your BufferedWriter as
bw = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(new File('my_file.txt),true)));
How to finish the work correctly at the output end of the pipe? I need the writing thread to terminate or do some other work, while the reading thread reads all written data up to end.
Should I close the pipe at the writing end or what?
UPDATE 1
I want to clarify... According to given answers, am I correct thinking that by-design pipes behavior does not suppose any graceful termination?
I.e. once opened, the only way to stop piping is to break the pipe?
Conventional streams expect end of the stream signal, when read() method returns -1. Am right thinking that this never happens with piped streams?
Yes, closing the PipedOutputStream results in a -1 on the PipedInputStream.
Looks pretty graceful to me! Here's my SSCCE:
import java.io.*;
import java.nio.charset.*;
public class SOPipe
{
public static void main(String[] args) throws Exception
{
PipedOutputStream os = new PipedOutputStream();
PipedInputStream is = new PipedInputStream(os);
ReaderThread readerThread = new ReaderThread(is);
WriterThread writerThread = new WriterThread(os);
readerThread.start();
writerThread.start();
readerThread.join();
writerThread.join();
System.out.println("Both Reader and Writer completed.");
System.out.println("Main method returning normally now.");
}
private static final Charset LATIN1 = Charset.forName("latin1");
public static class WriterThread extends Thread
{
private final PipedOutputStream _os;
public WriterThread(PipedOutputStream os)
{
_os = os;
}
public void run()
{
try
{
String msg = "Ceci n'est pas une pipe";
byte[] msgBytes = msg.getBytes(LATIN1);
System.out.println("WriterThread sending message: " + msg);
for(int i = 0; i < msgBytes.length; i++)
{
_os.write(msgBytes, i, 1);
System.out.println("WriterThread wrote a byte!");
_os.flush();
}
_os.close();
System.out.println("[COMPLETED] WriterThread");
}
catch (Exception e)
{
e.printStackTrace();
}
}
}
public static class ReaderThread extends Thread
{
private final PipedInputStream _is;
public ReaderThread(PipedInputStream is)
{
_is = is;
}
public void run()
{
try
{
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buffer = new byte[1];
int read;
while ((read = _is.read(buffer, 0, 1)) != -1)
{
System.out.println("ReaderThread read a byte!");
baos.write(buffer, 0, read);
}
System.out.println("[COMPLETED] ReaderThread; received: "
+ new String(baos.toByteArray(), LATIN1));
_is.close();
}
catch (Exception e)
{
e.printStackTrace();
}
}
}
}
It is enough just to close output stream.
You can extend PipedOutputStream class & override its write() method to add your custom logic, after it has written all the bytes to the piped output stream.
public class CustomPipedOutput extends PipedOutputStream {
#Override
public void write(byte[] byteArray, int offset, int length){
super.write(byteArray, offset, length);
//-- Code to be executed after writing bytes
}
#Override
public void close(){
super.close();
//-- Code to be executed after closing piped input stream
}
}
Similarly, can extend other methods if required accordingly.
I use Runtime exec() method to create a subprocess in Java. However, since the subprocess is an interactive program, I need to provide input to it as and when required by it. Also I need to show the output of the subprocess. How can I do this in the simplest possible way?
I was using a StreamGobbler to show the program output using process.getInputStream(). I, however, do not know how to identify when the program is waiting for input and when to provide it input using proc.getOutputStream. How can I do this?
You need to copy the input and output between the subprocess' streams and System streams (System.in, System.out and System.err). This is related to my recent quesion. The best solution I have found so far is:
import java.io.FileInputStream;
import java.io.FilterInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.lang.reflect.Field;
import java.nio.ByteBuffer;
import java.nio.channels.AsynchronousCloseException;
import java.nio.channels.FileChannel;
class StreamCopier implements Runnable {
private InputStream in;
private OutputStream out;
public StreamCopier(InputStream in, OutputStream out) {
this.in = in;
this.out = out;
}
public void run() {
try {
int n;
byte[] buffer = new byte[4096];
while ((n = in.read(buffer)) != -1) {
out.write(buffer, 0, n);
out.flush();
}
}
catch (IOException e) {
System.out.println(e);
}
}
}
class InputCopier implements Runnable {
private FileChannel in;
private OutputStream out;
public InputCopier(FileChannel in, OutputStream out) {
this.in = in;
this.out = out;
}
public void run() {
try {
int n;
ByteBuffer buffer = ByteBuffer.allocate(4096);
while ((n = in.read(buffer)) != -1) {
out.write(buffer.array(), 0, n);
out.flush();
}
out.close();
}
catch (AsynchronousCloseException e) {}
catch (IOException e) {
System.out.println(e);
}
}
}
public class Test {
private static FileChannel getChannel(InputStream in)
throws NoSuchFieldException, IllegalAccessException {
Field f = FilterInputStream.class.getDeclaredField("in");
f.setAccessible(true);
while (in instanceof FilterInputStream)
in = (InputStream)f.get((FilterInputStream)in);
return ((FileInputStream)in).getChannel();
}
public static void main(String[] args)
throws IOException, InterruptedException,
NoSuchFieldException, IllegalAccessException {
Process process = Runtime.getRuntime().exec("sh -i +m");
Thread outThread = new Thread(new StreamCopier(
process.getInputStream(), System.out));
outThread.start();
Thread errThread = new Thread(new StreamCopier(
process.getErrorStream(), System.err));
errThread.start();
Thread inThread = new Thread(new InputCopier(
getChannel(System.in), process.getOutputStream()));
inThread.start();
process.waitFor();
System.in.close();
outThread.join();
errThread.join();
inThread.join();
}
}
The tricky part here is to extract a channel from System.in. Without this you will not be able to interrupt the thread that reads input when the subprocess terminates.
This approach has a serious drawback: after closing System.in you can no longer read from it. The workaround that I'm currently using is to have a single input redirecting thread used for all subprocesses.
Ask yourself "How do I know when the program wants input when I run it from the command line"? You see what it prompts and enter data based on that prompt. The principle will be the same, except your code will need to interpret the program's output and provide the correct input.
To avoid reinventing the wheel, take a look at ExpectJ and/or Expect4J, which are Java implementations of the venerable *nix Expect tool, which is designed to handle this kind of programmatic interaction.