I have a data producer that runs in a separate thread and pushes generated data into PipedOutputStream which is connected to PipedInputStream. A reference of this input stream is exposed via public API so that any client can use it. The PipedInputStream contains a limited buffer which, if full, blocks the data producer. Basically, as the client reads data from the input stream, new data is generated by the data producer.
The problem is that the data producer may fail and throw an exception. But as the consumer is running in a separate thread, there is no nice way to get the exception to the client.
What I do is that I catch that exception and close the input stream. That will result in a IOException with message "Pipe closed" on the client side but I would really like to give the client the real reason behind that.
This is a rough code of my API:
public InputStream getData() {
final PipedInputStream inputStream = new PipedInputStream(config.getPipeBufferSize());
final PipedOutputStream outputStream = new PipedOutputStream(inputStream);
Thread thread = new Thread(() -> {
try {
// Start producing the data and push it into output stream.
// The production my fail and throw an Exception with the reason
} catch (Exception e) {
try {
// What to do here?
outputStream.close();
inputStream.close();
} catch (IOException e1) {
}
}
});
thread.start();
return inputStream;
}
I have two ideas how to fix that:
Store the exception in the parent object and expose it to the client via API. I. e. if the reading fails with an IOException, the client could ask the API for the reason.
Extend / re-implement the piped streams so that I could pass a reason to the close() method. Then the IOException thrown by the stream could contain that reason as a message.
Any better ideas?
Coincidentally I just wrote similar code to allow GZip compression of a stream. You don't need to extend PipedInputStream, just FilterInputStream will do and return a wrapped version, e.g.
final PipedInputStream in = new PipedInputStream();
final InputStreamWithFinalExceptionCheck inWithException = new InputStreamWithFinalExceptionCheck(in);
final PipedOutputStream out = new PipedOutputStream(in);
Thread thread = new Thread(() -> {
try {
// Start producing the data and push it into output stream.
// The production my fail and throw an Exception with the reason
} catch (final IOException e) {
inWithException.fail(e);
} finally {
inWithException.countDown();
}
});
thread.start();
return inWithException;
And then InputStreamWithFinalExceptionCheck is just
private static final class InputStreamWithFinalExceptionCheck extends FilterInputStream {
private final AtomicReference<IOException> exception = new AtomicReference<>(null);
private final CountDownLatch complete = new CountDownLatch(1);
public InputStreamWithFinalExceptionCheck(final InputStream stream) {
super(stream);
}
#Override
public void close() throws IOException {
try {
complete.await();
final IOException e = exception.get();
if (e != null) {
throw e;
}
} catch (final InterruptedException e) {
throw new IOException("Interrupted while waiting for synchronised closure");
} finally {
stream.close();
}
}
public void fail(final IOException e) {
exception.set(Preconditions.checkNotNull(e));
}
public void countDown() {complete.countDown();}
}
This is my implementation, taken from above accepted answer https://stackoverflow.com/a/33698661/5165540 , where I don't use the CountDownLatch complete.await() as it would cause a deadlock if the InputStream gets abruptly closed before the writer has finished writing the full content.
I still set the exception caught when PipedOutpuStream is being used, and I create the PipedOutputStream in the spawn thread, using a try-finally-resource pattern to ensure it gets closed, waiting in the Supplier until the 2 streams are piped.
Supplier<InputStream> streamSupplier = new Supplier<InputStream>() {
#Override
public InputStream get() {
final AtomicReference<IOException> osException = new AtomicReference<>();
final CountDownLatch piped = new CountDownLatch(1);
final PipedInputStream is = new PipedInputStream();
FilterInputStream fis = new FilterInputStream(is) {
#Override
public void close() throws IOException {
try {
IOException e = osException.get();
if (e != null) {
//Exception thrown by the write will bubble up to InputStream reader
throw new IOException("IOException in writer", e);
}
} finally {
super.close();
}
};
};
Thread t = new Thread(() -> {
try (PipedOutputStream os = new PipedOutputStream(is)) {
piped.countDown();
writeIozToStream(os, projectFile, dataFolder);
} catch (final IOException e) {
osException.set(e);
}
});
t.start();
try {
piped.await();
} catch (InterruptedException e) {
t.cancel();
Thread.currentThread().interrupt();
}
return fis;
}
};
Calling code is something like
try (InputStream is = streamSupplier.getInputStream()) {
//Read stream in full
}
So when is InputStream is closed this will be signaled in the PipedOutputStream causing eventually a "Pipe closed" IOException, ignored at that point.
If I keep instead the complete.await() line in the FilterInputStreamclose() I could suffer from deadlock (PipedInputStream trying to close, waiting on complete.await(), while PipedOutputStream is waiting forever on PipedInputStreamawaitSpace )
Related
I am using piped output streams to convert OutputStream to InputStream because the AWS java sdk does not allow puting objects on S3 using OutputStreams
I'm using the code below, however, this will intermittently just hang. This code is in a web application. Currently there is no load on the application...I am just trying it out on my personal computer.
ByteArrayOutputStream os = new ByteArrayOutputStream();
PipedInputStream inpipe = new PipedInputStream();
final PipedOutputStream out = new PipedOutputStream(inpipe);
try {
String xmpXml = "<dc:description>somedesc</dc:description>"
JpegXmpRewriter rewriter = new JpegXmpRewriter();
rewriter.updateXmpXml(isNew1,os, xmpXml);
new Thread(new Runnable() {
public void run () {
try {
// write the original OutputStream to the PipedOutputStream
println "starting writeto"
os.writeTo(out);
out.close();
println "ending writeto"
} catch (IOException e) {
System.out.println("Some exception)
}
}
}).start();
ObjectMetadata metadata1 = new ObjectMetadata();
metadata1.setContentLength(os.size());
client.putObject(new PutObjectRequest("test-bucket", "167_sample.jpg", inpipe, metadata1));
}
catch (Exception e) {
System.out.println("Some exception")
}
finally {
isNew1.close()
os.close()
}
Instead of bothering with the complexities of starting another thread, instantiating two concurrent classes, and then passing data from thread to thread, all to solve nothing but a minor limitation in the provided JDK API, you should just create a simple specialization of the ByteArrayOutputStream:
class BetterByteArrayOutputStream extends ByteArrayOutputStream {
public ByteArrayInputStream toInputStream() {
return new ByteArrayInputStream(buf, 0, count);
}
}
This converts it to an input stream with no copying.
I'm writing a program that writes to a single file from both different threads on the same JVM and from different JVM's/processes. Is there a way to lock a file for both threads and processes, so that no matter how many threads/processes are trying to write at the same time, only 1 can write at a time?
Currently I have something similar to the following which works for locking threads, but not for blocking processes. If I try using FileLock on top of the implementation below it appears the synchronized stops working.
import org.apache.commons.io.FileUtils;
import org.apache.commons.io.IOUtils;
import scripts.com.adz.commons.utils.FileUtilities;
import java.io.*;
public class Foo {
public static void main(String[] args) throws IOException {
Bar bar = new Bar();
bar.start();
while (true) {
FileUtilities.writeObjectToFile("C:\\test.html", "foo");
}
}
}
class Bar extends Thread {
#Override
public void run() {
while (true) {
try {
FileUtilities.writeObjectToFile("C:\\test.html", "bar");
} catch (IOException ignored) {}
}
}
}
class FileUtilitiess {
private static final Object _lock = new Object();
public static <T> T readObjectFromFile(File file) throws IOException, ClassNotFoundException {
synchronized (_lock) {
final byte[] bytes = FileUtils.readFileToByteArray(file);
ByteArrayInputStream bis = null;
ObjectInputStream ois = null;
try {
ois = new ObjectInputStream(bis = new ByteArrayInputStream(bytes));
return (T) ois.readObject();
} finally {
IOUtils.closeQuietly(ois);
IOUtils.closeQuietly(bis);
}
}
}
public static void writeObjectToFile(File file, Object object) throws IOException {
System.out.println("Sent object: " + object.toString());
synchronized (_lock) {
System.out.println("Writing object: " + object.toString());
ByteArrayOutputStream bos = null;
ObjectOutputStream oos = null;
try {
oos = new ObjectOutputStream(bos = new ByteArrayOutputStream());
oos.writeObject(object);
FileUtils.writeByteArrayToFile(file, bos.toByteArray());
// - Start: For testing lock.
try {
Thread.sleep(10000);
} catch (InterruptedException ignored) {}
// - End: For testing lock.
} finally {
IOUtils.closeQuietly(oos);
IOUtils.closeQuietly(bos);
}
}
}
}
See FileLock javadoc:
File locks are held on behalf of the entire Java virtual machine.
That means that on the OS level different threads of your application will have the same right to access the locked region.
To lock the file access from different threads you have to encapsulate your file IO code and to enforce synchronized execution.
I have a simple Server-Client socket connection. I encapsulate all my data in objects which are sent backward and forward between the sockets, sent through ObjectStreams.
I have created a "HeartBeat" monitor, which runs in a separate thread, where both the server and the client, every 500ms, send a HeartBeat (empty object) backward and forward to check for connectivity, which works great. However, because of this, when I want to send other data between the server and client, it is mixed up with these HeartBeat objects.
For example my Server is expecting a Login object, but instead gets an object of instance HeartBeat.
My code is a simple client/server setup, so I don't think it'd be necessary to post their code, however, the HeartBeat code is as follows:
private static final int HEARTBEAT_INTERVAL = 500;
private void addHeartBeatMonitor(final Socket socket) {
this.heartBeatTimer = new Timer();
this.heartBeatTimer.scheduleAtFixedRate(new TimerTask() {
#Override
public void run() {
try {
ObjectOutputStream os = new ObjectOutputStream(socket.getOutputStream());
os.writeObject(new HeartBeat());
ObjectInputStream is = new ObjectInputStream(socket.getInputStream());
if (!(is.readObject() instanceof HeartBeat)) { throw new IOException(); }
} catch (IOException e) {
LOG.info("Received disconnect from " + getClientSocket().getInetAddress());
heartBeatTimer.cancel();
if (clientSocket != null) {
try {
clientSocket.close();
} catch (IOException e1) {}
}
} catch (ClassNotFoundException e) {}
}
}, 0, HEARTBEAT_INTERVAL);
}
My options seem to be to as follows:
Ditch the HeartBeat functionality, although there seems to be no other reliable way to check the connection status.
Find some other kind of Socket implementation which will magically fix all of this for me.
Have a synchronized method which oversees all reads and writes to the socket, which discards HeartBeats and sends other objects where they're meant to be.
Some kind of synchronization magic.
Thanks in advance for any help!
EDIT:
Code which reads the Login object (server side):
User result = null;
try {
ObjectInputStream is = new ObjectInputStream(this.getInputStream());
Login request = (Login) is.readObject(); ### ERROR ###
result = this.mongoService.login(request);
ObjectOutputStream os = new ObjectOutputStream(this.getOutputStream());
os.writeObject(result);
} catch (IOException e) {
} catch (ClassNotFoundException e) {}
return result;
Exception as follows:
Exception in thread "Thread-0" java.lang.ClassCastException: model.HeartBeat cannot be cast to model.Login
at socket.SocketServerWorker.login(SocketServerWorker.java:78)
at socket.SocketServerWorker.<init>(SocketServerWorker.java:47)
at socket.SocketServer$2.run(SocketServer.java:50)
at java.lang.Thread.run(Thread.java:744)
Consider doing something like this. I just threw this together, so it's obviously untested, but I'm sure you'll get the idea:
public class HeartBeatMonitor
{
final Map<Class,Consumer> handlers = new HashMap<> ();
final Socket sock;
final ObjectInputStream is;
final ObjectOutputStream os;
public HeartBeatMonitor (final Socket sock)
{
try
{
this.sock = sock;
this.is = new ObjectInputStream (sock.getInputStream ());
this.os = new ObjectOutputStream (sock.getOutputStream ());
}
catch (final IOException e)
{
throw new RuntimeException (e);
}
}
public <T> void setHandler (final Class<T> type, final Consumer<? super T> handler)
{
this.handlers.put (type, handler);
}
// This would be called in a loop
void accept () throws ClassNotFoundException, IOException
{
final Object o = this.is.readObject ();
final Consumer handler = this.handlers.get (o.getClass ());
if (handler != null)
handler.accept (o);
// Else default handler?
}
}
While running debugger, the program pauses on initializing object streams from server main input output streams. Following is the code :
public TFileReader(Client cli)throws Exception{
this.cli = cli;
fileSock = new Socket(cli.ServerIp(), cli.FilePort());
fobjIn = new ObjectInputStream(fileSock.getInputStream());
fobjOut = new ObjectOutputStream(fileSock.getOutputStream());
fobjOut.flush();
}
#Override
public void run(){
try{
System.out.println("file reader thread online");
fobjOut.writeObject(cli.Name());
fobjOut.flush();
String per = (String) fobjIn.readObject();
System.out.println(per+"video filing...");
if(!per.equals("OKF"))
{
throw new Exception("Error In retriving video.");
}
It pauses on fobjIn and do not go to execute fobjOut although fobjIn it passes from fobjIn breakpoint but do not hit out breakpoint.
I would keep it simple like this
public TFileReader(Client cli) throws IOException {
this.cli = cli;
socket = new Socket(cli.ServerIp(), cli.FilePort());
out = new ObjectOutputStream(socket.getOutputStream());
out.flush();
in = new ObjectInputStream(socket.getInputStream());
}
public void writeObject(Object o) throw IOException {
out.writeObject(o);
out.reset();
out.flush();
}
public <T> T readObject() throw IOException {
return (T) in.readObject();
}
public void close() throws IOException {
in.close();
out.close();
socket.close();
}
The problem is that ObjectInputStream pre-reads data on initialization.
The preferred solution is at Java Creating a new ObjectInputStream Blocks: always initialize your ObjectOutputStream before initializing your ObjectInputStream, so that the "handshake" that the two use internally can be initiated.
When you don't control all the code and cannot change the order, consider delaying the OIS initialization until data is available (InputStream.available or mark/read/reset on a buffered stream wrapping it, etc).
I have written a Bluetooth API for connecting with an external accessory.
The way that the API is designed is that there are a bunch of blocking calls such as getTime, setTime, getVolume, setVolume, etc.
The way these work is that they create a payload to send and call a method called sendAndReceive() which does some prep work and eventually does the following:
byte[] retVal = null;
BluetoothSocket socket = getSocket();
// write
socket.getOutputStream().write(payload);
// read response
if(responseExpected){
byte[] buffer = new byte[1024]; // buffer store for the stream
int readbytes = socket.getInputStream().read(buffer);
retVal = new byte[readbytes];
System.arraycopy(buffer, 0, retVal, 0, readbytes);
}
return retVal;
The problem is that sometimes this device becomes slow or non-responsive so I would like to put a timeout on this call.
I have tried several methods of putting this code in a thread\future task and running it with a timeout, for example:
FutureTask<byte[]> theTask = null;
// create new task
theTask = new FutureTask<byte[]>(
new Callable<byte[]>() {
#Override
public byte[] call() {
byte[] retVal = null;
BluetoothSocket socket = getSocket();
// write
socket.getOutputStream().write(payload);
// read response
if(responseExpected){
byte[] buffer = new byte[1024]; // buffer store for the stream
int readbytes = socket.getInputStream().read(buffer);
retVal = new byte[readbytes];
System.arraycopy(buffer, 0, retVal, 0, readbytes);
}
return retVal;
}
});
// start task in a new thread
new Thread(theTask).start();
// wait for the execution to finish, timeout after 6 secs
byte[] response;
try {
response = theTask.get(6L, TimeUnit.SECONDS);
} catch (InterruptedException e) {
throw new CbtException(e);
} catch (ExecutionException e) {
throw new CbtException(e);
} catch (TimeoutException e) {
throw new CbtCallTimedOutException(e);
}
return response;
}
The problem with this approach is that I can't re-throw exceptions in the call method and since some of the methods throw exceptions I want to forward back to the API's client I can't use this methodology.
Can you recommend some other alternative?
Thanks!
You're saving you can't use the Future<> method because you want to re-throw the exception but in fact this is possible.
Most examples online do implement Callable with the prototype public ? call() but just change it to public ? call() throws Exception and all will be fine: you'll get the exception in the theTask.get() call and you can rethrow it to callers.
I have personally used Executors exactly for bluetooth socket timeout handling on android:
protected static String readAnswer(...)
throws Exception {
String timeoutMessage = "timeout";
ExecutorService executor = Executors.newCachedThreadPool();
Callable<String> task = new Callable<String>() {
public String call() throws Exception {
return readAnswerNoTimeout(...);
}
};
Future<String> future = executor.submit(task);
try {
return future.get(SOCKET_TIMEOUT_MS, TimeUnit.MILLISECONDS);
} catch (TimeoutException ex) {
future.cancel(true);
throw new Exception(timeoutMessage);
}
}
Why not try something like
public class ReadTask extends Thread {
private byte[] mResultBuffer;
private Exception mCaught;
private Thread mWatcher;
public ReadTask(Thread watcher) {
mWatcher = watcher;
}
public void run() {
try {
mResultBuffer = sendAndReceive();
} catch (Exception e) {
mCaught = e;
}
mWatcher.interrupt();
}
public Exception getCaughtException() {
return mCaught;
}
public byte[] getResults() {
return mResultBuffer;
}
}
public byte[] wrappedSendAndReceive() {
byte[] data = new byte[1024];
ReadTask worker = new ReadTask(data, Thread.currentThread());
try {
worker.start();
Thread.sleep(6000);
} catch (InterruptedException e) {
// either the read completed, or we were interrupted for another reason
if (worker.getCaughtException() != null) {
throw worker.getCaughtException();
}
}
// try to interrupt the reader
worker.interrupt();
return worker.getResults;
}
There is an edge case here that the Thread calling wrappedSendAndReceive() may get interrupted for some reason other than the interrupt from the ReadTask. I suppose a done bit could be added to the ReadTask to allow the other thread to test if the read finished or the interrupt was caused by something else, but I'm not sure how necessary this is.
A further note is that this code does contain the possibility for data loss. If the 6 seconds expires and some amount of data has been read this will end up being discarded. If you wanted to work around this, you'd need to read one byte at a time in ReadTask.run() and then appropriately catch the InterruptedException. This obviously requires a little rework of the existing code to keep a counter and appropriately resize the read buffer when the interrupt is received.