Goal:
I am looking for a bounded byte stream like implementation in java to have "write" and "read" functionality.
The write should be non-blocking and should drop if the capacity has been reached.
Ideally, the max size of the stream should be given in the beginning so that dynamic memory allocation overhead is not present.
What I have tried:
I have shared data structure using the following buffer class, shared between the producer and consumer:
public class Buffer<T> {
private ArrayBlockingQueue<T> buffer;
public Buffer(int maxSize) {
buffer = new ArrayBlockingQueue<T>(maxSize);
}
public boolean offer(T o) {
return buffer.offer(o);
}
public T take(){
try {
return buffer.take();
} catch (InterruptedException e) {
e.printStackTrace();
}
return null;
}
public static void main(String[] args){
Buffer<String> buffer = new Buffer<String>(10);
buffer.offer("hello");
System.out.println(buffer.take());
}
}
A Producer in Thread 1 which reads from a socket and sends data as follows:
private InputStream in;
while ( (bytesRead = in.read(request)) != -1) {
this.register.buffer.offer(new ByteObject(request,bytesRead)
}
A Consumer in Thread 2 which reads from the buffer and writes to a socket:
private OutputStream out;
while(true){
ByteObject poll = register.buffer.take();
bytesRead = poll.len;
if(bytesRead>0){
out.write(poll.arr,0,bytesRead);
out.flush();
}else
break;
}
Problem with current implementation:
Since it is not a stream implementation, I am unable to detect EOF in the consumer. Can anyone give an example for Bounded Stream operations in java which give a non-blocking write option (like offer() functionality)
Alternatively, how would I get EOF in the current code?
Related
I have a class that is constantly producing data and writing it to a ByteArrayOutputStream on its own thread. I have a 2nd thread that gets a reference to this ByteArrayOutputStream. I want the 2nd thread to read any data (and empty) the ByteArrayOutputStream and then stop when it doesn't get any bytes and sleep. After the sleep, I want it to try to get more data and empty it again.
The examples I see online say to use PipedOutputStream. If my first thread is making the ByteArrayOutputStream available to the outside world from a separate reusable library, I don't see how to hook up the inputStream to it.
How would one setup the PipedInputStream to connect it to the ByteArrayOutputStream to read from it as above? Also, when reading the last block from the ByteArrayOutputStream, will I see bytesRead == -1, indicating when the outputStream is closed from the first thread?
Many thanks,
Mike
Write to the PipedOutputStream directly (that is, don't use a ByteArrayOutputStream at all). They both extend OutputStream and so have the same interface.
There are connect methods in both PipedOutputStream and PipedInputStream that are used to wire two pipes together, or you can use one of the constructors to create a pair.
Writes to the PipedOutputStream will block when the buffer in the PipedInputStream fills up, and reads from the PipedInputStream will block when the buffer is empty, so the producer thread will sleep (block) if it gets "ahead" of the consumer and vice versa.
After blocking the threads wait for 1000ms before rechecking the buffer, so it's good practice to flush the output after writes complete (this will wake the reader if it is sleeping).
Your input stream will see the EOF (bytesRead == -1) when you close the output stream in the producer thread.
import java.io.*;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public class PipeTest {
public static void main(String[] args) throws IOException {
PipedOutputStream out = new PipedOutputStream();
// Wire an input stream to the output stream, and use a buffer of 2048 bytes
PipedInputStream in = new PipedInputStream(out, 2048);
ExecutorService executor = Executors.newCachedThreadPool();
// Producer thread.
executor.execute(() -> {
try {
for (int i = 0; i < 10240; i++) {
out.write(0);
// flush to wake the reader
out.flush();
}
out.close();
} catch (IOException e) {
throw new UncheckedIOException(e);
}
});
// Consumer thread.
executor.execute(() -> {
try {
int b, read = 0;
while ((b = in.read()) != -1) {
read++;
}
System.out.println("Read " + read + " bytes.");
} catch (IOException e) {
throw new UncheckedIOException(e);
}
});
executor.shutdown();
}
}
Info
I'm trying to find a way to read blocks of data from an incoming socket stream at a set interval, but ignoring the rest of the data and not closing the connection between reads. I was wondering if anyone had some advice?
The reason I ask is I have been given a network connected analogue to digital converter (ADC) and I want to write a simple oscilloscope application.
Basically once I connect to the ADC and send a few initialisation commands it then takes a few minutes to stabilise, at which point it starts throwing out measurements in a byte stream.
I want to read 1MB of data every few seconds and discard the rest, if I don't discard the rest the ADC will buffer 512kB of readings then pause so any subsequent reads will be of old data. If I close the connection between reads the ADC then takes a while before it sends data again.
Problem
I wrote a simple Python script as a test, in this I used a continuously running thread which would read bytes to an unused buffer, if a flag was set, which seems to work fine.
When I tried this on Android I ran into problems as it seems that only some of the data is being discarded, the ADC still pauses if the update interval is too long.
Where have I made the mistake(s)? My first guess is synchronisation as I'm not sure its working as intended (see the ThreadBucket class). I'll have to admit spending many hours on playing with this, trying different sync permutations, buffer sizes, BufferedInputStream and NIO, but with no luck.
Any input on this would be appreciated, I'm not sure if using a thread like this is the right way to go in Java.
Code
The Reader class sets up the thread, connects to the ADC, reads data on request and in between activates the bit bucket thread (I've omitted the initialisation and closing for clarity).
class Reader {
private static final int READ_SIZE = 1024 * 1024;
private String mServer;
private int mPort;
private Socket mSocket;
private InputStream mIn;
private ThreadBucket mThreadBucket;
private byte[] mData = new byte[1];
private final byte[] mBuffer = new byte[READ_SIZE];
Reader(String server, int port) {
mServer = server;
mPort = port;
}
void setup() throws IOException {
mSocket = new Socket(mServer, mPort);
mIn = mSocket.getInputStream();
mThreadBucket = new ThreadBucket(mIn);
mThreadBucket.start();
// Omitted: Send a few init commands a look at the response
// Start discarding data
mThreadBucket.bucket(true);
}
private int readRaw(int samples) throws IOException {
int current = 0;
// Probably fixed size but may change
if (mData.length != samples)
mData = new byte[samples];
// Stop discarding data
mThreadBucket.bucket(false);
// Read in number of samples to mData
while (current < samples) {
int len = mIn.read(mBuffer);
if (current > samples)
current = samples;
if (current + len > samples)
len = samples - current;
System.arraycopy(mBuffer, 0, mData, current, len);
current += mBuffer.length;
}
// Discard data again until the next read
mThreadBucket.bucket(true);
return current;
}
}
The ThreadBucket class runs continuously, on slurping data to the bit bucket if mBucket is true.
The synchronisation is meant to stop either thread from reading data whilst the other one is.
public class ThreadBucket extends Thread {
private static final int BUFFER_SIZE = 1024;
private final InputStream mIn;
private Boolean mBucket = false;
private boolean mCancel = false;
public ThreadBucket(final InputStream in) throws IOException {
mIn = in;
}
#Override
public void run() {
while (!mCancel && !Thread.currentThread().isInterrupted()) {
synchronized (this) {
if (mBucket)
try {
mIn.skip(BUFFER_SIZE);
} catch (final IOException e) {
break;
}
}
}
}
public synchronized void bucket(final boolean on) {
mBucket = on;
}
public void cancel() {
mCancel = true;
}
}
Thank you.
You need to read continuously, period, as fast as you can code it, and then manage what you do with the data separately. Don't mix the two up.
Is there an implemetation of GZIPOutputStream that would do the heavy lifting (compressing + writing to disk) in a separate thread?
We are continuously writing huge amounts of GZIP-compressed data. I am looking for a drop-in replacement that could be used instead of GZIPOutputStream.
You can write to a PipedOutputStream and have a thread which reads the PipedInputStream and copies it to any stream you like.
This is a generic implementation. You give it an OutputStream to write to and it returns an OutputStream for you to write to.
public static OutputStream asyncOutputStream(final OutputStream out) throws IOException {
PipedOutputStream pos = new PipedOutputStream();
final PipedInputStream pis = new PipedInputStream(pos);
new Thread(new Runnable() {
#Override
public void run() {
try {
byte[] bytes = new byte[8192];
for(int len; (len = pis.read(bytes)) > 0;)
out.write(bytes, 0, len);
} catch(IOException ioe) {
ioe.printStackTrace();
} finally {
close(pis);
close(out);
}
}
}, "async-output-stream").start();
return pos;
}
static void close(Closeable closeable) {
if (closeable != null) try {
closeable.close();
} catch (IOException ignored) {
}
}
I published some code that does exactly what you are looking for. It has always frustrated me that Java doesn't automatically pipeline calls like this across multiple threads, in order to overlap computation, compression, and disk I/O:
https://github.com/lukehutch/PipelinedOutputStream
This class splits writing to an OutputStream into separate producer and consumer threads (actually, starts a new thread for the consumer), and inserts a blocking bounded buffer between them. There is some data copying between buffers, but this is done as efficiently as possible.
You can even layer this twice to do the disk writing in a separate thread from the gzip compression, as shown in README.md.
How do you design a read and write loop which operates on a single socket (which supports parallel read and write operations)? Do I have to use multiple threads? Is my (java) solution any good? What about that sleep command? How do you use that within such a loop?
I'm trying to use 2 Threads:
Read
public void run() {
InputStream clientInput;
ByteArrayOutputStream byteBuffer;
BufferedInputStream bufferedInputStream;
byte[] data;
String dataString;
int lastByte;
try {
clientInput = clientSocket.getInputStream();
byteBuffer = new ByteArrayOutputStream();
bufferedInputStream = new BufferedInputStream(clientInput);
while(isRunning) {
while ((lastByte = bufferedInputStream.read()) > 0) {
byteBuffer.write(lastByte);
}
data = byteBuffer.toByteArray();
dataString = new String(data);
byteBuffer.reset();
}
} catch (IOException e) {
e.printStackTrace();
}
}
Write
public void run() {
OutputStream clientOutput;
byte[] data;
String dataString;
try {
clientOutput = clientSocket.getOutputStream();
while(isOpen) {
if(!commandQueue.isEmpty()) {
dataString = commandQueue.poll();
data = dataString.getBytes();
clientOutput.write(data);
}
Thread.sleep(1000);
}
clientOutput.close();
}
catch (IOException e) {
e.printStackTrace();
}
catch (InterruptedException e) {
e.printStackTrace();
}
}
Read fails to deliver a proper result, since there is no -1 sent.
How do I solve this issue?
Is this sleep / write loop a good solution?
There are basically three ways to do network I/O:
Blocking. In this mode reads and writes will block until they can be fulfilled, so if you want to do both simultaneously you need separate threads for each.
Non-blocking. In this mode reads and writes will return zero (Java) or in some languages (C) a status indication (return == -1, errno=EAGAIN/EWOULDBLOCK) when they cannot be fulfilled, so you don't need separate threads, but you do need a third API that tells you when the operations can be fulfilled. This is the purpose of the select() API.
Asynchronous I/O, in which you schedule the transfer and are given back some kind of a handle via which you can interrogate the status of the transfer, or, in more advanced APIs, a callback.
You should certainly never use the while (in.available() > 0)/sleep() style you are using here. InputStream.available() has few correct uses and this isn't one of them, and the sleep is literally a waste of time. The data can arrive within the sleep time, and a normal read() would wake up immediately.
You should rather use a boolean variable instead of while(true) to properly close your thread when you will want to. Also yes, you should create multiple thread, one per client connected, as the thread will block itself until a new data is received (with DataInputStream().read() for example). And no, this is not really a design question, each library/Framework or languages have its own way to listen from a socket, for example to listen from a socket in Qt you should use what is called "signals and slots", not an infinite loop.
Basically, I have a URL that streams xml updates from a chat room when new messages are posted. I'd like to turn that URL into an InputStream and continue reading from it as long as the connection is maintained and as long as I haven't sent a Thread.interrupt(). The problem I'm experiencing is that BufferedReader.ready() doesn't seem to become true when there is content to be read from the stream.
I'm using the following code:
BufferedReader buf = new BufferedReader(new InputStreamReader(ins));
String str = "";
while(Thread.interrupted() != true)
{
connected = true;
debug("Listening...");
if(buf.ready())
{
debug("Something to be read.");
if ((str = buf.readLine()) != null) {
// str is one line of text; readLine() strips the newline character(s)
urlContents += String.format("%s%n", str);
urlContents = filter(urlContents);
}
}
// Give the system a chance to buffer or interrupt.
try{Thread.sleep(1000);} catch(Exception ee) {debug("Caught thread exception.");}
}
When I run the code, and post something to the chat room, buf.ready() never becomes true, resulting in the lines never being read. However, if I skip the "buf.ready()" part and just read lines directly, it blocks further action until lines are read.
How do I either a) get buf.ready() to return true, or b) do this in such a way as to prevent blocking?
Thanks in advance,
James
How to create a Java non-blocking InputStream
You can't. Your question embodies a contradiciton in terms. Streams in Java are blocking. There is therefore no such thing as a 'non-blocking InputStream'.
Reader.ready() returns true when data can be read without blocking. Period. InputStreams and Readers are blocking. Period. Everything here is working as designed. If you want more concurrency with these APIs you will have to use multiple threads. Or Socket.setSoTimeout() and its near relation in HttpURLConnection.
For nonblocking IO don't use InputStream and Reader (or OutputStream/Writer), but use the java.nio.* classes, in this case a SocketChannel (and additional a CharsetDecoder).
Edit: as an answer to your comment:
Specifically looking for how to create a socket channel to an https url.
Sockets (and also SocketChannels) work on the transport layer (TCP), one (or two) level(s) below application layer protocols like HTTP. So you can't create a socket channel to an https url.
You would instead have to open a Socket-Channel to the right server and the right port (443 if nothing else given in the URI), create an SSLEngine (in javax.net.ssl) in client mode, then read data from the channel, feeding it to the SSL engine and the other way around, and send/get the right HTTP protocol lines to/from your SSLEngine, always checking the return values to know how many bytes were in fact processed and what would be the next step to take.
This is quite complicated (I did it once), and you don't really want to do this if you are not implementing a server with lots of clients connected at the same time (where you can't have a single thread for each connection). Instead, stay with your blocking InputStream which reads from your URLConnection, and put it simply in a spare thread which does not hinder the rest of your application.
You can use the Java NIO library which provides non-blocking I/O capabilities. Take a look at this article for details and sample code: http://www.drdobbs.com/java/184406242.
There is no HTTP/HTTPS implementation using Channels. There is no way to read the inputstream from a httpurlconnaction in a non-blocking way. You either have to use a third party lib or implement http over SocketChannel yourself.
import java.io.InputStream;
import java.util.Arrays;
/**
* This code demonstrates non blocking read from standard input using separate
* thread for reading.
*/
public class NonBlockingRead {
// Holder for temporary store of read(InputStream is) value
private static String threadValue = "";
public static void main(String[] args) throws InterruptedException {
NonBlockingRead test = new NonBlockingRead();
while (true) {
String tmp = test.read(System.in, 100);
if (tmp.length() > 0)
System.out.println(tmp);
Thread.sleep(1000);
}
}
/**
* Non blocking read from input stream using controlled thread
*
* #param is
* — InputStream to read
* #param timeout
* — timeout, should not be less that 10
* #return
*/
String read(final InputStream is, int timeout) {
// Start reading bytes from stream in separate thread
Thread thread = new Thread() {
public void run() {
byte[] buffer = new byte[1024]; // read buffer
byte[] readBytes = new byte[0]; // holder of actually read bytes
try {
Thread.sleep(5);
// Read available bytes from stream
int size = is.read(buffer);
if (size > 0)
readBytes = Arrays.copyOf(buffer, size);
// and save read value in static variable
setValue(new String(readBytes, "UTF-8"));
} catch (Exception e) {
System.err.println("Error reading input stream\nStack trace:\n" + e.getStackTrace());
}
}
};
thread.start(); // Start thread
try {
thread.join(timeout); // and join it with specified timeout
} catch (InterruptedException e) {
System.err.println("Data were note read in " + timeout + " ms");
}
return getValue();
}
private synchronized void setValue(String value) {
threadValue = value;
}
private synchronized String getValue() {
String tmp = new String(threadValue);
setValue("");
return tmp;
}
}