Basically, I have a URL that streams xml updates from a chat room when new messages are posted. I'd like to turn that URL into an InputStream and continue reading from it as long as the connection is maintained and as long as I haven't sent a Thread.interrupt(). The problem I'm experiencing is that BufferedReader.ready() doesn't seem to become true when there is content to be read from the stream.
I'm using the following code:
BufferedReader buf = new BufferedReader(new InputStreamReader(ins));
String str = "";
while(Thread.interrupted() != true)
{
connected = true;
debug("Listening...");
if(buf.ready())
{
debug("Something to be read.");
if ((str = buf.readLine()) != null) {
// str is one line of text; readLine() strips the newline character(s)
urlContents += String.format("%s%n", str);
urlContents = filter(urlContents);
}
}
// Give the system a chance to buffer or interrupt.
try{Thread.sleep(1000);} catch(Exception ee) {debug("Caught thread exception.");}
}
When I run the code, and post something to the chat room, buf.ready() never becomes true, resulting in the lines never being read. However, if I skip the "buf.ready()" part and just read lines directly, it blocks further action until lines are read.
How do I either a) get buf.ready() to return true, or b) do this in such a way as to prevent blocking?
Thanks in advance,
James
How to create a Java non-blocking InputStream
You can't. Your question embodies a contradiciton in terms. Streams in Java are blocking. There is therefore no such thing as a 'non-blocking InputStream'.
Reader.ready() returns true when data can be read without blocking. Period. InputStreams and Readers are blocking. Period. Everything here is working as designed. If you want more concurrency with these APIs you will have to use multiple threads. Or Socket.setSoTimeout() and its near relation in HttpURLConnection.
For nonblocking IO don't use InputStream and Reader (or OutputStream/Writer), but use the java.nio.* classes, in this case a SocketChannel (and additional a CharsetDecoder).
Edit: as an answer to your comment:
Specifically looking for how to create a socket channel to an https url.
Sockets (and also SocketChannels) work on the transport layer (TCP), one (or two) level(s) below application layer protocols like HTTP. So you can't create a socket channel to an https url.
You would instead have to open a Socket-Channel to the right server and the right port (443 if nothing else given in the URI), create an SSLEngine (in javax.net.ssl) in client mode, then read data from the channel, feeding it to the SSL engine and the other way around, and send/get the right HTTP protocol lines to/from your SSLEngine, always checking the return values to know how many bytes were in fact processed and what would be the next step to take.
This is quite complicated (I did it once), and you don't really want to do this if you are not implementing a server with lots of clients connected at the same time (where you can't have a single thread for each connection). Instead, stay with your blocking InputStream which reads from your URLConnection, and put it simply in a spare thread which does not hinder the rest of your application.
You can use the Java NIO library which provides non-blocking I/O capabilities. Take a look at this article for details and sample code: http://www.drdobbs.com/java/184406242.
There is no HTTP/HTTPS implementation using Channels. There is no way to read the inputstream from a httpurlconnaction in a non-blocking way. You either have to use a third party lib or implement http over SocketChannel yourself.
import java.io.InputStream;
import java.util.Arrays;
/**
* This code demonstrates non blocking read from standard input using separate
* thread for reading.
*/
public class NonBlockingRead {
// Holder for temporary store of read(InputStream is) value
private static String threadValue = "";
public static void main(String[] args) throws InterruptedException {
NonBlockingRead test = new NonBlockingRead();
while (true) {
String tmp = test.read(System.in, 100);
if (tmp.length() > 0)
System.out.println(tmp);
Thread.sleep(1000);
}
}
/**
* Non blocking read from input stream using controlled thread
*
* #param is
* — InputStream to read
* #param timeout
* — timeout, should not be less that 10
* #return
*/
String read(final InputStream is, int timeout) {
// Start reading bytes from stream in separate thread
Thread thread = new Thread() {
public void run() {
byte[] buffer = new byte[1024]; // read buffer
byte[] readBytes = new byte[0]; // holder of actually read bytes
try {
Thread.sleep(5);
// Read available bytes from stream
int size = is.read(buffer);
if (size > 0)
readBytes = Arrays.copyOf(buffer, size);
// and save read value in static variable
setValue(new String(readBytes, "UTF-8"));
} catch (Exception e) {
System.err.println("Error reading input stream\nStack trace:\n" + e.getStackTrace());
}
}
};
thread.start(); // Start thread
try {
thread.join(timeout); // and join it with specified timeout
} catch (InterruptedException e) {
System.err.println("Data were note read in " + timeout + " ms");
}
return getValue();
}
private synchronized void setValue(String value) {
threadValue = value;
}
private synchronized String getValue() {
String tmp = new String(threadValue);
setValue("");
return tmp;
}
}
Related
I can't manage to get all data from byteBuffer.
I have to methods as follows:
Client side:
public String sendMessage(String msg) {
buffer = ByteBuffer.wrap(msg.getBytes());
String response = null;
try {
client.write(buffer);
buffer.clear();
client.read(buffer);
response = new String(buffer.array()).trim();
System.out.println("response=" + response);
buffer.clear();
} catch (IOException e) {
e.printStackTrace();
}
return response;
}
Server side (There's another method which accepts clients and work with selection keys, I leave it out):
private static void serverResponse(ByteBuffer buffer, SelectionKey key) throws IOException {
SocketChannel client = (SocketChannel) key.channel();
client.read(buffer);
if (new String(buffer.array()).trim().equals("exit")) {
client.close();
}
else {
ByteBuffer responseBuffer = ByteBuffer.wrap("Example message".getBytes());
client.write(responseBuffer);
responseBuffer.clear();
}
}
When I call sendMessage() and get data from server within this method, I only receive a small piece of data (e.g. I get only "Examp" from original string "Example message"). And only when I call sendMessage() again I receive the rest of the line (Also separately, I need to call sendMessage() a couple of times). Once I reached the end of the line, it started looping and the next calling sendMessage() returns start of the line. How can I get the full data at once?
I'm pretty sure the question has been answered, but I didn't find the solution myself. Please, help me by giving either the answer or a link to a related question
Note: I noticed that I only get as many characters from buffer as I sent to. So, I believe that the problem is in buffer capacity.
Two issues:
After calling write you should call flush (in both cases).
When you are reading the server's response (on the client side), you are using the same buffer which may not be big enough to hold the whole answer. The buffer size you are using is the size of the initial buffer you created in this line:
ByteBuffer.wrap(msg.getBytes());
Try to improve it with those suggestions, and see what happen.
I have a piece of code
...
InputStream inputStream = new BufferedInputStream(new ByteArrayInputStream("test".getBytes()));
...
and this line makes string "test" an input for an InputStream, however this is a static InputStream.
is there any way without a Scanner, System.in or user external input to make this InputStream dynamic
what I need is something like this
...
InputStream inputStream = new BufferedInputStream(new
ByteArrayInputStream(generateContinuousDynamicString().getBytes()));
// So, basically input stream will be blocked until generateContinuousDynamicString()
// returns a result?
...
I've tried something like this
private static byte[] generateContinuousDynamicString(String s) {
String t = "";
// here comes the realization
// that the source for an input stream
// cannot be generated dynamically on the
// fly it only can be read from already
// existing (fully generated and available
// resource). Am I right? Otherwise how
// can I adjust this method in such a way that
// input stream would continuously have a new
// string to read from?
for (int i = 0; i < 1000; i++){
t += "<str>"+s+i+"</str>";
}
return ("<test>"+t+"</test>").getBytes();
}
So, if we have
...
InputStream inputStream = new BufferedInputStream(readFromADatabaseStream());
...
This is also not dynamic input stream as a resource is already in a database.
You want a pipe. Specifically, you want one of the following pairs of classes:
PipedInputStream and PipedOutputStream
PipedReader and PipedWriter
Your question asks for an InputStream, but since you’re dealing with text, you probably should use a Reader, which is intended for characters. In particular, note that getBytes() will return different values on Windows systems compared to non-Windows systems, for any String with non-ASCII characters. Using a Reader and Writer will remove the need to worry about that.
Either way, the approach is the same: create the readable end of the pipe, then create and feed the writable end of the pipe in another thread.
Using a PipedReader and PipedWriter:
PipedReader pipedReader = new PipedReader();
Reader reader = new BufferedReader(pipedReader);
ExecutorService executor = Executors.newSingleThreadExecutor();
Future<?> pipeFeeder = executor.submit(
() -> generateContinuousDynamicString(pipedReader));
// ...
private Void generateContinuousDynamicString(PipedReader pipedReader)
throws IOException {
try (Writer writer = new PipedWriter(pipedReader)) {
writer.write("<test>");
for (int i = 0; i < 1000; i++) {
writer.write("<str>" + i + "</str>");
}
writer.write("</test>");
}
return null;
}
Using a PipedInputStream and PipedOutputStream:
PipedInputStream pipedInputStream = new PipedInputStream();
InputStream inputStream = new BufferedInputStream(pipedInputStream);
ExecutorService executor = Executors.newSingleThreadExecutor();
Future<?> pipeFeeder = executor.submit(
() -> generateContinuousDynamicString(pipedInputStream));
// ...
private Void generateContinuousDynamicString(PipedInputStream pipedInputStream)
throws IOException {
Charset charset = StandardCharsets.UTF_8;
try (Writer writer = new OutputStreamWriter(
new PipedInputStream(pipedinputStream),
StandardCharsets.UTF_8)) {
writer.write("<test>");
for (int i = 0; i < 1000; i++) {
writer.write("<str>" + i + "</str>");
}
writer.write("</test>");
}
return null;
}
Sure. But you have a bit of an issue: Whatever code is generating the endless stream of dynamic data cannot just be in the method that 'returns the inputstream' just by itself, that's what your realisation is about.
You have two major options:
Threads
Instead, you could fire off a thread which is continually generating data. Note that whatever it 'generates' needs to be cached; this is not a good fit if, say, you want to dynamically generate an inputstream that just serves up an endless amount of 0 bytes, for example. It's a good fit if the data is coming from, say, a USB connected arduino that from time to time sends information about a temperature sensor that it's connected to. Note that you need the thread to store the data it receives someplace, and then have an inputstream that will 'pull' from this queue of data you're making. To make an inputstream that pulls from a queue, see the next section. As this will involve threads, use something from java.util.concurrent, such as ArrayBlockingQueue - this has the double benefit that you won't get infinite buffers, either (the act of putting something in the buffer will block if the buffer is full).
subclassing
What you can also do is take the code that can generate new values, but, put it in an envelope - a thing you can pass around. You want to make some code, but not run it - you want to run that later, when the thing you hand the inputstream to, calls .read().
One easy way to do that, is to extend InputStream - and then implement your own zero method. Looks something like this:
class InfiniteZeroesInputStream extends InputStream {
public int read() {
return 0;
}
}
It's that simple. Given:
try (InputStream in = new InfiniteZeroesInputStream()) {
in.read(); // returns 0.. and will always do so.
byte[] b = new byte[65536];
in.read(b); // fills the whole array with zeroes.
}
I'm trying to create a java program that downloads certain asset files from an FTP server to a local file. Because my (free) FTP server doesn't support file sizes over a few megabytes, I decided to split up the files when they are uploaded and recombine them when the program downloads them. This works, but it is rather slow, because for each file, it has to get the InputStream, which takes some time.
The FTP server I use has a way to download the files without actually logging into the server, so I'm using this code to get the InputStream:
private static final InputStream getInputStream(String file) throws IOException {
return new URL("http://site.website.com/path/" + file).openStream();
}
To get the InputStream of a part of the asset file I'm using this code:
public static InputStream getAssetInputStream(String asset, int num) throws IOException, FTPException {
try {
return getInputStream("assets/" + asset + "_" + num + ".raf");
} catch (Exception e) {
// error handling
}
}
Because the getAssetInputStreams(String, int) method takes some time to run (especially if the file size is more then a megabyte), I decided to make the code that actually downloads the file multi-threaded. Here is where my problem lies.
final Map<Integer, Boolean> done = new HashMap<Integer, Boolean>();
final Map<Integer, byte[]> parts = new HashMap<Integer, byte[]>();
for (int i = 0; i < numParts; i++) {
final int part = i;
done.put(part, false);
new Thread(new Runnable() {
#Override
public void run() {
try {
InputStream is = FTP.getAssetInputStream(asset, part);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buf = new byte[DOWNLOAD_BUFFER_SIZE];
int len = 0;
while ((len = is.read(buf)) > 0) {
baos.write(buf, 0, len);
curDownload.addAndGet(len);
totAssets.addAndGet(len);
}
parts.put(part, baos.toByteArray());
done.put(part, true);
} catch (IOException e) {
// error handling
} catch (FTPException e) {
// error handling
}
}
}, "Download-" + asset + "-" + i).start();
}
while (done.values().contains(false)) {
try {
Thread.sleep(100);
} catch(InterruptedException e) {
e.printStackTrace();
}
}
File assetFile = new File(dir, "assets/" + asset + ".raf");
assetFile.createNewFile();
FileOutputStream fos = new FileOutputStream(assetFile);
for (int i = 0; i < numParts; i++) {
fos.write(parts.get(i));
}
fos.close();
This code works, but not always. When I run it on my desktop computer, it works almost always. Not 100% of the time, but often it works just fine. On my laptop, which has a far worse internet connection, it almost never works. The result is a file that is incomplete. Sometimes, it downloads 50% of the file. Sometimes, it downloads 90% of the file, it differs every time.
Now, if I replace the .start() by .run(), the code works just fine, 100% of the time, even on my laptop. It is, however, incredibly slow, so I'd rather not use .run().
Is there a way I could change my code so it does work multi-threaded? Any help will be appreciated.
Firstly, get your FTP server replaced, there are plenty of free FTP servers that support arbitrary file size serving with additional features, but I digress...
Your code seems to have many unrelated problems that could potentially all cause the behavior you are seeing, addressed below:
You have race conditions from accessing the done and parts maps from unprotected/unsynchronized access from multiple threads. This could cause data corruption and loss of synchronization for these variables between threads, potentially causing done.values().contains(false) to return true even when it's really not.
You are calling done.values().contains() repeatedly at a high frequency. Whilst the javadoc doesn't explicitly state, a hash map likely traverses every value in a O(n) fashion to check if a given map contains a value. Coupled with the fact that other threads are modifying the map, you'll get undefined behavior. According to values() javadoc:
If the map is modified while an iteration over the collection is in progress (except through the iterator's own remove operation), the results of the iteration are undefined.
You are somehow calling new URL("http://site.website.com/path/" + file).openStream(); but stating you are using FTP. The http:// in the link defines the protocol openStream() tries to open in and http:// is not ftp://. Not sure if this is a typo or did you mean HTTP (or do you have an HTTP server serving identical files).
Any thread raising any type of Exception will cause the code to fail given that not all parts will have "completed" (based on your busy-wait loop design). Granted, you may be redacted some other logic to guard against this, but otherwise this is a potential problem with the code.
You aren't closing any streams that you've opened. This could mean that the underlying socket itself is also left open. Not only does this constitute resource leakage, if the server itself has some sort of maximum number of simultaneous connection limit, you are only causing new connections to fail because the old, completed transfers are not closed.
Based on the issues above, I propose moving the download logic into a Callable task and running them through an ExecutorService as follows:
LinkedList<Callable<byte[]>> tasksToExecute = new LinkedList<>();
// Populate tasks to run
for(int i = 0; i < numParts; i++){
final int part = i;
// Lambda to
tasksToExecute.add(() -> {
InputStream is = null;
try{
is = FTP.getAssetInputStream(asset, part);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buf = new byte[DOWNLOAD_BUFFER_SIZE];
int len = 0;
while((len = is.read(buf)) > 0){
baos.write(buf, 0, len);
curDownload.addAndGet(len);
totAssets.addAndGet(len);
}
return baos.toByteArray();
}catch(IOException e){
// handle exception
}catch(FTPException e){
// handle exception
}finally{
if(is != null){
try{
is.close();
}catch(IOException ignored){}
}
}
return null;
});
}
// Retrieve an ExecutorService instance, note the use of work stealing pool is Java 8 only
// This can be substituted for newFixedThreadPool(nThreads) for Java < 8 as well for tight control over number of simultaneous links
ExecutorService executor = Executors.newWorkStealingPool(4);
// Tells the executor to execute all the tasks and give us the results
List<Future<byte[]>> resultFutures = executor.invokeAll(tasksToExecute);
// Populates the file
File assetFile = new File(dir, "assets/" + asset + ".raf");
assetFile.createNewFile();
try(FileOutputStream fos = new FileOutputStream(assetFile)){
// Iterate through the futures, writing them to file in order
for(Future<byte[]> result : resultFutures){
byte[] partData = result.get();
if(partData == null){
// exception occured during downloading this part, handle appropriately
}else{
fos.write(partData);
}
}
}catch(IOException ex(){
// handle exception
}
Using the executor service, you further optimize your multi-threading scenario since the output file will start writing as soon as pieces (in order) are available and that threads themselves are reused to save on thread creation costs.
As mentioned, there could be the case where too many simultaneous links causes the server to reject connections (or even more dangerously, write an EOF to make you think the part was downloaded). In this case, the number of worker threads can be tweaked by newFixedThreadPool(nThreads) to ensure at any given time, only nThreads amount of downloads can happen concurrently.
I am developing a tool to get client information, send to a server, and receive the information again (a proxy). I'm also trying to dump the data being received from the server. I can read the Integer representation of the inputStream, but I am not able to read the String format. I've tried the below example, but it hangs and never connects to the server. Also, System.out.println(inputStream.nextLine()) displays only one line and hangs.
public void run() {
try {
int i;
while ((i = inputStream.read()) != -1){
System.out.println(IOUtils.toString(inputStream));
outputStream.write(i);
}
} catch (IOException e) {
System.out.println("Lost connection to the client.");
}
}
My guess at this is that you're reading from the input stream, and then using the IOUtils library to read from the stream too. My suspicion is that your application is reading the first byte from the input stream, then reading the remainder of the inputstream with the IOUtils library, and then printing out the initial byte that was read.
It doesn't make any sense to call IOUtils.toString(inputstream) from within a loop. That method call will put all the data from the inputstream into a string. Why have the loop at all in this case?
You might want to try not using the IOUtils library for this. Just read a byte of data, push it into a StringBuilder, and then print that byte. In this approach, the loop would be necessary, and you'll probably get what you're looking for.
Try something like this, but modify it as necessary to print the data at the same time to your output stream:
public static String inputStreamToString(final InputStream is, final int bufferSize)
{
final char[] buffer = new char[bufferSize];
final StringBuilder out = new StringBuilder();
try {
final Reader in = new InputStreamReader(is, "UTF-8");
try {
for (;;) {
int rsz = in.read(buffer, 0, buffer.length);
if (rsz < 0)
break;
out.append(buffer, 0, rsz);
}
}
finally {
in.close();
}
}
catch (UnsupportedEncodingException ex) {
/* ... */
}
catch (IOException ex) {
/* ... */
}
return out.toString();
}
The code you posted doesn't attempt to connect to the server, but if any of it executes you must already have connected.
If your program is hanging in this code, either the server hasn't sent any data yet, or the IOUtils.toString() method probably tries to read to EOS, so if the peer doesn't close the connection you will block here forever.
If your program hangs at a readLine() call it means the peer hasn't sent a line to read.
How do you design a read and write loop which operates on a single socket (which supports parallel read and write operations)? Do I have to use multiple threads? Is my (java) solution any good? What about that sleep command? How do you use that within such a loop?
I'm trying to use 2 Threads:
Read
public void run() {
InputStream clientInput;
ByteArrayOutputStream byteBuffer;
BufferedInputStream bufferedInputStream;
byte[] data;
String dataString;
int lastByte;
try {
clientInput = clientSocket.getInputStream();
byteBuffer = new ByteArrayOutputStream();
bufferedInputStream = new BufferedInputStream(clientInput);
while(isRunning) {
while ((lastByte = bufferedInputStream.read()) > 0) {
byteBuffer.write(lastByte);
}
data = byteBuffer.toByteArray();
dataString = new String(data);
byteBuffer.reset();
}
} catch (IOException e) {
e.printStackTrace();
}
}
Write
public void run() {
OutputStream clientOutput;
byte[] data;
String dataString;
try {
clientOutput = clientSocket.getOutputStream();
while(isOpen) {
if(!commandQueue.isEmpty()) {
dataString = commandQueue.poll();
data = dataString.getBytes();
clientOutput.write(data);
}
Thread.sleep(1000);
}
clientOutput.close();
}
catch (IOException e) {
e.printStackTrace();
}
catch (InterruptedException e) {
e.printStackTrace();
}
}
Read fails to deliver a proper result, since there is no -1 sent.
How do I solve this issue?
Is this sleep / write loop a good solution?
There are basically three ways to do network I/O:
Blocking. In this mode reads and writes will block until they can be fulfilled, so if you want to do both simultaneously you need separate threads for each.
Non-blocking. In this mode reads and writes will return zero (Java) or in some languages (C) a status indication (return == -1, errno=EAGAIN/EWOULDBLOCK) when they cannot be fulfilled, so you don't need separate threads, but you do need a third API that tells you when the operations can be fulfilled. This is the purpose of the select() API.
Asynchronous I/O, in which you schedule the transfer and are given back some kind of a handle via which you can interrogate the status of the transfer, or, in more advanced APIs, a callback.
You should certainly never use the while (in.available() > 0)/sleep() style you are using here. InputStream.available() has few correct uses and this isn't one of them, and the sleep is literally a waste of time. The data can arrive within the sleep time, and a normal read() would wake up immediately.
You should rather use a boolean variable instead of while(true) to properly close your thread when you will want to. Also yes, you should create multiple thread, one per client connected, as the thread will block itself until a new data is received (with DataInputStream().read() for example). And no, this is not really a design question, each library/Framework or languages have its own way to listen from a socket, for example to listen from a socket in Qt you should use what is called "signals and slots", not an infinite loop.