We Keep Coding

Reading stdout of nodejs from Java (using apache commons exec). Thread safe or not?

I'm trying to write a torrent streaming client in Java using webtorrent-cli, which runs on NodeJS. When installed as a node module, webtorrent-cli gives a nice webtorrent.cmd script which can be used to work with it. When download for a torrent starts, the cli updates the standard output each second with useful details like download speed, % of torrent downloaded, seeds available etc.
To observe such a "dynamic" stdout in Java (with commons exec), I am using the following snippet:
private static Thread processCreator() {
return new Thread(() -> {
try {
// Read stdout in a thread safe manner (hopefully)
final ByteArrayOutputStream baos = new ByteArrayOutputStream();
PumpStreamHandler handler = new PumpStreamHandler(baos);
String command = getCommand();
CommandLine cmd = CommandLine.parse(command);
Executor cmdExecutor = new DefaultExecutor();
cmdExecutor.setStreamHandler(handler);
// Schedule a service to print the content of baos each second
final ScheduledExecutorService service = Executors.newSingleThreadScheduledExecutor();
service.scheduleAtFixedRate(() -> {
try {
// Read and reset atomically
synchronized (baos) {
System.out.println(baos.toString("UTF-8"));
// Resetting so that buffer size doesn't grow arbitrarily
baos.reset();
}
}
catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
}, 0, 1, TimeUnit.SECONDS);
cmdExecutor.execute(cmd);
// Let the remaining bytes be processed
sleep(1000);
// Shutdown
service.shutdown();
} catch (IOException ioe) {
ioe.printStackTrace();
}
});
}
public static void main(String[] args) throws InterruptedException {
Thread process = processCreator();
process.start();
process.join();
}
I'm concerned about how the ByteArrayOutputStream is being written. The class itself is thread safe, but if the implementation writes to the buffer byte by byte, or in a way that "updated output" (from webtorrent-cli) is only partially written to the buffer by the time scheduled service captures the monitor and starts processing, then that's going to cause problems. In this case, because I'm just printing content of the buffer, it won't be that much of trouble I guess. But I've to process the output and extract out a couple of details in the fixed scheduled service. I can think of a different way to achieve proper co-ordination (e.g.: observe the completeness of an update by marking the event when buffer receives bytes that form the first line in webtorrent-cli's stdout...and mark the update as completed when buffer receives bytes that form the last line. Each update has identical first and last lines...or at least a few bytes in the beginning and end are identical). But that would be a bit more work than this. My question is, can I be certain that write to the buffer has happened in a single atomic call to ByteArrayOutputStream.write(byte[], ...)'. I hope I've explained my question well enough. If you need more details, let me know in the comments. BTW, when the code above is run, the output suggests that co-ordination is being properly managed. But maybe I'm just lucky that the race condition has been avoided so far?

What is the root cause of getting Out Of Memory Exception? How can we overcome this?

Here is my sample snippet for reading and writing by output stream and I am getting out of memory exception.
public static void readFileContent(InputStream in, OutputStream out) throws IOException {
byte[] buf = new byte[500000];
int nread;
int navailable;
int total = 0;
synchronized (in) {
try {
while((nread = in.read(buf, 0, buf.length)) >= 0) {
out.write(buf, 0, nread);
total += nread;
}
}
finally {
if (in != null) {
try {
in.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
out.flush();
buf = null;
}
What are the possible scenarios with the above snippet to get "out of memory exception" ?
Is it necessary to close the output stream here? And does stream, flush is enough or do we need to close the stream always? If so why?
How could I avoid Out of memory exception in general?
Please clarify me.

What are the possible scenarios with the above snippet to get "out of memory exception" ?
There are various root causes for out of memory exceptions. Refer to oracle documentation page for more details.
java.lang.OutOfMemoryError: Java heap space:
Cause: The detail message Java heap space indicates object could not be allocated in the Java heap.
java.lang.OutOfMemoryError: GC Overhead limit exceeded:
Cause: The detail message "GC overhead limit exceeded" indicates that the garbage collector is running all the time and Java program is making very slow progress
java.lang.OutOfMemoryError: Requested array size exceeds VM limit:
Cause: The detail message "Requested array size exceeds VM limit" indicates that the application (or APIs used by that application) attempted to allocate an array that is larger than the heap size.
java.lang.OutOfMemoryError: Metaspace:
Cause: Java class metadata (the virtual machines internal presentation of Java class) is allocated in native memory (referred to here as metaspace)
java.lang.OutOfMemoryError: request size bytes for reason. Out of swap space?:
Cause: The detail message "request size bytes for reason. Out of swap space?" appears to be an OutOfMemoryError exception. However, the Java HotSpot VM code reports this apparent exception when an allocation from the native heap failed and the native heap might be close to exhaustion
Is it necessary to close the output stream here? And does stream, flush is enough or do we need to close the stream always? If so why?
since you are using raw InputStream and OutputStream in your method, we don' t know which type of actual Stream is getting passed to this method and hence explicitly close these Streams is good idea.
How could I avoid Out of memory exception in general?
This question is already answered in response to your first question.
Refer to this SE question on handling large files for IO operations :
Java OutOfMemoryError in reading a large text file

Change the buf to new byte[1*1024]
Read using just buf no need to specify length e.g. pos = in.read(buf)
The rest of the code looks good. No need to increase the memory.
Also, any points of synchronised inputStream?

I think it's obvious that the problem is that you allocate 500000 bytes at once, and they may not be available in the heap at runtime.
Explanation:
I would not suggest it, but you could increment the heap size of your program.
The default heap size for a java program is determined at runtime, but it can also be parameterized.
Recommendation:
As far as I can see by the provided snippet, it's not absolutely necessary to read 500000 bytes at once. So, you can initialize your byte array with a smaller number that would result in having more reading loops. But if it's not a problem for your program... I guess.
Conclusion:
Try by setting the initial byte array size to 5000, or even 1000.
EDIT:
An extra point to take into consideration is that in the above code snippet you only flush once at the end. The bytes you are writting to the OutputStream are kept in memory, and their size may cause an OutOfMemoryException also.
In order to overcome this, you should flush more often. It will affect your performance if you flush too often, but you can always experiment with a condition in your loop e.g.
...
if (total % 5000 == 0) {
out.flush();
}
...
EDIT 2:
As the InputStream and OutputStream objects are passed to the given method as parameters, so, in my opinion this method is not responsible for closing them. The method that initializes the Streams is also responsible for close them gracefully. Flush is enough for this method. But consider doing it in smaller chunks.
EDIT 3:
To summarize the suggested tweaks:
public static void readFileContent(InputStream in, OutputStream out) throws IOException {
byte[] buf = new byte[1000];
// wrap your OutputStream in a BufferedOutputStream
BufferedOutputStream bos = new BufferedOutputStream(out, 5000);
int nread;
int navailable;
int total = 0;
synchronized (in) {
try {
while((nread = in.read(buf, 0, buf.length)) >= 0) {
// use the BufferedOutputStream to write data
// you don't need to flush regularly as it is handled automatically every time the buffer is full
bos.write(buf, 0, nread);
total += nread;
}
}
finally {
if (in != null) {
try {
in.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
// flush the last contents of the BufferedOutputStream
bos.flush();
buf = null;
}
Please note also that BufferedOutputStream will automatically call flush() when you will close it gracefully.
EDIT 4:
Example calling the above method:
public static void main(String[] args) {
String filename = "test.txt";
String newFilename = "newtest.txt";
File file = new File(filename);
File newFile = new File(newFilename);
try (InputStream fis = new FileInputStream(file);
OutputStream fout = new FileOutputStream(newFile)) {
readFileContent(fis, fout);
}
catch(IOException ioe) {
System.out.println(ioe.getMessage());
}
}

In Java there isn't any brute way of freeing memory. Even calling the built-in Garbage Collector (System.gC()) might not solve the problem, as the GC only frees objects that are not referenced anymore. You need to take care of the code that you are writing so that it can employ the resources in its best way it can. Certainly, there are cases that you are left out of options, especially when you are using big or giant data structures regardless of any code optimization you can think of (in your case, you are creating an array with half a million records of bytes).
As a partial solution, you can increase your Heap Size Memory so that Java can allocate more memory.

Inconsistent output from multithreaded FTP InputStreams

I'm trying to create a java program that downloads certain asset files from an FTP server to a local file. Because my (free) FTP server doesn't support file sizes over a few megabytes, I decided to split up the files when they are uploaded and recombine them when the program downloads them. This works, but it is rather slow, because for each file, it has to get the InputStream, which takes some time.
The FTP server I use has a way to download the files without actually logging into the server, so I'm using this code to get the InputStream:
private static final InputStream getInputStream(String file) throws IOException {
return new URL("http://site.website.com/path/" + file).openStream();
}
To get the InputStream of a part of the asset file I'm using this code:
public static InputStream getAssetInputStream(String asset, int num) throws IOException, FTPException {
try {
return getInputStream("assets/" + asset + "_" + num + ".raf");
} catch (Exception e) {
// error handling
}
}
Because the getAssetInputStreams(String, int) method takes some time to run (especially if the file size is more then a megabyte), I decided to make the code that actually downloads the file multi-threaded. Here is where my problem lies.
final Map<Integer, Boolean> done = new HashMap<Integer, Boolean>();
final Map<Integer, byte[]> parts = new HashMap<Integer, byte[]>();
for (int i = 0; i < numParts; i++) {
final int part = i;
done.put(part, false);
new Thread(new Runnable() {
#Override
public void run() {
try {
InputStream is = FTP.getAssetInputStream(asset, part);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buf = new byte[DOWNLOAD_BUFFER_SIZE];
int len = 0;
while ((len = is.read(buf)) > 0) {
baos.write(buf, 0, len);
curDownload.addAndGet(len);
totAssets.addAndGet(len);
}
parts.put(part, baos.toByteArray());
done.put(part, true);
} catch (IOException e) {
// error handling
} catch (FTPException e) {
// error handling
}
}
}, "Download-" + asset + "-" + i).start();
}
while (done.values().contains(false)) {
try {
Thread.sleep(100);
} catch(InterruptedException e) {
e.printStackTrace();
}
}
File assetFile = new File(dir, "assets/" + asset + ".raf");
assetFile.createNewFile();
FileOutputStream fos = new FileOutputStream(assetFile);
for (int i = 0; i < numParts; i++) {
fos.write(parts.get(i));
}
fos.close();
This code works, but not always. When I run it on my desktop computer, it works almost always. Not 100% of the time, but often it works just fine. On my laptop, which has a far worse internet connection, it almost never works. The result is a file that is incomplete. Sometimes, it downloads 50% of the file. Sometimes, it downloads 90% of the file, it differs every time.
Now, if I replace the .start() by .run(), the code works just fine, 100% of the time, even on my laptop. It is, however, incredibly slow, so I'd rather not use .run().
Is there a way I could change my code so it does work multi-threaded? Any help will be appreciated.

Firstly, get your FTP server replaced, there are plenty of free FTP servers that support arbitrary file size serving with additional features, but I digress...
Your code seems to have many unrelated problems that could potentially all cause the behavior you are seeing, addressed below:
You have race conditions from accessing the done and parts maps from unprotected/unsynchronized access from multiple threads. This could cause data corruption and loss of synchronization for these variables between threads, potentially causing done.values().contains(false) to return true even when it's really not.
You are calling done.values().contains() repeatedly at a high frequency. Whilst the javadoc doesn't explicitly state, a hash map likely traverses every value in a O(n) fashion to check if a given map contains a value. Coupled with the fact that other threads are modifying the map, you'll get undefined behavior. According to values() javadoc:
If the map is modified while an iteration over the collection is in progress (except through the iterator's own remove operation), the results of the iteration are undefined.
You are somehow calling new URL("http://site.website.com/path/" + file).openStream(); but stating you are using FTP. The http:// in the link defines the protocol openStream() tries to open in and http:// is not ftp://. Not sure if this is a typo or did you mean HTTP (or do you have an HTTP server serving identical files).
Any thread raising any type of Exception will cause the code to fail given that not all parts will have "completed" (based on your busy-wait loop design). Granted, you may be redacted some other logic to guard against this, but otherwise this is a potential problem with the code.
You aren't closing any streams that you've opened. This could mean that the underlying socket itself is also left open. Not only does this constitute resource leakage, if the server itself has some sort of maximum number of simultaneous connection limit, you are only causing new connections to fail because the old, completed transfers are not closed.
Based on the issues above, I propose moving the download logic into a Callable task and running them through an ExecutorService as follows:
LinkedList<Callable<byte[]>> tasksToExecute = new LinkedList<>();
// Populate tasks to run
for(int i = 0; i < numParts; i++){
final int part = i;
// Lambda to
tasksToExecute.add(() -> {
InputStream is = null;
try{
is = FTP.getAssetInputStream(asset, part);
ByteArrayOutputStream baos = new ByteArrayOutputStream();
byte[] buf = new byte[DOWNLOAD_BUFFER_SIZE];
int len = 0;
while((len = is.read(buf)) > 0){
baos.write(buf, 0, len);
curDownload.addAndGet(len);
totAssets.addAndGet(len);
}
return baos.toByteArray();
}catch(IOException e){
// handle exception
}catch(FTPException e){
// handle exception
}finally{
if(is != null){
try{
is.close();
}catch(IOException ignored){}
}
}
return null;
});
}
// Retrieve an ExecutorService instance, note the use of work stealing pool is Java 8 only
// This can be substituted for newFixedThreadPool(nThreads) for Java < 8 as well for tight control over number of simultaneous links
ExecutorService executor = Executors.newWorkStealingPool(4);
// Tells the executor to execute all the tasks and give us the results
List<Future<byte[]>> resultFutures = executor.invokeAll(tasksToExecute);
// Populates the file
File assetFile = new File(dir, "assets/" + asset + ".raf");
assetFile.createNewFile();
try(FileOutputStream fos = new FileOutputStream(assetFile)){
// Iterate through the futures, writing them to file in order
for(Future<byte[]> result : resultFutures){
byte[] partData = result.get();
if(partData == null){
// exception occured during downloading this part, handle appropriately
}else{
fos.write(partData);
}
}
}catch(IOException ex(){
// handle exception
}
Using the executor service, you further optimize your multi-threading scenario since the output file will start writing as soon as pieces (in order) are available and that threads themselves are reused to save on thread creation costs.
As mentioned, there could be the case where too many simultaneous links causes the server to reject connections (or even more dangerously, write an EOF to make you think the part was downloaded). In this case, the number of worker threads can be tweaked by newFixedThreadPool(nThreads) to ensure at any given time, only nThreads amount of downloads can happen concurrently.

Thread not releasing memory

I'm trying to play a sound effect in my program using threads, I searched the web and as I understand when a thread reaches the end of the run function it will become free for the GC to collect.
However when I call for the function many times one after another the task manager shows a high increase at memory usage and it never went back down, I waited for 2 minutes for the GC but there was no effect.
Here is the code that I use for playing sound effect:
public static void playSfx(final String path) {
new Thread(new Runnable() {
public void run() {
try {
AudioInputStream audioInputStream = AudioSystem.getAudioInputStream(new File(path));
final int BUFFER_SIZE = 128000;
SourceDataLine sourceLine = null;
AudioFormat audioFormat = audioInputStream.getFormat();
DataLine.Info info = new DataLine.Info(SourceDataLine.class, audioFormat);
sourceLine = (SourceDataLine) AudioSystem.getLine(info);
sourceLine.open(audioFormat);
if (sourceLine == null)
return;
sourceLine.start();
int nBytesRead = 0;
byte[] abData = new byte[BUFFER_SIZE];
while (nBytesRead != -1) {
try {
nBytesRead = audioInputStream.read(abData, 0, abData.length);
} catch (IOException e) {
e.printStackTrace();
}
if (nBytesRead >= 0) {
sourceLine.write(abData, 0, nBytesRead);
}
}
sourceLine.drain();
sourceLine.close();
audioInputStream.close();
} catch (IOException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
System.exit(1);
}
}
}).start();
}
What should I do to reclaim the lost memory?

I can't reproduce this. For me this will peak at about 60k memory usage, which
considering the language in question is perfectly ok imo. Do you have any small
code which reliably reproduces the problem?
public static void main(String[] args) {
for(int i = 0; i < 1000; i++) {
playSfx("somesortsound.wav");
try {
Thread.sleep(100);
} catch (Exception e) {
}
}
}
In addition, what are your expectations when it comes to memory usage? Predicting GC
behaviour is not easy, and you are not guaranteed to see memory freed just because
you wait a while. Especially so if you are not allocating a lot.
I would also get rid of the try catch in your read loop. You don't handle any
exception there anyway, and you catch it outside of the loop as well. If the
stream causes an IO exception chances are you want to abort the loop anyway, right?

Running that method from the following class with compatible files and without any exceptions; I did not see that behaviour.
public class PlaySound {
public static void main(String[] args) throws Exception {
String filePath = "C:"+File.separator+"bach.wav";
playSfx(filePath);
playSfx(filePath);
playSfx(filePath);
while(true){
Thread.sleep(1000);
}
}
}
It could be a memory leak. You are not managing the resources with a finally block (or 'try-with-resources'), coupling this with return statements could result in the internals of the AudioInputStream driver maintaining a reference to the file.
I would use a try-with-resources block.
try(AudioInputStream audioInputStream = AudioSystem.getAudioInputStream(new File(path));){
//your code
}
Also, I wouldn't use the task manager to reliably determine the garbage collectors behaviour, there are many components that determine the size of a Java process in memory.
To gain an understanding of what's going on; use VisualVM. (This is located in the bin directory of your jdk). You can see when the garbage collector is active, as well as the size of the heap compared to its current maximum size. You can also 'Request' (does not guarantee) garbage collection, which will give you an indication of the amount of memory required for 'survivors', and the amount that can be freed.
Admittedly 2 minutes is a long time, but it is worth mentioning garbage collection usually occurs as a failure to allocate memory to a particular generation, but this is dependant on the particular GC algorithm being used. In other words, if it didn't reach an upper limit then it is possible it wouldn't have performed a garbage collection.
This article is potentially worth a read/glance; it details various different garbage collectors and how they are expected to function.

java: decomprss files into string too slow

Here is how I compressed the string into a file:
public static void compressRawText(File outFile, String src) {
FileOutputStream fo = null;
GZIPOutputStream gz = null;
try {
fo = new FileOutputStream(outFile);
gz = new GZIPOutputStream(fo);
gz.write(src.getBytes());
gz.flush();
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
gz.close();
fo.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
Here is how I decompressed it:
static int BUFFER_SIZE = 8 * 1024;
static int STRING_SIZE = 2 * 1024 * 1024;
public static String decompressRawText(File inFile) {
InputStream in = null;
InputStreamReader isr = null;
StringBuilder sb = new StringBuilder(STRING_SIZE);//constant resizing is costly, so set the STRING_SIZE
try {
in = new FileInputStream(inFile);
in = new BufferedInputStream(in, BUFFER_SIZE);
in = new GZIPInputStream(in, BUFFER_SIZE);
isr = new InputStreamReader(in);
char[] cbuf = new char[BUFFER_SIZE];
int length = 0;
while ((length = isr.read(cbuf)) != -1) {
sb.append(cbuf, 0, length);
}
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
in.close();
} catch (Exception e1) {
e1.printStackTrace();
}
}
return sb.toString();
}
The decompression seems to take forever to do. I have got a feeling that I am doing too much redundant steps in the decompression bit. any idea of how I could speed it up?
EDIT: have modified the code to the above based on the following given recommendations,
1. I chaged the pattern, so to simply my code a bit, but if I couldn't use IOUtils is this still ok to use this pattern?
2. I set the StringBuilder buffer to be of 2M, as suggested by entonio, should I set it to be a little bit more? the memory is still OK, I still have around 10M available as it is suggested by the heap monitor from eclipse
3. I cut the BufferedReader and added a BufferedInputStream, but I am still not sure about the BUFFER_SIZE, any suggestions?
The above modification has improved the time taken to loop all my 30 2M files from almost 30 seconds to around 14, but I need to reduce it to under 10, is it even possible on android? Ok, basically, I need to process a text file in all 60M, I have divided them up into 30 2M, and before I start processing on each strings, I did the above timing on the time cost for me just to loop all the files and get the String in the file into my memory. Since I don't have much experience, will it be better, if I use 60 of 1M files instead? or any other improvement should I adopt? Thanks.
ALSO: Since physical IO is quite time consuming, and since my compressed version of files are all quite small(around 2K from 2M of text), is it possible for me to still do the above, but on a file that is already mapped to memory? possibly using java NIO? Thanks

The BufferedReader's only purpose is the readLine() method you don't use, so why not just read from the InputStreamReader? Also, maybe decreasing the buffer size may be helpful. Also, you should probably specify the encoding while both reading and writing, though that shouldn't have an impact on performance.
edit: more data
If you know the size of the string ahead, you should add a length parameter to decompressRawText and use it to initialise the StringBuilder. Otherwise it will be constantly resized in order to accomodate the result, and that's costly.
edit: clarification
2MB implies a lot of resizes. There is no harm if you specify a capacity higher than the length you end up with after reading (other than temporarily using more memory, of course).

You should wrap the FileInputStream with a BufferedInputStream before wrapping with a GZipInputStream, rather than using a BufferedReader.
The reason is that, depending on implementation, any of the various input classes in your decoration hierarchy could decide to read on a byte-by-byte basis (and I'd say the InputStreamReader is most likely to do this). And that would translate into many read(2) calls once it gets to the FileInputStream.
Of course, this may just be superstition on my part. But, if you're running on Linux, you can always test with strace.
Edit: once nice pattern to follow when building up a bunch of stream delegates is to use a single InputStream variable. Then, you only have one thing to close in your finally block (and can use Jakarta Commons IOUtils to avoid lots of nested try-catch-finally blocks).
InputStream in = null;
try
{
in = new FileInputStream("foo");
in = new BufferedInputStream(in);
in = new GZIPInputStream(in);
// do something with the stream
}
finally
{
IOUtils.closeQuietly(in);
}

Add a BufferedInputStream between the FileInputStream and the GZIPInputStream.
Similarly when writing.