We Keep Coding

java.io.IOException: Stream closed. what is the best way to write stream data to multiple files?

My java code receives stream data like twitter. I need to store the data e.g. 10000 records for each file. So, I need to recreate the file writer and buffered writer to create a new file then write data on it.
// global variables
String stat;
long counter = 0;
boolean first = true;
Date date;
SimpleDateFormat format;
String currentTime;
String fileName;
BufferedWriter bw = null;
FileWriter fw = null;
public static void main(String[] args) {
String dirToSave = args[0];
String fileIdentifier = args[1];
createFile(dirToSave, fileIdentifier);
StatusListener listener = new StatusListener() {
#Override
public void onStatus(Status status) {
stat = TwitterObjectFactory.getRawJSON(status);
try {
if(bw!=null){
bw.write(stat + "\n");
}
} catch (IOException ex) {
System.out.println(ex.getMessage());
}
counter++;
if (counter == 10000) {
createFile(dirToSave, fileIdentifier);
try {
TimeUnit.SECONDS.sleep(5);
} catch (InterruptedException ex) {
System.out.println(ex.getMessage());
}
counter = 0;
}
}
};
TwitterStream twitterStream = new TwitterStreamFactory(confBuild.build()).getInstance();
twitterStream.addListener(listener);
// twitterStream.filter(filQuery);
}
public static void createFile(String path, String fileIdentifier) {
date = new Date();
format = new SimpleDateFormat("yyyyMMddHHmm");
currentTime = format.format(date);
fileName = path + "/" + fileIdentifier + currentTime + ".json";
// if there was buffer before, flush & close it first before creating new file
if (!first) {
try {
bw.flush();
bw.close();
fw.close();
} catch (IOException ex) {
Logger.getLogger(LocalFile_All_en.class
.getName()).log(Level.SEVERE, null, ex);
}
} else {
first = false;
}
// create a new file
try {
fw = new FileWriter(fileName);
bw = new BufferedWriter(fw);
} catch (IOException ex) {
Logger.getLogger(Stack.class
.getName()).log(Level.SEVERE, null, ex);
}
}
However, i always get error after some hours.
SEVERE: null
java.io.IOException: Stream closed
EDIT: The error message says that, these codes throw the error
if (counter == 10000) {
createFile(dirToSave, fileIdentifier);
...
and
bw.flush();
What is the problem of my code? or is there a better way to write stream data like this?

If this error comes every now and then and writing after this error is ok again i think it can happen that bw is closed and not yet reopened while onStatus() tries to write of flush it.
So bw can be be not null but closed. You need to synchronize the closing/opening somehow.
For example make this stuff in onStatus() like so that you do not just write directly to bw but with some callbacks that handle the close/reopen new file.
Update: assuming here that this twitterStream can call onStatus() without waiting previous call finished. The first call has just closed the stream and the second is right after that writing to. Rare, but will happen in a long period of time.
Update2: this applies also to the flush() part.
I added this also as a short comment already but people often tell to get rid of static and especially global statics in java argumenting that it will cause big problems later which are hard to resolve/debug. This might be good case of it.
Read also:
Why are static variables considered evil?
Volatile Vs Static in java
Latter has an example how to sychronize concurrent requests.

Do not get input when doing Java Compiler

I'm doing a simple java compiler. My program is running, but if it is going to scan, it will not receive input and will freeze.
A small code from my compiler
public class ProcessBuilderMultipleCommandsExample {
static String backSlashFl = "C:\\Users\\xxA\\Desktop";
public static void main(String[] args) throws InterruptedException,
IOException {
// multiple commands
// /C Carries out the command specified by string and then terminates
ProcessBuilder pbC = new ProcessBuilder( //COMPİLE
"cmd.exe", "/c", "cd " + backSlashFl + "&& javac " + "Test" + ".java");
Process processC = pbC.start();
ProcessBuilder pb = new ProcessBuilder( //RUN
"cmd.exe", "/c", "cd " + backSlashFl + "&& java " + "Test");
Process process = pb.start();
IOThreadHandler outputHandler = new IOThreadHandler(
process.getInputStream());
outputHandler.start();
process.waitFor();
System.out.println(outputHandler.getOutput());
}
private static class IOThreadHandler extends Thread {
private InputStream inputStream;
private StringBuilder output = new StringBuilder();
IOThreadHandler(InputStream inputStream) {
this.inputStream = inputStream;
}
public void run() {
Scanner br = null;
try {
br = new Scanner(new InputStreamReader(inputStream));
String line = null;
while (br.hasNextLine()) {
line = br.nextLine();
output.append(line
+ System.getProperty("line.separator"));
}
} finally {
br.close();
}
}
public StringBuilder getOutput() {
return output;
}
}
}
I think it's working in the back, but how do I get the input part?
Here's the file I want to compile and run.
import java.awt.*;
import java.awt.event.*;
import java.util.Scanner;
public class numberScan {
public static void main(String[] args){
Scanner scan = new Scanner(System.in);
System.out.println("Enter the number: ");
int a=scan.nextInt();
System.out.println("Number= " + a);
}
}
I'm waiting for your help.
Editted
Now when I run the GUI, the Run key is pressed. What do you think I should do?
buttonRun.addActionListener(new ActionListener(){
public void actionPerformed(ActionEvent e){
String backSlashFl = file.getAbsolutePath().replace("\\", "\\\\");
backSlashFl = backSlashFl.replace(flName + ".java", "");
try {
execute("cmd.exe", "/c", "cd " + backSlashFl + " && java " + flName);
JOptionPane.showMessageDialog(null, "Dosya çalıştı!","Bilgilendirme",
JOptionPane.INFORMATION_MESSAGE);
} catch (FileNotFoundException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
} catch (IOException ae) {
// TODO Auto-generated catch block
ae.printStackTrace();
} catch (Exception e2){
e2.printStackTrace();
}
}
});
When the GUI app runs, this run button remains.

There are several issues with the given code. I tried to squeeze it into a comment, but now decided to extend it to an answer:
The class that you want to compile is called numberScan, but obviously stored in a file called Test.java. A public class can only be stored in a file that has the same name as the class. Call the class NumberScan, and call the file NumberScan.java.
You are only trying to print the output that is provided by the input stream. But you are not printing the result that is provided by the error stream (so if there are errors, you will not see them!). Also, you only printed the output of the Process process (which is used for running the program). You did not print the output of the Process processC, which was used for compiling the program.
The reason of why you don't see any output is that the line
System.out.println(outputHandler.getOutput());
is executed before the process is finished. You used waitFor, but the output is filled by a different thread - namely, the IOThreadHandler. The getOutput method could only be called after the IOThreadHandler has finished, but if you want to continuously update the output, then this will not work.
It is not entirely clear what you want to achieve, but guessing from the code that you provided, your goal seems to be to create a program that
Compiles the given Java file
Executes the resulting Java program
Prints possible error messages and the output that is created by the program
Important: Allows interacting with the program, in the sense that it should be possible to send input to the System.in of the program.
The last two points are particularly hard to achive manually. You would have to set up threads for reading the input stream and the error stream. These would require some trickery to make sure that the data is read continuously while the program is executed. Additionally, you would have to set up a thread that forwards the data that the user enters to the Java program that is executed in its own process.
Fortunately, all this has become fairly trivial with Java 7: You can simply set an appropriate ProcessBuilder.Redirect for all the streams (namely, the redirect INHERIT), so that all the streams are mapped to the corresponding streams of the surrounding program.
Here is an example:
import java.io.IOException;
import java.lang.ProcessBuilder.Redirect;
import java.util.Arrays;
public class ProcessBuilderMultipleCommandsExample {
private static String path = "C:\\Users\\xxA\\Desktop";
public static void main(String[] args)
throws InterruptedException, IOException {
execute("cmd.exe", "/c", "cd " + path + " && javac " + "NumberScan" + ".java");
execute("cmd.exe", "/c", "cd " + path + " && java " + "NumberScan");
}
private static void execute(String ... commands)
throws IOException, InterruptedException
{
System.out.println("Executing "+Arrays.asList(commands));
ProcessBuilder processBuilder = new ProcessBuilder(commands);
processBuilder.redirectInput(Redirect.INHERIT);
processBuilder.redirectOutput(Redirect.INHERIT);
processBuilder.redirectError(Redirect.INHERIT);
Process process = processBuilder.start();
process.waitFor();
}
}

Jar file not accepting huge string in java

What is the best way of passing a string (arg1 in case of code below) with about 800K characters (yes, that's a huge number) to a java jar. Following is the code I am using to invoke the jar file:
p = Runtime.getRuntime().exec(jrePath+"/bin/java -jar C:/folder/myjar.jar methodName" + arg1);
ALternately, how can I create a jar file to accept one String input and one byte[] input in main{ in void main(String args[])}
Or any other ideas? The requirement is to somehow pass the huge String/byte[] of String to a java jar file that I am creating

As mentioned in this question, there is a maximum argument length set by the operating system. Seeing as this argument is 800K characters, its fairly safe to say that you have exceeded this max value on most computers. To get around this, you can write arg1 to a temp file using the built in API:
final File temp;
try {
temp = File.createTempFile("temp-string", ".tmp");
} catch (final IOException e){
//TODO handle error
return;
}
try (final BufferedWriter writer = new BufferedWriter(new FileWriter(temp))) {
writer.write(arg1);
} catch (final IOException e){
//TODO handle error
return;
}
try {
// run process and wait for completion
final Process process = Runtime.getRuntime().exec(
jrePath + "/bin/java -jar C:/folder/myjar.jar methodName " +
temp.getAbsolutePath());
final int exitCode = process.waitFor();
if (exitCode != 0) {
//TODO handle error
}
} catch (final IOException | InterruptedException e){
//TODO handle error
return;
}
if (!file.delete()) {
//TODO handle error
}

Reading output from command line

I'm really stuck here, I've read number of articles and answers on stackoverflow, but nothing solved my problem.
I've got the method which runs Selenium Server Hub from cmd.exe using batch file:
public static boolean startGrid() throws IOException {
ProcessBuilder builder = new ProcessBuilder("cmd", "/c", "start", START_GRID_BAT_PATH);
builder.redirectErrorStream(true);
Process process = builder.start();
String out = getCmdOutput(process);
return out.contains("Selenium Grid hub is up and running");
}
The server is started successfully and is running in open cmd.
To make sure that server is up I use the following method to get the output from the cmd:
protected static String getCmdOutput(Process proc) throws java.io.IOException {
String res = "";
BufferedReader stdInput = new BufferedReader(new
InputStreamReader(proc.getInputStream()));
String s = "";
while ((s = stdInput.readLine()) != null) {
res += s;
}
return res;
}
And here is where the problem starts - method hangs at the the line s = stdInput.readLine()) != null
It seems that it can't read any line from the cmd, although I can see that there is number of lines in the output produced by running server.
Any ideas what is wrong?

Maybe the output produced by the server doesn't contain any \r or \n characters but is simply line-wrapped.
Anyhow, you shouldn't rely on the line feed characters when reading the output.
The following example shows how to read binary data from the InputStream of the process non-blocking with java.nio, convert the bytes to chars and write the result to a StringWriter - which is preferrable to using string concatenation.
protected static String getCmdOutput(Process proc, Consumer<String> consumer)
final InputStream source = proc.getInputStream();
final StringWriter sink = new StringWriter();
final ByteBuffer buf = ByteBuffer.allocate(1024);
final CharBuffer chars = CharBuffer.allocate(1024);
final CharsetDecoder decoder = Charset.defaultCharset().newDecoder();
try(final ReadableByteChannel from = Channels.newChannel(source)) {
while (from.read(buf) != -1) {
buf.flip();
decoder.decode(buf, chars, false);
chars.flip();
sink.write(chars.array(), chars.position(), chars.remaining());
buf.compact();
chars.clear();
}
} catch (IOException e) {
e.printStackTrace();
}
return sink.toString();
}
Edit:
As you are starting a process that is continuously producing output, the loop won't finish as long as the process is running. So to continue your parent process, you have to start a thread that is capturing and processing the output of the process continuously
For example
Consumer<String> consumer = ...;
Thread t = new Thread(() -> {
final InputStream source = proc.getInputStream();
final StringWriter sink = new StringWriter();
final ByteBuffer buf = ByteBuffer.allocate(1024);
final CharBuffer chars = CharBuffer.allocate(1024);
final CharsetDecoder decoder = Charset.defaultCharset().newDecoder();
try(final ReadableByteChannel from = Channels.newChannel(source)) {
while (from.read(buf) != -1) {
buf.flip();
decoder.decode(buf, chars, false);
chars.flip();
sink.write(chars.array(), chars.position(), chars.remaining());
forward(sink, consumer); //forward the captured content to a consumer
buf.compact();
chars.clear();
}
} catch (IOException e) {
e.printStackTrace();
}
});
t.start();
private void forward(StringWriter sink, Consumer<String> consumer) {
StringBuffer buf = sink.getBuffer();
int pos = buf.lastIndexOf("\n");
//you may use other options for triggering the process here,
//i.e. on every invocation, on fixed length, etc
if(pos != -1){
consumer.accept(buf.substring(0, pos + 1));
buf.delete(0, pos + 1);
}
//or to send the entire buffer
/*
consumer.accept(sink.toString());
sink.getBuffer().delete(0, sink.getBuffer().length());
*/
}
Depending on your use case, you may not necessarily need a separate thread. It might be ok to process the child-process' output in the main thread and do what you want with the output in the consumer.

It doesn't hang, it just blocks waiting for a line to be read. And after the line has been read, it waits for the next. And so on.
Getting the output stream of a child process should be made in a separate thread, because the child process could in fact hang, than your main thread will hang, too.
An example:
given the instance methods
private static final long WAITTIME_FOR_CHILD = Duration.ofMinutes(5).toNanos();
private static final String START_SEQUENCE = "whatever";
private final Lock lock = new ReentrantLock();
private final Condition waitForStart = lock.newCondition();
private boolean started;
the code that starts the child process
// prepare child process
lock.lock();
try {
Process process = processBuilder.start();
// both standard and error output streams get redirected
pipe(process.getInputStream(), System.out, START_SEQUENCE);
pipe(process.getErrorStream(), System.err, START_SEQUENCE);
// loop because of 'spurious wakeups' - should happen only on Linux,
// but better take care of it
// wait until WAITTIME_FOR_CHILD (default 5 minutes, see above)
long waitTime = WAITTIME_FOR_CHILD;
while (!started) {
// we wait hier until the child process starts or timeout happens
// value <= 0 means timeout
waitTime = waitForStart.awaitNanos(waitTime);
if (waitTime <= 0) {
process.destroyForcibly();
throw new IOException("Prozess xxx couldn't be started");
}
}
} catch (IOException | InterruptedException e) {
throw new RuntimeException("Error during start of xxx", e);
} finally {
lock.unlock();
}
private void getCmdOutput(InputStream in, PrintStream out, final String startSeq) {
startDaemon(() -> {
try (Scanner sc = new Scanner(in)) {
while (sc.hasNextLine()) {
String line = sc.nextLine();
// check your know output sequence
if (line.contains(startSeq)) {
lock.lock();
try {
if (!started) {
started = true;
// waiting for process start finished
waitForStart.signal();
}
} finally {
lock.unlock();
}
}
out.println(line);
}
}
});
}
private void startDaemon(Runnable task) {
Thread thread = new Thread(task);
thread.setDaemon(true);
thread.start();
}

Try replacing while condition like this:
while ((s = stdInput.readLine()) != null && !s.equals("")) {

ProcessBuilder pb =
new ProcessBuilder("myCommand", "myArg1", "myArg2");
Map<String, String> env = pb.environment();
env.put("VAR1", "myValue");
env.remove("OTHERVAR");
env.put("VAR2", env.get("VAR1") + "suffix");
pb.directory(new File("myDir"));
File log = new File("log");
pb.redirectErrorStream(true);
pb.redirectOutput(Redirect.appendTo(log));
Process p = pb.start();
assert pb.redirectInput() == Redirect.PIPE;
assert pb.redirectOutput().file() == log;
assert p.getInputStream().read() == -1;
You can redirect output to a file as oracle documentation offers in basic example:
https://docs.oracle.com/javase/7/docs/api/java/lang/ProcessBuilder.html

Java Runtime exec() get stuck after a while

I'm building a simple UI for ffmpeg launching ffmpeg.exe with parameters using exec(). it works on Os X but on Windows 7-8 after few seconds the ffmpeg process suspends itself and resumes only when I kill the father process. (also ddlhost.exe is created)
Ffmpeg.exe converts successfully the same video from cmd.
Searching on the internet I've found this answer but I have the same problem running a simple test program which is not using the Input and Error streams.
Here is the test program code which has the same problem of the main one:
public class Main {
static String param_ffmpeg_1 = "./data/ffmpeg.exe";
static String param_ffmpeg_2 = "-i";
static String in = "./data/source.mov";
static String out = "./data/out.flv";
static Process p;
public static void main(String[] args) {
/*File f = new File(out);
if (f.exists()){
f.delete();
}*/
Runtime rt = Runtime.getRuntime() ;
//String cmd1 = param_ffmpeg_1 + param_ffmpeg_2 + in_path + param_ffmpeg_3 + out_path ;
System.out.println(in);
System.out.println(out);
String[] cmd1 = new String[] { param_ffmpeg_1, param_ffmpeg_2, in, "-ar", "44100", "-vb", "2500k", "-s", "882x496", "-f", "flv", out};
try {
p = rt.exec(cmd1);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
int r = 123456;
try {
r = p.waitFor();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
System.out.println(r);
}
}

Does ffmpg write anything to the stdout or stderr? If yes you have to consume that. (In seperate threads as you need to consume the stderr and the stdout in parallel) see Process.getInputStream() and Process.getErrorStream() for the details. When the buffer is buffer is full your called programm is stopped and hangs.
The fact that it works in OS/X but not Windows might be caused by different buffer sizes.

You should call getInputStream() and getErrorStream() on the Process returned by Runtime.exec and drain that all the time the process is running. If you do not then it will eventually fill up and block and stop the process from running.
See Java Process with Input/Output Stream for examples.
From the accepted answer
ProcessBuilder builder = new ProcessBuilder("/bin/bash");
builder.redirectErrorStream(true);
Process process = builder.start();
InputStream itsOutput = process.getInputStream();
// Wrap the stream in a Reader ...
while ((line = reader.readLine ()) != null) {
System.out.println ("Stdout: " + line);
}