We Keep Coding

java.io.IOException: Stream closed. what is the best way to write stream data to multiple files?

My java code receives stream data like twitter. I need to store the data e.g. 10000 records for each file. So, I need to recreate the file writer and buffered writer to create a new file then write data on it.
// global variables
String stat;
long counter = 0;
boolean first = true;
Date date;
SimpleDateFormat format;
String currentTime;
String fileName;
BufferedWriter bw = null;
FileWriter fw = null;
public static void main(String[] args) {
String dirToSave = args[0];
String fileIdentifier = args[1];
createFile(dirToSave, fileIdentifier);
StatusListener listener = new StatusListener() {
#Override
public void onStatus(Status status) {
stat = TwitterObjectFactory.getRawJSON(status);
try {
if(bw!=null){
bw.write(stat + "\n");
}
} catch (IOException ex) {
System.out.println(ex.getMessage());
}
counter++;
if (counter == 10000) {
createFile(dirToSave, fileIdentifier);
try {
TimeUnit.SECONDS.sleep(5);
} catch (InterruptedException ex) {
System.out.println(ex.getMessage());
}
counter = 0;
}
}
};
TwitterStream twitterStream = new TwitterStreamFactory(confBuild.build()).getInstance();
twitterStream.addListener(listener);
// twitterStream.filter(filQuery);
}
public static void createFile(String path, String fileIdentifier) {
date = new Date();
format = new SimpleDateFormat("yyyyMMddHHmm");
currentTime = format.format(date);
fileName = path + "/" + fileIdentifier + currentTime + ".json";
// if there was buffer before, flush & close it first before creating new file
if (!first) {
try {
bw.flush();
bw.close();
fw.close();
} catch (IOException ex) {
Logger.getLogger(LocalFile_All_en.class
.getName()).log(Level.SEVERE, null, ex);
}
} else {
first = false;
}
// create a new file
try {
fw = new FileWriter(fileName);
bw = new BufferedWriter(fw);
} catch (IOException ex) {
Logger.getLogger(Stack.class
.getName()).log(Level.SEVERE, null, ex);
}
}
However, i always get error after some hours.
SEVERE: null
java.io.IOException: Stream closed
EDIT: The error message says that, these codes throw the error
if (counter == 10000) {
createFile(dirToSave, fileIdentifier);
...
and
bw.flush();
What is the problem of my code? or is there a better way to write stream data like this?

If this error comes every now and then and writing after this error is ok again i think it can happen that bw is closed and not yet reopened while onStatus() tries to write of flush it.
So bw can be be not null but closed. You need to synchronize the closing/opening somehow.
For example make this stuff in onStatus() like so that you do not just write directly to bw but with some callbacks that handle the close/reopen new file.
Update: assuming here that this twitterStream can call onStatus() without waiting previous call finished. The first call has just closed the stream and the second is right after that writing to. Rare, but will happen in a long period of time.
Update2: this applies also to the flush() part.
I added this also as a short comment already but people often tell to get rid of static and especially global statics in java argumenting that it will cause big problems later which are hard to resolve/debug. This might be good case of it.
Read also:
Why are static variables considered evil?
Volatile Vs Static in java
Latter has an example how to sychronize concurrent requests.

functions having different speed in java

In my program I have something like that: two functions in the same thread
function1();
function2();
Function1 uses a process builder to execute some commands and then writes the results in 3 different files.
Function2 reads the files and uses their data to continue the program.
The problem is that apparently function1 takes too much time that function2 often does not find the files created by function1.
Maybe the process runs the second function before function1 ends its processing.
Is there a way to solve this?
I thought about a thread.sleep
try {
Thread.sleep(2000);
} catch(InterruptedException ex) {
Thread.currentThread().interrupt();
}
The problem with this is that the program became too slowly especially that I am working on an important amount of data.

I think you can use waitFor() method until your process terminates in the function1() and then call function2().
Process p = new ProcessBuilder(<process>).start();
p.waitFor();

Here is my code: (input is an hashmap)
for (String fileName : input.get(project)) {
fileInformation[0] = projectName;
fileInformation[1] = fileName;
runCommands(nameParameter, fileName);
// try {
// Thread.sleep(2000); // 1000 milliseconds is one second.
// } catch (InterruptedException ex) {
// Thread.currentThread().interrupt();
// }
ArrayList<ArrayList<String>> allPathsAndDatesResults = allPathsAndDates(nameParameter, fileName);
function 1 is here runCommands
public static void runCommands(String nameParameter, String fileName) throws InterruptedException {
String batPath = Tool.getBatPath();
List<String> cmdAndArgs = Arrays.asList(batPath);
String dirPath = Tool.getShellDir() + nameParameter;
File dir = new File(dirPath);
ProcessBuilder pb = new ProcessBuilder(cmdAndArgs);
pb.environment().put("fileName", fileName);
pb.directory(dir);
File log = new File("log");
pb.redirectErrorStream(true);
pb.redirectOutput(Redirect.appendTo(log));
Process p;
try {
p = pb.start();
assert pb.redirectInput() == Redirect.PIPE;
assert pb.redirectOutput().file() == log;
assert p.getInputStream().read() == -1;
//p.waitFor();
} catch (IOException e) {
e.printStackTrace();
}
}
And function 2 is allPathsAndDates
public static ArrayList<ArrayList<String>> allPathsAndDates(String nameParameter, String fileName)
throws IOException {
ArrayList<ArrayList<String>> allFiles = new ArrayList<ArrayList<String>>();
String allPaths = Tool.getResultsPath() + "\\" + fileName + "_Path.txt";
String allDates = Tool.getResultsPath() + "\\" + fileName + "_Date.txt";
BufferedReader br = new BufferedReader(new FileReader(allPaths)); //etc...

If you call function1() then function2(), you won't have a problem. Don't use threads if the operations don't have to be done simultaneously.
If you have to use threads, as said Jiga Joshi in the comments, you can use the join() function to force a thread to wait for another thread's execution end.

Using ProcessBuilder to read/write to telnet process

I am trying to read/write values from/to telnet process by means of ProcessBuilder.
public static void main(String[] args) {
try {
telnetProcess = new ProcessBuilder("C:\\Windows\\System32\\telnet.exe","x.x.x.x").start();
telnetInputReader = new BufferedReader(new InputStreamReader(telnetProcess.getInputStream()));
telnetOuputWriter = new BufferedWriter(new OutputStreamWriter(telnetProcess.getOutputStream()));
expectPattern("login:");
sendCmd("user");
expectPattern("password:");
sendCmd("pwd");
expectPattern("switch>#");
sendCmd("exit");
expectPattern("Connection to host lost");
} catch (IOException ex) {
System.out.println("Exception : " + ex);
}
}
I got the following error
java.io.IOException: Cannot run program "C:\Windows\System32\telnet.exe": CreateProcess error=2, The system cannot find the file specified
I tried to change the file path to unix formatted style like C:/Windows/System32/telnet.exe and no luck. (Though I expected it to not to work). Then copied the telnet.exe from it's location to some other user's home directory and I was not getting any errors. (???)
But, I didn't see the output as expected. I didn't get any response from the process and the code exited.
public static void sendCmd(String cmd) {
System.out.println(cmd);
try {
telnetOuputWriter.write(cmd + "\n", 0, cmd.length());
} catch (IOException ex) {
Logger.getLogger(TelnetProcessHandler.class.getName()).log(Level.SEVERE, null, ex);
}
}
public static String expectPattern(String pattern) {
String cmdResponse = "";
try {
String line = "";
// Always getting 'null' here
while ((line = telnetInputReader.readLine()) != null) {
System.out.println(line);
cmdResponse += line;
if (line.contains(pattern)) {
break;
}
}
} catch (IOException ex) {
System.out.println("ex : " + ex);
}
return cmdResponse;
}
What is wrong in this ? Then, one other query. I have tried using PrintWriter for writing to process which in turn has BufferedWriter in it, like,
telnetOuputWriter = new PrintWriter(new BufferedWriter(new OutputStreamWriter(telnetProcess.getOutputStream())));
Is this fine to use PrintWriter in this context ?
Note : Due to some reasons, I would like to stick with using telnet by means of process, not with Socket or TelnetClient or expect4j.

The telnet program does not use the standard input and output streams to communicate with the user, it needs to use the console device directly. You'll have to find an alternative way of doing what you're trying to do.
For example you could use a Java library that implements the telnet protocol. See this question for example: Open source Telnet Java API

How does Java circumvent the windows MAX_PATH WinAPI limitation

Does anyone know how Java is able to circumvent the windows MAX_PATH limitations. Using the below code I was able to create a really long path in Java and was able to perform I/O, which would have been impossible using windows without prefixing \\?\.
public static void main(String[] args) throws IOException {
BufferedWriter bufWriter = null;
try {
StringBuilder s = new StringBuilder();
for (int i = 0; i < 130; i++) {
s.append("asdf\\");
}
String filePath = "C:\\" + s.toString();;
System.out.println("File Path = " + filePath);
File f = new File(filePath);
f.mkdirs();
f = new File(f, "dummy.txt");
System.out.println("Full path = " + f);
bufWriter = new BufferedWriter(new FileWriter(f));
bufWriter.write("Hello");
}
catch (Exception e) {
e.printStackTrace();
}
finally {
if (bufWriter != null) {
bufWriter.close();
}
}
}

From the JVM's canonicalize_md.c:
/* copy \\?\ or \\?\UNC\ to the front of path*/
WCHAR* getPrefixed(const WCHAR* path, int pathlen) {
[download JVM source code (below) to see implementation]
}
The function getPrefixed is called:
by the function wcanonicalize if ((pathlen = wcslen(path)) > MAX_PATH - 1)
by the function wcanonicalizeWithPrefix.
I didn't trace the call chain farther than that, but I assume the JVM always uses these canonicalization routines before accessing the filesystem, and so always hits this code one way or another. If you want to trace the call chain farther yourself, you too can partake in the joys of browsing the JVM source code! Download at: http://download.java.net/openjdk/jdk6/

Windows bypasses that limitation if the path is prefixed with \\?\.

Most likely Java is in fact using UNC paths (\?) internally.

Java: opening and reading from a file without locking it

I need to be able to mimic 'tail -f' with Java. I'm trying to read a log file as it's being written by another process, but when I open the file to read it, it locks the file and the other process can't write to it anymore. Any help would be greatly appreciated!
Here is the code that I'm using currently:
public void read(){
Scanner fp = null;
try{
fp = new Scanner(new FileReader(this.filename));
fp.useDelimiter("\n");
}catch(java.io.FileNotFoundException e){
System.out.println("java.io.FileNotFoundException e");
}
while(true){
if(fp.hasNext()){
this.parse(fp.next());
}
}
}

Rebuilding tail is tricky due to some special cases like file truncation and (intermediate) deletion. To open the file without locking, use StandardOpenOption.READ with the new Java file API:
try (InputStream is = Files.newInputStream(path, StandardOpenOption.READ)) {
InputStreamReader reader = new InputStreamReader(is, fileEncoding);
BufferedReader lineReader = new BufferedReader(reader);
// Process all lines.
String line;
while ((line = lineReader.readLine()) != null) {
// Line content content is in variable line.
}
}
For my attempt to create a tail in Java see:
Method examineFile(…) in https://github.com/AugustusKling/yield/blob/master/src/main/java/yield/input/file/FileMonitor.java
The above is used by https://github.com/AugustusKling/yield/blob/master/src/main/java/yield/input/file/FileInput.java to create a tail operation. The queue.feed(lineContent) passes line content for processing by listeners and would equal your this.parse(…).
Feel free to take inspiration from that code or simply copy the parts you require. Let me know if you find any issues that I'm not aware of.

Look at the FileChannel API here. For locking the file you can check here

java.io gives you a mandatory file lock and java.nio gives you an
advisory file lock
If you want to read any file without any lock you can use below classes
import java.nio.channels.FileChannel;
import java.nio.file.Paths;
If you want to tail a file line by line use below code for the same
public void tail(String logPath){
String logStr = null;
FileChannel fc = null;
try {
fc = FileChannel.open(Paths.get(logPath), StandardOpenOption.READ);
fc.position(fc.size());
} catch (FileNotFoundException e1) {
System.out.println("FileNotFoundException occurred in Thread : " + Thread.currentThread().getName());
return;
} catch (IOException e) {
System.out.println("IOException occurred while opening FileChannel in Thread : " + Thread.currentThread().getName());
}
while (true) {
try {
logStr = readLine(fc);
if (logStr != null) {
System.out.println(logStr);
} else {
Thread.sleep(1000);
}
} catch (IOException|InterruptedException e) {
System.out.println("Exception occurred in Thread : " + Thread.currentThread().getName());
try {
fc.close();
} catch (IOException e1) {
}
break;
}
}
}
private String readLine(FileChannel fc) throws IOException {
ByteBuffer buffers = ByteBuffer.allocate(128);
// Standard size of a line assumed to be 128 bytes
long lastPos = fc.position();
if (fc.read(buffers) > 0) {
byte[] data = buffers.array();
boolean foundTmpTerminator = false;
boolean foundTerminator = false;
long endPosition = 0;
for (byte nextByte : data) {
endPosition++;
switch (nextByte) {
case -1:
foundTerminator = true;
break;
case (byte) '\r':
foundTmpTerminator = true;
break;
case (byte) '\n':
foundTmpTerminator = true;
break;
default:
if (foundTmpTerminator) {
endPosition--;
foundTerminator = true;
}
}
if (foundTerminator) {
break;
}
}
fc.position(lastPos + endPosition);
if (foundTerminator) {
return new String(data, 0, (int) endPosition);
} else {
return new String(data, 0, (int) endPosition) + readLine(fc);
}
}
return null;
}

Windows uses mandatory locking for files unless you specify the right share flags while you open. If you want to open a busy file, you need to Win32-API CreateFile a handle with the sharing flags FILE_SHARE_DELETE | FILE_SHARE_READ | FILE_SHARE_WRITE.
This is used inside the JDK in a few places to open files for reading attributes and stuff, but as far as I can see it is not exported/available to Java Class Library level. So you would need to find a native library to do that.
I think as a quick work around you can read process.getInputStream() from the command "cmd /D/C type file.lck"