We Keep Coding

Java FileWriter thread safety [duplicate]

This question already has answers here:
Threads and file writing
(6 answers)
Closed 9 years ago.
i have 20 threads that write with the println() function on a file called results.txt. How can i synchronize them all?
I note every time my program run i have different number of lines of text in results.txt.
Thank you.

Access the file through a class that contains a synchronized method to write to the file. Only one thread at a time will be able to execute the method.
I think that Singleton pattern would fit for your problem:
package com.test.singleton;
public class Singleton {
private static final Singleton inst= new Singleton();
private Singleton() {
super();
}
public synchronized void writeToFile(String str) {
// Do whatever
}
public static Singleton getInstance() {
return inst;
}
}
Every time you need to write to your file, you only would have to call:
Singleton.getInstance().writeToFile("Hello!!");

Duplicate question ... duplicate answer. As I said here:
If you can hold your file as a FileOutputStream you can lock it like this:
FileOutputStream file = ...
....
// Thread safe version.
void write(byte[] bytes) {
try {
boolean written = false;
do {
try {
// Lock it!
FileLock lock = file.getChannel().lock();
try {
// Write the bytes.
file.write(bytes);
written = true;
} finally {
// Release the lock.
lock.release();
}
} catch ( OverlappingFileLockException ofle ) {
try {
// Wait a bit
Thread.sleep(0);
} catch (InterruptedException ex) {
throw new InterruptedIOException ("Interrupted waiting for a file lock.");
}
}
} while (!written);
} catch (IOException ex) {
log.warn("Failed to lock " + fileName, ex);
}
}

You are intend to write data into one file. So if you try to lock the whole file, it'd better to use a single thread to do this job. Although you spawn 20 threads, but there is only one of them is running every time you call the method, the others are just waiting for the lock.
I recommend you use RandomAccessFile to write data to your file. Then each thread can write some unique data into to the file without locking the whole file.
Some demo code as following
try {
final RandomAccessFile file = new RandomAccessFile("/path/to/your/result.txt", "rw");
final int numberOfThread = 20;
final int bufferSize = 512;
ExecutorService pool = Executors.newFixedThreadPool(numberOfThread);
final AtomicInteger byteCounter = new AtomicInteger(0);
final byte[] yourText = "Your data".getBytes();
for (int i = 0; i < yourText.length; i++) {
pool.submit(new Runnable() {
#Override
public void run() {
int start = byteCounter.getAndAdd(bufferSize);
int chunkSize = bufferSize;
if (start + bufferSize > yourText.length) {
chunkSize = yourText.length - start;
}
byte[] chunkData = new byte[chunkSize];
System.arraycopy(yourText, start, chunkData, 0, chunkSize);
try {
file.write(chunkData);
} catch (IOException e) {
//exception handle
}
}
});
}
file.close();
} catch (Exception e) {
//clean up
}

Write to text file from multiple threads? [duplicate]

This question already has answers here:
Threads and file writing
(6 answers)
Closed 9 years ago.
i have 20 threads that write with the println() function on a file called results.txt. How can i synchronize them all?
I note every time my program run i have different number of lines of text in results.txt.
Thank you.

Access the file through a class that contains a synchronized method to write to the file. Only one thread at a time will be able to execute the method.
I think that Singleton pattern would fit for your problem:
package com.test.singleton;
public class Singleton {
private static final Singleton inst= new Singleton();
private Singleton() {
super();
}
public synchronized void writeToFile(String str) {
// Do whatever
}
public static Singleton getInstance() {
return inst;
}
}
Every time you need to write to your file, you only would have to call:
Singleton.getInstance().writeToFile("Hello!!");

Duplicate question ... duplicate answer. As I said here:
If you can hold your file as a FileOutputStream you can lock it like this:
FileOutputStream file = ...
....
// Thread safe version.
void write(byte[] bytes) {
try {
boolean written = false;
do {
try {
// Lock it!
FileLock lock = file.getChannel().lock();
try {
// Write the bytes.
file.write(bytes);
written = true;
} finally {
// Release the lock.
lock.release();
}
} catch ( OverlappingFileLockException ofle ) {
try {
// Wait a bit
Thread.sleep(0);
} catch (InterruptedException ex) {
throw new InterruptedIOException ("Interrupted waiting for a file lock.");
}
}
} while (!written);
} catch (IOException ex) {
log.warn("Failed to lock " + fileName, ex);
}
}

You are intend to write data into one file. So if you try to lock the whole file, it'd better to use a single thread to do this job. Although you spawn 20 threads, but there is only one of them is running every time you call the method, the others are just waiting for the lock.
I recommend you use RandomAccessFile to write data to your file. Then each thread can write some unique data into to the file without locking the whole file.
Some demo code as following
try {
final RandomAccessFile file = new RandomAccessFile("/path/to/your/result.txt", "rw");
final int numberOfThread = 20;
final int bufferSize = 512;
ExecutorService pool = Executors.newFixedThreadPool(numberOfThread);
final AtomicInteger byteCounter = new AtomicInteger(0);
final byte[] yourText = "Your data".getBytes();
for (int i = 0; i < yourText.length; i++) {
pool.submit(new Runnable() {
#Override
public void run() {
int start = byteCounter.getAndAdd(bufferSize);
int chunkSize = bufferSize;
if (start + bufferSize > yourText.length) {
chunkSize = yourText.length - start;
}
byte[] chunkData = new byte[chunkSize];
System.arraycopy(yourText, start, chunkData, 0, chunkSize);
try {
file.write(chunkData);
} catch (IOException e) {
//exception handle
}
}
});
}
file.close();
} catch (Exception e) {
//clean up
}

To check if a file is written completely

How do I know if a software is done writing a file if I am executing that software from java?For example, I am executing geniatagger.exe with an input file RawText that will produce an output file TAGGEDTEXT.txt. When geniatagger.exe is finished writing the TAGGEDTEXT.txt file, I can do some other staffs with this file. The problem is- how can I know that geniatagger is finished writing the text file?
try{
Runtime rt = Runtime.getRuntime();
Process p = rt.exec("geniatagger.exe -i "+ RawText+ " -o TAGGEDTEXT.txt");
}

You can't, or at least not reliably.
In this particular case your best bet is to watch the Process complete.
You get the process' return code as a bonus, this could tell you if an error occurred.
If you are actually talking about this GENIA tagger, below is a practical example which demonstrates various topics (see explanation about numbered comments beneath the code). The code was tested with v1.0 for Linux and demonstrates how to safely run a process which expects both input and output stream piping to work correctly.
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.util.concurrent.Callable;
import org.apache.commons.io.IOUtils;
public class GeniaTagger {
/**
* #param args
*/
public static void main(String[] args) {
tagFile(new File("inputText.txt"), new File("outputText.txt"));
}
public static void tagFile(File input, File output) {
FileInputStream ifs = null;
FileOutputStream ofs = null;
try {
ifs = new FileInputStream(input);
ofs = new FileOutputStream(output);
final FileInputStream ifsRef = ifs;
final FileOutputStream ofsRef = ofs;
// {1}
ProcessBuilder pb = new ProcessBuilder("geniatagger.exe");
final Process pr = pb.start();
// {2}
runInThread(new Callable<Void>() {
public Void call() throws Exception {
IOUtils.copy(ifsRef, pr.getOutputStream());
IOUtils.closeQuietly(pr.getOutputStream()); // {3}
return null;
}
});
runInThread(new Callable<Void>() {
public Void call() throws Exception {
IOUtils.copy(pr.getInputStream(), ofsRef); // {4}
return null;
}
});
runInThread(new Callable<Void>() {
public Void call() throws Exception {
IOUtils.copy(pr.getErrorStream(), System.err);
return null;
}
});
// {5}
pr.waitFor();
// output file is written at this point.
} catch (Exception e) {
e.printStackTrace();
} finally {
// {6}
IOUtils.closeQuietly(ifs);
IOUtils.closeQuietly(ofs);
}
}
public static void runInThread(final Callable<?> c) {
new Thread() {
public void run() {
try {
c.call();
} catch (Exception e) {
e.printStackTrace();
} finally {
}
}
}.start();
}
}
Use a ProcessBuilder to start your process, it has a better interface than plain-old Runtime.getRuntime().exec(...).
Set up stream piping in different threads, otherwhise the waitFor() call in ({5}) might never complete.
Note that I piped a FileInputStream to the process. According to the afore-mentioned GENIA page, this command expects actual input instead of a -i parameter. The OutputStream which connects to the process must be closed, otherwhise the program will keep running!
Copy the result of the process to a FileOutputStream, the result file your are waiting for.
Let the main thread wait until the process completes.
Clean up all streams.

If the program exits after generating the output file then you can call Process.waitFor() to let it run to completion then you can process the file. Note that you will likely have to drain both the standard output and error streams (at least on Windows) for the process to finish.
[Edit]
Here is an example, untested and likely fraught with problems:
// ...
Process p = rt.exec("geniatagger.exe -i "+ RawText+ " -o TAGGEDTEXT.txt");
drain(p.getInputStream());
drain(p.getErrorStream());
int exitCode = p.waitFor();
// Now you should be able to process the output file.
}
private static void drain(InputStream in) throws IOException {
while (in.read() != -1);
}

How could I read Java Console Output into a String buffer

I have a Java program that outputs some text into console. It uses print, println, and some other methods to do this.
At the end of the program , I want to read all the text in console and copy it into a String buffer. How could I do this in Java ? I need to read stdout and stderr separately.

Ok, this was a fun problem. Dosen't seem to be an elegant way of solving it for all PrintStream methods at once. (Unfortunately there is no FilterPrintStream.)
I did write up an ugly reflection-based workaround though (not to be used in production code I suppose :)
class LoggedPrintStream extends PrintStream {
final StringBuilder buf;
final PrintStream underlying;
LoggedPrintStream(StringBuilder sb, OutputStream os, PrintStream ul) {
super(os);
this.buf = sb;
this.underlying = ul;
}
public static LoggedPrintStream create(PrintStream toLog) {
try {
final StringBuilder sb = new StringBuilder();
Field f = FilterOutputStream.class.getDeclaredField("out");
f.setAccessible(true);
OutputStream psout = (OutputStream) f.get(toLog);
return new LoggedPrintStream(sb, new FilterOutputStream(psout) {
public void write(int b) throws IOException {
super.write(b);
sb.append((char) b);
}
}, toLog);
} catch (NoSuchFieldException shouldNotHappen) {
} catch (IllegalArgumentException shouldNotHappen) {
} catch (IllegalAccessException shouldNotHappen) {
}
return null;
}
}
...that can be used like this:
public class Test {
public static void main(String[] args) {
// Create logged PrintStreams
LoggedPrintStream lpsOut = LoggedPrintStream.create(System.out);
LoggedPrintStream lpsErr = LoggedPrintStream.create(System.err);
// Set them to stdout / stderr
System.setOut(lpsOut);
System.setErr(lpsErr);
// Print some stuff
System.out.print("hello ");
System.out.println(5);
System.out.flush();
System.err.println("Some error");
System.err.flush();
// Restore System.out / System.err
System.setOut(lpsOut.underlying);
System.setErr(lpsErr.underlying);
// Print the logged output
System.out.println("----- Log for System.out: -----\n" + lpsOut.buf);
System.out.println("----- Log for System.err: -----\n" + lpsErr.buf);
}
}
Resulting output:
hello 5
Some error
----- Log for System.out: -----
hello 5
----- Log for System.err: -----
Some error
(Note though, that the out field in FilterOutputStream is protected and documented, so it is part of the API :-)

You can't do that once the program is finished running. You need to do it before the program starts to write output.
See this article(archive.org) for details on how to replace stdout and stderr. The core calls are System.setOut() and System.setErr().

You can use PipedInputStream and PipedOutputStream.
//create pairs of Piped input and output streasm for std out and std err
final PipedInputStream outPipedInputStream = new PipedInputStream();
final PrintStream outPrintStream = new PrintStream(new PipedOutputStream(
outPipedInputStream));
final BufferedReader outReader = new BufferedReader(
new InputStreamReader(outPipedInputStream));
final PipedInputStream errPipedInputStream = new PipedInputStream();
final PrintStream errPrintStream = new PrintStream(new PipedOutputStream(
errPipedInputStream));
final BufferedReader errReader = new BufferedReader(
new InputStreamReader(errPipedInputStream));
final PrintStream originalOutStream = System.out;
final PrintStream originalErrStream = System.err;
final Thread writingThread = new Thread(new Runnable() {
#Override
public void run() {
try {
System.setOut(outPrintStream);
System.setErr(errPrintStream);
// You could also set the System.in here using a
// PipedInputStream
DoSomething();
// Even better would be to refactor DoSomething to accept
// PrintStream objects as parameters to replace all uses of
// System.out and System.err. DoSomething could also have
// an overload with DoSomething() calling:
DoSomething(outPrintStream, errPrintStream);
} finally {
// may also want to add a catch for exceptions but it is
// essential to restore the original System output and error
// streams since it can be very confusing to not be able to
// find System.out output on your console
System.setOut(originalOutStream);
System.setErr(originalErrStream);
//You must close the streams which will auto flush them
outPrintStream.close();
errPrintStream.close();
}
} // end run()
}); // end writing thread
//Start the code that will write into streams
writingThread.start();
String line;
final List<String> completeOutputStreamContent = new ArrayList<String>();
while ((line = outReader.readLine()) != null) {
completeOutputStreamContent.add(line);
} // end reading output stream
final List<String> completeErrorStreamContent = new ArrayList<String>();
while ((line = errReader.readLine()) != null) {
completeErrorStreamContent.add(line);
} // end reading output stream

Here is a utility Class named ConsoleOutputCapturer. It allows the output to go to the existing console however behind the scene keeps capturing the output text. You can control what to capture with the start/stop methods. In other words call start to start capturing the console output and once you are done capturing you can call the stop method which returns a String value holding the console output for the time window between start-stop calls. This class is not thread-safe though.
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.io.PrintStream;
import java.util.Arrays;
import java.util.List;
public class ConsoleOutputCapturer {
private ByteArrayOutputStream baos;
private PrintStream previous;
private boolean capturing;
public void start() {
if (capturing) {
return;
}
capturing = true;
previous = System.out;
baos = new ByteArrayOutputStream();
OutputStream outputStreamCombiner =
new OutputStreamCombiner(Arrays.asList(previous, baos));
PrintStream custom = new PrintStream(outputStreamCombiner);
System.setOut(custom);
}
public String stop() {
if (!capturing) {
return "";
}
System.setOut(previous);
String capturedValue = baos.toString();
baos = null;
previous = null;
capturing = false;
return capturedValue;
}
private static class OutputStreamCombiner extends OutputStream {
private List<OutputStream> outputStreams;
public OutputStreamCombiner(List<OutputStream> outputStreams) {
this.outputStreams = outputStreams;
}
public void write(int b) throws IOException {
for (OutputStream os : outputStreams) {
os.write(b);
}
}
public void flush() throws IOException {
for (OutputStream os : outputStreams) {
os.flush();
}
}
public void close() throws IOException {
for (OutputStream os : outputStreams) {
os.close();
}
}
}
}

Don't do it afterwards, create two StringBuilder objects before the first System.out.print() gets called and just append every string you want to save to the appropriate StringBuilder.

These two line of code will put your output in a text file or u can change the destination as u require.
// Create a file:
System.setOut(new PrintStream( new FileOutputStream("D:/MyOutputFile.txt")));
// Redirect the output to the file:
System.out.println("Hello to custom output stream!");
hope its help u .. :)

How to Cache InputStream for Multiple Use

I have an InputStream of a file and i use apache poi components to read from it like this:
POIFSFileSystem fileSystem = new POIFSFileSystem(inputStream);
The problem is that i need to use the same stream multiple times and the POIFSFileSystem closes the stream after use.
What is the best way to cache the data from the input stream and then serve more input streams to different POIFSFileSystem ?
EDIT 1:
By cache i meant store for later use, not as a way to speedup the application. Also is it better to just read up the input stream into an array or string and then create input streams for each use ?
EDIT 2:
Sorry to reopen the question, but the conditions are somewhat different when working inside desktop and web application.
First of all, the InputStream i get from the org.apache.commons.fileupload.FileItem in my tomcat web app doesn't support markings thus cannot reset.
Second, I'd like to be able to keep the file in memory for faster acces and less io problems when dealing with files.

you can decorate InputStream being passed to POIFSFileSystem with a version that when close() is called it respond with reset():
class ResetOnCloseInputStream extends InputStream {
private final InputStream decorated;
public ResetOnCloseInputStream(InputStream anInputStream) {
if (!anInputStream.markSupported()) {
throw new IllegalArgumentException("marking not supported");
}
anInputStream.mark( 1 << 24); // magic constant: BEWARE
decorated = anInputStream;
}
#Override
public void close() throws IOException {
decorated.reset();
}
#Override
public int read() throws IOException {
return decorated.read();
}
}
testcase
static void closeAfterInputStreamIsConsumed(InputStream is)
throws IOException {
int r;
while ((r = is.read()) != -1) {
System.out.println(r);
}
is.close();
System.out.println("=========");
}
public static void main(String[] args) throws IOException {
InputStream is = new ByteArrayInputStream("sample".getBytes());
ResetOnCloseInputStream decoratedIs = new ResetOnCloseInputStream(is);
closeAfterInputStreamIsConsumed(decoratedIs);
closeAfterInputStreamIsConsumed(decoratedIs);
closeAfterInputStreamIsConsumed(is);
}
EDIT 2
you can read the entire file in a byte[] (slurp mode) then passing it to a ByteArrayInputStream

Try BufferedInputStream, which adds mark and reset functionality to another input stream, and just override its close method:
public class UnclosableBufferedInputStream extends BufferedInputStream {
public UnclosableBufferedInputStream(InputStream in) {
super(in);
super.mark(Integer.MAX_VALUE);
}
#Override
public void close() throws IOException {
super.reset();
}
}
So:
UnclosableBufferedInputStream bis = new UnclosableBufferedInputStream (inputStream);
and use bis wherever inputStream was used before.

This works correctly:
byte[] bytes = getBytes(inputStream);
POIFSFileSystem fileSystem = new POIFSFileSystem(new ByteArrayInputStream(bytes));
where getBytes is like this:
private static byte[] getBytes(InputStream is) throws IOException {
byte[] buffer = new byte[8192];
ByteArrayOutputStream baos = new ByteArrayOutputStream(2048);
int n;
baos.reset();
while ((n = is.read(buffer, 0, buffer.length)) != -1) {
baos.write(buffer, 0, n);
}
return baos.toByteArray();
}

Use below implementation for more custom use -
public class ReusableBufferedInputStream extends BufferedInputStream
{
private int totalUse;
private int used;
public ReusableBufferedInputStream(InputStream in, Integer totalUse)
{
super(in);
if (totalUse > 1)
{
super.mark(Integer.MAX_VALUE);
this.totalUse = totalUse;
this.used = 1;
}
else
{
this.totalUse = 1;
this.used = 1;
}
}
#Override
public void close() throws IOException
{
if (used < totalUse)
{
super.reset();
++used;
}
else
{
super.close();
}
}
}

What exactly do you mean with "cache"? Do you want the different POIFSFileSystem to start at the beginning of the stream? If so, there's absolutely no point caching anything in your Java code; it will be done by the OS, just open a new stream.
Or do you wan to continue reading at the point where the first POIFSFileSystem stopped? That's not caching, and it's very difficult to do. The only way I can think of if you can't avoid the stream getting closed would be to write a thin wrapper that counts how many bytes have been read and then open a new stream and skip that many bytes. But that could fail when POIFSFileSystem internally uses something like a BufferedInputStream.

If the file is not that big, read it into a byte[] array and give POI a ByteArrayInputStream created from that array.
If the file is big, then you shouldn't care, since the OS will do the caching for you as best as it can.
[EDIT] Use Apache commons-io to read the File into a byte array in an efficient way. Do not use int read() since it reads the file byte by byte which is very slow!
If you want to do it yourself, use a File object to get the length, create the array and the a loop which reads bytes from the file. You must loop since read(byte[], int offset, int len) can read less than len bytes (and usually does).

This is how I would implemented, to be safely used with any InputStream :
write your own InputStream wrapper where you create a temporary file to mirror the original stream content
dump everything read from the original input stream into this temporary file
when the stream was completely read you will have all the data mirrored in the temporary file
use InputStream.reset to switch(initialize) the internal stream to a FileInputStream(mirrored_content_file)
from now on you will loose the reference of the original stream(can be collected)
add a new method release() which will remove the temporary file and release any open stream.
you can even call release() from finalize to be sure the temporary file is release in case you forget to call release()(most of the time you should avoid using finalize, always call a method to release object resources). see Why would you ever implement finalize()?

public static void main(String[] args) throws IOException {
BufferedInputStream inputStream = new BufferedInputStream(IOUtils.toInputStream("Foobar"));
inputStream.mark(Integer.MAX_VALUE);
System.out.println(IOUtils.toString(inputStream));
inputStream.reset();
System.out.println(IOUtils.toString(inputStream));
}
This works. IOUtils is part of commons IO.

This answer iterates on previous ones 1|2 based on the BufferInputStream. The main changes are that it allows infinite reuse. And takes care of closing the original source input stream to free-up system resources. Your OS defines a limit on those and you don't want the program to run out of file handles (That's also why you should always 'consume' responses e.g. with the apache EntityUtils.consumeQuietly()). EDIT Updated the code to handle for gready consumers that use read(buffer, offset, length), in that case it may happen that BufferedInputStream tries hard to look at the source, this code protects against that use.
public class CachingInputStream extends BufferedInputStream {
public CachingInputStream(InputStream source) {
super(new PostCloseProtection(source));
super.mark(Integer.MAX_VALUE);
}
#Override
public synchronized void close() throws IOException {
if (!((PostCloseProtection) in).decoratedClosed) {
in.close();
}
super.reset();
}
private static class PostCloseProtection extends InputStream {
private volatile boolean decoratedClosed = false;
private final InputStream source;
public PostCloseProtection(InputStream source) {
this.source = source;
}
#Override
public int read() throws IOException {
return decoratedClosed ? -1 : source.read();
}
#Override
public int read(byte[] b) throws IOException {
return decoratedClosed ? -1 : source.read(b);
}
#Override
public int read(byte[] b, int off, int len) throws IOException {
return decoratedClosed ? -1 : source.read(b, off, len);
}
#Override
public long skip(long n) throws IOException {
return decoratedClosed ? 0 : source.skip(n);
}
#Override
public int available() throws IOException {
return source.available();
}
#Override
public void close() throws IOException {
decoratedClosed = true;
source.close();
}
#Override
public void mark(int readLimit) {
source.mark(readLimit);
}
#Override
public void reset() throws IOException {
source.reset();
}
#Override
public boolean markSupported() {
return source.markSupported();
}
}
}
To reuse it just close it first if it wasn't.
One limitation though is that if the stream is closed before the whole content of the original stream has been read, then this decorator will have incomplete data, so make sure the whole stream is read before closing.

I just add my solution here, as this works for me. It basically is a combination of the top two answers :)
private String convertStreamToString(InputStream is) {
Writer w = new StringWriter();
char[] buf = new char[1024];
Reader r;
is.mark(1 << 24);
try {
r = new BufferedReader(new InputStreamReader(is, "UTF-8"));
int n;
while ((n=r.read(buf)) != -1) {
w.write(buf, 0, n);
}
is.reset();
} catch(UnsupportedEncodingException e) {
Logger.debug(this.getClass(), "Cannot convert stream to string.", e);
} catch(IOException e) {
Logger.debug(this.getClass(), "Cannot convert stream to string.", e);
}
return w.toString();
}