We Keep Coding

Exception propagation within PipedInputStream and PipedOutputStream

I have a data producer that runs in a separate thread and pushes generated data into PipedOutputStream which is connected to PipedInputStream. A reference of this input stream is exposed via public API so that any client can use it. The PipedInputStream contains a limited buffer which, if full, blocks the data producer. Basically, as the client reads data from the input stream, new data is generated by the data producer.
The problem is that the data producer may fail and throw an exception. But as the consumer is running in a separate thread, there is no nice way to get the exception to the client.
What I do is that I catch that exception and close the input stream. That will result in a IOException with message "Pipe closed" on the client side but I would really like to give the client the real reason behind that.
This is a rough code of my API:
public InputStream getData() {
final PipedInputStream inputStream = new PipedInputStream(config.getPipeBufferSize());
final PipedOutputStream outputStream = new PipedOutputStream(inputStream);
Thread thread = new Thread(() -> {
try {
// Start producing the data and push it into output stream.
// The production my fail and throw an Exception with the reason
} catch (Exception e) {
try {
// What to do here?
outputStream.close();
inputStream.close();
} catch (IOException e1) {
}
}
});
thread.start();
return inputStream;
}
I have two ideas how to fix that:
Store the exception in the parent object and expose it to the client via API. I. e. if the reading fails with an IOException, the client could ask the API for the reason.
Extend / re-implement the piped streams so that I could pass a reason to the close() method. Then the IOException thrown by the stream could contain that reason as a message.
Any better ideas?

Coincidentally I just wrote similar code to allow GZip compression of a stream. You don't need to extend PipedInputStream, just FilterInputStream will do and return a wrapped version, e.g.
final PipedInputStream in = new PipedInputStream();
final InputStreamWithFinalExceptionCheck inWithException = new InputStreamWithFinalExceptionCheck(in);
final PipedOutputStream out = new PipedOutputStream(in);
Thread thread = new Thread(() -> {
try {
// Start producing the data and push it into output stream.
// The production my fail and throw an Exception with the reason
} catch (final IOException e) {
inWithException.fail(e);
} finally {
inWithException.countDown();
}
});
thread.start();
return inWithException;
And then InputStreamWithFinalExceptionCheck is just
private static final class InputStreamWithFinalExceptionCheck extends FilterInputStream {
private final AtomicReference<IOException> exception = new AtomicReference<>(null);
private final CountDownLatch complete = new CountDownLatch(1);
public InputStreamWithFinalExceptionCheck(final InputStream stream) {
super(stream);
}
#Override
public void close() throws IOException {
try {
complete.await();
final IOException e = exception.get();
if (e != null) {
throw e;
}
} catch (final InterruptedException e) {
throw new IOException("Interrupted while waiting for synchronised closure");
} finally {
stream.close();
}
}
public void fail(final IOException e) {
exception.set(Preconditions.checkNotNull(e));
}
public void countDown() {complete.countDown();}
}

This is my implementation, taken from above accepted answer https://stackoverflow.com/a/33698661/5165540 , where I don't use the CountDownLatch complete.await() as it would cause a deadlock if the InputStream gets abruptly closed before the writer has finished writing the full content.
I still set the exception caught when PipedOutpuStream is being used, and I create the PipedOutputStream in the spawn thread, using a try-finally-resource pattern to ensure it gets closed, waiting in the Supplier until the 2 streams are piped.
Supplier<InputStream> streamSupplier = new Supplier<InputStream>() {
#Override
public InputStream get() {
final AtomicReference<IOException> osException = new AtomicReference<>();
final CountDownLatch piped = new CountDownLatch(1);
final PipedInputStream is = new PipedInputStream();
FilterInputStream fis = new FilterInputStream(is) {
#Override
public void close() throws IOException {
try {
IOException e = osException.get();
if (e != null) {
//Exception thrown by the write will bubble up to InputStream reader
throw new IOException("IOException in writer", e);
}
} finally {
super.close();
}
};
};
Thread t = new Thread(() -> {
try (PipedOutputStream os = new PipedOutputStream(is)) {
piped.countDown();
writeIozToStream(os, projectFile, dataFolder);
} catch (final IOException e) {
osException.set(e);
}
});
t.start();
try {
piped.await();
} catch (InterruptedException e) {
t.cancel();
Thread.currentThread().interrupt();
}
return fis;
}
};
Calling code is something like
try (InputStream is = streamSupplier.getInputStream()) {
//Read stream in full
}
So when is InputStream is closed this will be signaled in the PipedOutputStream causing eventually a "Pipe closed" IOException, ignored at that point.
If I keep instead the complete.await() line in the FilterInputStreamclose() I could suffer from deadlock (PipedInputStream trying to close, waiting on complete.await(), while PipedOutputStream is waiting forever on PipedInputStreamawaitSpace )

How to synchronize multiple threads writing into a file

I am using a ExecutorService to have multiple threads writing text into a file, but i cannot manage to synchronize the run() method and instead of having the proper line by line String i ask, i have a mixup of all the characters of the Strings because they write it at the same time.
import java.io.BufferedReader
...
class WriteDns implements Runnable {
File file;
String text;
WriteDns(File file, String text) {
this.file = file;
this.text = text;
}
public void run() {
synchronized (this) {
try (BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(
new FileOutputStream(file)))) {
bw.write(turnDns() + "\n");
} catch (IOException e) {
System.out.println("Error");
}
}
}
public String turnDns() {
int space = text.indexOf(' ');
String ip = text.substring(0, space);
String theRest = text.substring(space);
String temp = ip;
try {
ip = InetAddress.getByName(ip).getHostName();
if (ip == temp)
return "NotFound " + theRest;
return ip + " " + theRest;
} catch (UnknownHostException e) {
System.out.println("Error in change");
return "-changeErr " + theRest;
}
}
}
public class Main00 {
static File oldFile = new File("oldfile.txt");
public static void main(String[] args) {
readLines();
}
public static void readLines() {
try (BufferedReader br = new BufferedReader(new FileReader(oldFile))) {
File f = new File("file.txt");
ExecutorService service = Executors.newFixedThreadPool(10);
for (String t = br.readLine(); t != null; t = br.readLine()) {
service.execute(new WriteDns(f, t));
}
service.shutdown();
} catch (IOException e) {
e.printStackTrace();
}
}
}

You're synchronising on this but you're making a new instance of your thread worker for every thread, so each thread is locking on itself and never waiting for any other threads. You need to lock on an object that is visible to all threads, perhaps a static object or pass in a lock object when you instantiate your WriteDns.
With that said, having multiple threads open on one file is inherently prone to problems like you're experiencing, and you gain nothing from multiple threads writing since your bottleneck is your storage medium and not your processor. You should rather have multiple threads providing information/data to one dedicated writer thread that has exclusive access to the file you want to write to, as #FlorianSchaetz suggested.

read and write files in java using separate threads

I have created two threads and modified the run function so that one thread reads one line and the other writes the same line to the new file. This happens till the whole file is copied. The problem i am getting is that even though i have used variables to control that the threads execute one by one but still the threads are executing unevenly i.e one thread executes multiple times and then the control transfers. Any solutions i have attached the code. I am new to java as it is only for class assignment so the code might not be the most optimized.
public class thread1 extends Thread {
//To create producer and consumer as threads
//Shared variable
public static int x = 0;//checks if all lines are read
public static String line; /holds lines from file
public static int j = 0;//variable to switch between threads based upon its value
public thread1(String threadName) { //Constuctor
super(threadName); //Call to constructor of Thread class
}
public void run() {
while (x != -1)
{
if (Thread.currentThread().getName().contains("Reader")) {
if (x != -1&&j==0)
{
j=1;
String fileName = "d:/salfar.txt";
try {
// FileReader reads text files in the default encoding.
FileReader fileReader =
new FileReader(fileName);
// Always wrap FileReader in BufferedReader.
BufferedReader bufferedReader =
new BufferedReader(fileReader);
for (int check = 0; check <= x; check++) {
line = bufferedReader.readLine();
}
if (line == null) {
x = -1;
} else {
System.out.println(line);
x++;
}
// Always close files.
bufferedReader.close();
} catch (FileNotFoundException ex) {
System.out.println(
"Unable to open file '"
+ fileName + "'");
} catch (IOException ex) {
System.out.println(
"Error reading file '"
+ fileName + "'");
// Or we could just do this:
// ex.printStackTrace();
}
}
yield();
}
else if (Thread.currentThread().getName().contains("writer")) {
if (x != -1 && line != null&&j==1)
{
j=0;
String fileName = "d:/salfar1.txt";
try {
// Assume default encoding.
FileWriter fileWriter =
new FileWriter(fileName, true);
// Always wrap FileWriter in BufferedWriter.
BufferedWriter bufferedWriter =
new BufferedWriter(fileWriter);
// Note that write() does not automatically
// append a newline character.
bufferedWriter.write(line);
bufferedWriter.newLine();
System.out.println("y");
// Always close files.
bufferedWriter.close();
} catch (IOException ex) {
System.out.println(
"Error writing to file '"
+ fileName + "'");
// Or we could just do this:
// ex.printStackTrace();
}
}
Thread.yield();
}
else{}
}
}
public static void main(String[] args) {
thread1 p = new thread1("Reader");
thread1 c = new thread1("writer");
p.start();
c.start();
}
}
Thanks

You cannot control the order of thread execution. However, to perform read and write operation via separate threads, you should use BlockingQueue which has the following properties:
A Queue that additionally supports operations that wait for the queue to become non-empty when retrieving an element, and wait for space to become available in the queue when storing an element.
ReaderThread will read from the input file.
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.util.concurrent.BlockingQueue;
public class ReaderThread implements Runnable{
protected BlockingQueue<String> blockingQueue = null;
public ReaderThread(BlockingQueue<String> blockingQueue){
this.blockingQueue = blockingQueue;
}
#Override
public void run() {
BufferedReader br = null;
try {
br = new BufferedReader(new FileReader(new File("./inputFile.txt")));
String buffer =null;
while((buffer=br.readLine())!=null){
blockingQueue.put(buffer);
}
blockingQueue.put("EOF"); //When end of file has been reached
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} catch(InterruptedException e){
}finally{
try {
br.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
WriterThread will write to output file.
import java.io.File;
import java.io.FileNotFoundException;
import java.io.PrintWriter;
import java.util.concurrent.BlockingQueue;
public class WriterThread implements Runnable{
protected BlockingQueue<String> blockingQueue = null;
public WriterThread(BlockingQueue<String> blockingQueue){
this.blockingQueue = blockingQueue;
}
#Override
public void run() {
PrintWriter writer = null;
try {
writer = new PrintWriter(new File("outputFile.txt"));
while(true){
String buffer = blockingQueue.take();
//Check whether end of file has been reached
if(buffer.equals("EOF")){
break;
}
writer.println(buffer);
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch(InterruptedException e){
}finally{
writer.close();
}
}
}
From Launcher class start your multithreaded read and write.
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.BlockingQueue;
public class Launcher {
public static void main(String[] args) {
BlockingQueue<String> queue = new ArrayBlockingQueue<String>(1024);
ReaderThread reader = new ReaderThread(queue);
WriterThread writer = new WriterThread(queue);
new Thread(reader).start();
new Thread(writer).start();
}
}

Here is my solutions. My idea is to use the actually file name that our threads will be reading and writing to. There is only one issue that we need to make sure of, that no two threads are trying to operate on the same file. The solution to this is to simply have a synchronized code block in your run method.
We must recall that Strings are not mutable in Java. Consider the following:
String s1 = "test.txt";
String s2 = "test.txt";
Now, we must ask our selves how does the jvm reuse the immutable "test.txt". In this case both s1 & s2 String objects point to the same "test.txt".
Understanding this concept will also do the trick for us:
public class Client {
public static void main( String args [] ) {
String filename = "test.txt";
String filename2 = "test.txt";
Reader reader = new Reader( filename ) ;
Writer writer = new Writer( filename2 ) ;
while(true) {
reader.run();
writer.run();
}
}
}
public class Writer implements Runnable {
public String filename;
public Writer( String filename ) {
this.filename = filename;
}
#Override
public void run() {
synchronized( this.filename ) {
System.out.println( "writing to a file:" + this.filename );
}
}
}
public class Reader implements Runnable {
public String filename;
public Reader( String filename ) {
this.filename = filename;
}
#Override
public void run() {
synchronized( this.filename ) {
System.out.println( "reading a file:" + this.filename );
}
}
}

Program pauses on initializing object input stream in Java

While running debugger, the program pauses on initializing object streams from server main input output streams. Following is the code :
public TFileReader(Client cli)throws Exception{
this.cli = cli;
fileSock = new Socket(cli.ServerIp(), cli.FilePort());
fobjIn = new ObjectInputStream(fileSock.getInputStream());
fobjOut = new ObjectOutputStream(fileSock.getOutputStream());
fobjOut.flush();
}
#Override
public void run(){
try{
System.out.println("file reader thread online");
fobjOut.writeObject(cli.Name());
fobjOut.flush();
String per = (String) fobjIn.readObject();
System.out.println(per+"video filing...");
if(!per.equals("OKF"))
{
throw new Exception("Error In retriving video.");
}
It pauses on fobjIn and do not go to execute fobjOut although fobjIn it passes from fobjIn breakpoint but do not hit out breakpoint.

I would keep it simple like this
public TFileReader(Client cli) throws IOException {
this.cli = cli;
socket = new Socket(cli.ServerIp(), cli.FilePort());
out = new ObjectOutputStream(socket.getOutputStream());
out.flush();
in = new ObjectInputStream(socket.getInputStream());
}
public void writeObject(Object o) throw IOException {
out.writeObject(o);
out.reset();
out.flush();
}
public <T> T readObject() throw IOException {
return (T) in.readObject();
}
public void close() throws IOException {
in.close();
out.close();
socket.close();
}

The problem is that ObjectInputStream pre-reads data on initialization.
The preferred solution is at Java Creating a new ObjectInputStream Blocks: always initialize your ObjectOutputStream before initializing your ObjectInputStream, so that the "handshake" that the two use internally can be initiated.
When you don't control all the code and cannot change the order, consider delaying the OIS initialization until data is available (InputStream.available or mark/read/reset on a buffered stream wrapping it, etc).

synchronizing file output in a static class

I have several threads which need to write to two different text files. So far I have this code:
public class Logger {
public static void printToGameLog(String value){
Writer writer = null;
try {
writer = new BufferedWriter(new OutputStreamWriter(
new FileOutputStream("GameLog.txt", true), "utf-8"));
synchronized(writer){
writer.write(outputString + "\r\n");
}
} catch (IOException ex){
System.out.println("cannot create log file");
}
}
public static void printToServerLog(String value){
Writer writer = null;
try {
writer = new BufferedWriter(new OutputStreamWriter(
new FileOutputStream("serverLog.txt", true), "utf-8"));
synchronized(writer){
writer.write(outputString + "\r\n");
}
} catch (IOException ex){
System.out.println("cannot create log file");
}
}
}
Is this an acceptable way of ensuring no more than one thread is writing to the same file at the same time?
If a thread calls one of these methods and enters the sync block, then what happens if another thread comes along and tries to execute the same method. When it tries to use the local variable writer, will it try and obtain the same object that has been locked by the other thread and therefore block? I would have thought that it would simply create its own separate variable, which would mean I should make writer a static class variable instead?

Since there are separate log files, I don't see why you need to have class-level synchronization. Seems like a needless bottleneck. I'd provide sync for each method separately (since it's fine for them to hit separate files simultaneously):
public class Logger
{
private static final Object GAME_LOG_LOCK = new Object();
private static final Object SERVER_LOG_LOCK = new Object();
public static void printToGameLog(String value){
synchronized (GAME_LOG_LOCK) {
Writer writer = null;
try {
writer = new BufferedWriter(new OutputStreamWriter(
new FileOutputStream("GameLog.txt", true), "utf-8"));
writer.write(outputString + "\r\n");
} catch (IOException ex){
System.out.println("cannot create log file");
}
}
}
public static void printToServerLog(String value){
synchronized (SERVER_LOG_LOCK) {
Writer writer = null;
try {
writer = new BufferedWriter(new OutputStreamWriter(
new FileOutputStream("serverLog.txt", true), "utf-8"));
writer.write(outputString + "\r\n");
} catch (IOException ex){
System.out.println("cannot create log file");
}
}
}
}

That's a null pointer exception in your code, try this way of using the synchronized block on a static method
synchronized(Logger.class){
or another alternative is to set the whole methods synchronized, like this
public static synchronized void printToGameLog(String value){
and
public static synchronized void printToServerLog(String value){
I'm not convinced that you need synchronization in here, you only need synchronization if you have a state that is being read/written from multiple threads.

Here is another take on your problem. It uses a single thread to write the log file and only this thread has access to the files. The threads that have to log something write against a BlockingQueue:
public class ThreadedLog {
//This is some code to test the logger
public static void main(String[] args) throws UnsupportedEncodingException, FileNotFoundException {
ThreadedLog log = new ThreadedLog("/tmp/test.txt");
// Start 100 thread that write against the log
for (int i = 0; i < 100; i++) {
new Thread(new TestLogger(log)).start();
}
}
private static class TestLogger implements Runnable {
private ThreadedLog log;
public TestLogger(ThreadedLog log) {
this.log = log;
}
#Override
public void run() {
for (int i = 0; i < 5000; i++) {
try {
log.log("This is entry " + i + " from thread " + Thread.currentThread().getId());
} catch (InterruptedException ex) {
}
}
System.out.println(Thread.currentThread().getId() + " is done");
}
}
//________________________________________________________________________________________________
/*
* This is the code for the actual logger
*
*/
private final BlockingQueue<String> queue = new ArrayBlockingQueue<>(10000);
private String fileName;
private Thread thread;
private Writer writer;
public ThreadedLog(String fileName) throws UnsupportedEncodingException, FileNotFoundException {
this.fileName = fileName;
thread = new Thread(new LoggingThread());
writer = new BufferedWriter(new OutputStreamWriter(
new FileOutputStream(fileName, true), "utf-8"));
thread.start();
}
private class LoggingThread implements Runnable {
#Override
public void run() {
try {
for (;;) {
ThreadedLog.this.writer.write(queue.take() + "\r\n");
ThreadedLog.this.writer.flush();
}
} catch (InterruptedException | IOException e) {
e.printStackTrace();
try {
ThreadedLog.this.writer.close();
} catch (Exception ex) {
}
}
}
}
public void log(String string) throws InterruptedException {
queue.put(string);
}
}