I have several threads which need to write to two different text files. So far I have this code:
public class Logger {
public static void printToGameLog(String value){
Writer writer = null;
try {
writer = new BufferedWriter(new OutputStreamWriter(
new FileOutputStream("GameLog.txt", true), "utf-8"));
synchronized(writer){
writer.write(outputString + "\r\n");
}
} catch (IOException ex){
System.out.println("cannot create log file");
}
}
public static void printToServerLog(String value){
Writer writer = null;
try {
writer = new BufferedWriter(new OutputStreamWriter(
new FileOutputStream("serverLog.txt", true), "utf-8"));
synchronized(writer){
writer.write(outputString + "\r\n");
}
} catch (IOException ex){
System.out.println("cannot create log file");
}
}
}
Is this an acceptable way of ensuring no more than one thread is writing to the same file at the same time?
If a thread calls one of these methods and enters the sync block, then what happens if another thread comes along and tries to execute the same method. When it tries to use the local variable writer, will it try and obtain the same object that has been locked by the other thread and therefore block? I would have thought that it would simply create its own separate variable, which would mean I should make writer a static class variable instead?
Since there are separate log files, I don't see why you need to have class-level synchronization. Seems like a needless bottleneck. I'd provide sync for each method separately (since it's fine for them to hit separate files simultaneously):
public class Logger
{
private static final Object GAME_LOG_LOCK = new Object();
private static final Object SERVER_LOG_LOCK = new Object();
public static void printToGameLog(String value){
synchronized (GAME_LOG_LOCK) {
Writer writer = null;
try {
writer = new BufferedWriter(new OutputStreamWriter(
new FileOutputStream("GameLog.txt", true), "utf-8"));
writer.write(outputString + "\r\n");
} catch (IOException ex){
System.out.println("cannot create log file");
}
}
}
public static void printToServerLog(String value){
synchronized (SERVER_LOG_LOCK) {
Writer writer = null;
try {
writer = new BufferedWriter(new OutputStreamWriter(
new FileOutputStream("serverLog.txt", true), "utf-8"));
writer.write(outputString + "\r\n");
} catch (IOException ex){
System.out.println("cannot create log file");
}
}
}
}
That's a null pointer exception in your code, try this way of using the synchronized block on a static method
synchronized(Logger.class){
or another alternative is to set the whole methods synchronized, like this
public static synchronized void printToGameLog(String value){
and
public static synchronized void printToServerLog(String value){
I'm not convinced that you need synchronization in here, you only need synchronization if you have a state that is being read/written from multiple threads.
Here is another take on your problem. It uses a single thread to write the log file and only this thread has access to the files. The threads that have to log something write against a BlockingQueue:
public class ThreadedLog {
//This is some code to test the logger
public static void main(String[] args) throws UnsupportedEncodingException, FileNotFoundException {
ThreadedLog log = new ThreadedLog("/tmp/test.txt");
// Start 100 thread that write against the log
for (int i = 0; i < 100; i++) {
new Thread(new TestLogger(log)).start();
}
}
private static class TestLogger implements Runnable {
private ThreadedLog log;
public TestLogger(ThreadedLog log) {
this.log = log;
}
#Override
public void run() {
for (int i = 0; i < 5000; i++) {
try {
log.log("This is entry " + i + " from thread " + Thread.currentThread().getId());
} catch (InterruptedException ex) {
}
}
System.out.println(Thread.currentThread().getId() + " is done");
}
}
//________________________________________________________________________________________________
/*
* This is the code for the actual logger
*
*/
private final BlockingQueue<String> queue = new ArrayBlockingQueue<>(10000);
private String fileName;
private Thread thread;
private Writer writer;
public ThreadedLog(String fileName) throws UnsupportedEncodingException, FileNotFoundException {
this.fileName = fileName;
thread = new Thread(new LoggingThread());
writer = new BufferedWriter(new OutputStreamWriter(
new FileOutputStream(fileName, true), "utf-8"));
thread.start();
}
private class LoggingThread implements Runnable {
#Override
public void run() {
try {
for (;;) {
ThreadedLog.this.writer.write(queue.take() + "\r\n");
ThreadedLog.this.writer.flush();
}
} catch (InterruptedException | IOException e) {
e.printStackTrace();
try {
ThreadedLog.this.writer.close();
} catch (Exception ex) {
}
}
}
}
public void log(String string) throws InterruptedException {
queue.put(string);
}
}
Related
I am using a ExecutorService to have multiple threads writing text into a file, but i cannot manage to synchronize the run() method and instead of having the proper line by line String i ask, i have a mixup of all the characters of the Strings because they write it at the same time.
import java.io.BufferedReader
...
class WriteDns implements Runnable {
File file;
String text;
WriteDns(File file, String text) {
this.file = file;
this.text = text;
}
public void run() {
synchronized (this) {
try (BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(
new FileOutputStream(file)))) {
bw.write(turnDns() + "\n");
} catch (IOException e) {
System.out.println("Error");
}
}
}
public String turnDns() {
int space = text.indexOf(' ');
String ip = text.substring(0, space);
String theRest = text.substring(space);
String temp = ip;
try {
ip = InetAddress.getByName(ip).getHostName();
if (ip == temp)
return "NotFound " + theRest;
return ip + " " + theRest;
} catch (UnknownHostException e) {
System.out.println("Error in change");
return "-changeErr " + theRest;
}
}
}
public class Main00 {
static File oldFile = new File("oldfile.txt");
public static void main(String[] args) {
readLines();
}
public static void readLines() {
try (BufferedReader br = new BufferedReader(new FileReader(oldFile))) {
File f = new File("file.txt");
ExecutorService service = Executors.newFixedThreadPool(10);
for (String t = br.readLine(); t != null; t = br.readLine()) {
service.execute(new WriteDns(f, t));
}
service.shutdown();
} catch (IOException e) {
e.printStackTrace();
}
}
}
You're synchronising on this but you're making a new instance of your thread worker for every thread, so each thread is locking on itself and never waiting for any other threads. You need to lock on an object that is visible to all threads, perhaps a static object or pass in a lock object when you instantiate your WriteDns.
With that said, having multiple threads open on one file is inherently prone to problems like you're experiencing, and you gain nothing from multiple threads writing since your bottleneck is your storage medium and not your processor. You should rather have multiple threads providing information/data to one dedicated writer thread that has exclusive access to the file you want to write to, as #FlorianSchaetz suggested.
I'm working on compiling a bunch of tweets for an information retrieval class. I'm trying this using both the REST API and the Streaming API through twitter4j. When using the Streaming API, I use the following modifications to this example:
final LimitedFileWriter output = new LimitedFileWriter("Tweets","tweets");
TwitterStream twitterStream = new TwitterStreamFactory(cb.build()).getInstance();
StatusListener listener = new StatusListener() {
#Override
public void onStatus(Status status) {
try{
output.write("#" + status.getUser().getScreenName() + " -- " + status.getText()+"\n");
}
catch(IOException e){
e.printStackTrace();
}
}
}
twitterStream.addListener(listener);
twitterStream.sample("en");
//output.close();
It seems I can't ever close my writer. The writer I am using simply wraps BufferedWriter, while keeping track of file size. If the file exceeds a certain size (128MB), the writer will close the current file and create a new file. Here are the relevant class functions:
public void write(String s) throws IOException
{
if(bytesWritten + s.getBytes(charset).length >= MAXSIZE){
output.close();
bytesWritten = 0;
fileNum++;
String fileName = directory + "/" + baseName+fmt.format(fileNum);
currentFile = new File(fileName);
output = new BufferedWriter
(new OutputStreamWriter(new FileOutputStream(fileName),charset));
}
output.write(s);
bytesWritten += s.getBytes(charset).length;
}
public void close() throws IOException{
output.close();
}
If I try to close the writer after twitterStream.sample() (commented out), the program crashes because I am trying to write to a closed file. If my understanding is correct, this is because the TwitterStream class creates a new thread which runs concurrently with the main thread. Then, the main thread closes the stream and the twitterStream can no longer write to it.
If that's the case, where should I close my writer?
If I have understood your question correctly, you want to be able to turn of the tweets collection at some point, close your open file writers and have a clean exit. To achieve it you can use a synchronized block.
final Object lock = new Object();
final LimitedFileWriter output = new LimitedFileWriter("Tweets","tweets");
TwitterStream twitterStream = new TwitterStreamFactory(cb.build()).getInstance();
StatusListener listener = new StatusListener() {
#Override
public void onStatus(Status status) {
try{
output.write("#" + status.getUser().getScreenName() + " -- " + status.getText()+"\n");
// free the lock
if (some_condition_like_I_have_enough_files) {
synchronized (lock) {
lock.notify();
}
}
catch(IOException e){
e.printStackTrace();
}
}
}
twitterStream.addListener(listener);
twitterStream.sample("en");
try {
synchronized (lock) {
lock.wait();
}
} catch (InterruptedException e) {
e.printStackTrace();
}
// close the twitterstream
// close the writer
I have created two threads and modified the run function so that one thread reads one line and the other writes the same line to the new file. This happens till the whole file is copied. The problem i am getting is that even though i have used variables to control that the threads execute one by one but still the threads are executing unevenly i.e one thread executes multiple times and then the control transfers. Any solutions i have attached the code. I am new to java as it is only for class assignment so the code might not be the most optimized.
public class thread1 extends Thread {
//To create producer and consumer as threads
//Shared variable
public static int x = 0;//checks if all lines are read
public static String line; /holds lines from file
public static int j = 0;//variable to switch between threads based upon its value
public thread1(String threadName) { //Constuctor
super(threadName); //Call to constructor of Thread class
}
public void run() {
while (x != -1)
{
if (Thread.currentThread().getName().contains("Reader")) {
if (x != -1&&j==0)
{
j=1;
String fileName = "d:/salfar.txt";
try {
// FileReader reads text files in the default encoding.
FileReader fileReader =
new FileReader(fileName);
// Always wrap FileReader in BufferedReader.
BufferedReader bufferedReader =
new BufferedReader(fileReader);
for (int check = 0; check <= x; check++) {
line = bufferedReader.readLine();
}
if (line == null) {
x = -1;
} else {
System.out.println(line);
x++;
}
// Always close files.
bufferedReader.close();
} catch (FileNotFoundException ex) {
System.out.println(
"Unable to open file '"
+ fileName + "'");
} catch (IOException ex) {
System.out.println(
"Error reading file '"
+ fileName + "'");
// Or we could just do this:
// ex.printStackTrace();
}
}
yield();
}
else if (Thread.currentThread().getName().contains("writer")) {
if (x != -1 && line != null&&j==1)
{
j=0;
String fileName = "d:/salfar1.txt";
try {
// Assume default encoding.
FileWriter fileWriter =
new FileWriter(fileName, true);
// Always wrap FileWriter in BufferedWriter.
BufferedWriter bufferedWriter =
new BufferedWriter(fileWriter);
// Note that write() does not automatically
// append a newline character.
bufferedWriter.write(line);
bufferedWriter.newLine();
System.out.println("y");
// Always close files.
bufferedWriter.close();
} catch (IOException ex) {
System.out.println(
"Error writing to file '"
+ fileName + "'");
// Or we could just do this:
// ex.printStackTrace();
}
}
Thread.yield();
}
else{}
}
}
public static void main(String[] args) {
thread1 p = new thread1("Reader");
thread1 c = new thread1("writer");
p.start();
c.start();
}
}
Thanks
You cannot control the order of thread execution. However, to perform read and write operation via separate threads, you should use BlockingQueue which has the following properties:
A Queue that additionally supports operations that wait for the queue to become non-empty when retrieving an element, and wait for space to become available in the queue when storing an element.
ReaderThread will read from the input file.
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.util.concurrent.BlockingQueue;
public class ReaderThread implements Runnable{
protected BlockingQueue<String> blockingQueue = null;
public ReaderThread(BlockingQueue<String> blockingQueue){
this.blockingQueue = blockingQueue;
}
#Override
public void run() {
BufferedReader br = null;
try {
br = new BufferedReader(new FileReader(new File("./inputFile.txt")));
String buffer =null;
while((buffer=br.readLine())!=null){
blockingQueue.put(buffer);
}
blockingQueue.put("EOF"); //When end of file has been reached
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} catch(InterruptedException e){
}finally{
try {
br.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
WriterThread will write to output file.
import java.io.File;
import java.io.FileNotFoundException;
import java.io.PrintWriter;
import java.util.concurrent.BlockingQueue;
public class WriterThread implements Runnable{
protected BlockingQueue<String> blockingQueue = null;
public WriterThread(BlockingQueue<String> blockingQueue){
this.blockingQueue = blockingQueue;
}
#Override
public void run() {
PrintWriter writer = null;
try {
writer = new PrintWriter(new File("outputFile.txt"));
while(true){
String buffer = blockingQueue.take();
//Check whether end of file has been reached
if(buffer.equals("EOF")){
break;
}
writer.println(buffer);
}
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch(InterruptedException e){
}finally{
writer.close();
}
}
}
From Launcher class start your multithreaded read and write.
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.BlockingQueue;
public class Launcher {
public static void main(String[] args) {
BlockingQueue<String> queue = new ArrayBlockingQueue<String>(1024);
ReaderThread reader = new ReaderThread(queue);
WriterThread writer = new WriterThread(queue);
new Thread(reader).start();
new Thread(writer).start();
}
}
Here is my solutions. My idea is to use the actually file name that our threads will be reading and writing to. There is only one issue that we need to make sure of, that no two threads are trying to operate on the same file. The solution to this is to simply have a synchronized code block in your run method.
We must recall that Strings are not mutable in Java. Consider the following:
String s1 = "test.txt";
String s2 = "test.txt";
Now, we must ask our selves how does the jvm reuse the immutable "test.txt". In this case both s1 & s2 String objects point to the same "test.txt".
Understanding this concept will also do the trick for us:
public class Client {
public static void main( String args [] ) {
String filename = "test.txt";
String filename2 = "test.txt";
Reader reader = new Reader( filename ) ;
Writer writer = new Writer( filename2 ) ;
while(true) {
reader.run();
writer.run();
}
}
}
public class Writer implements Runnable {
public String filename;
public Writer( String filename ) {
this.filename = filename;
}
#Override
public void run() {
synchronized( this.filename ) {
System.out.println( "writing to a file:" + this.filename );
}
}
}
public class Reader implements Runnable {
public String filename;
public Reader( String filename ) {
this.filename = filename;
}
#Override
public void run() {
synchronized( this.filename ) {
System.out.println( "reading a file:" + this.filename );
}
}
}
I'm writing a program that writes to a single file from both different threads on the same JVM and from different JVM's/processes. Is there a way to lock a file for both threads and processes, so that no matter how many threads/processes are trying to write at the same time, only 1 can write at a time?
Currently I have something similar to the following which works for locking threads, but not for blocking processes. If I try using FileLock on top of the implementation below it appears the synchronized stops working.
import org.apache.commons.io.FileUtils;
import org.apache.commons.io.IOUtils;
import scripts.com.adz.commons.utils.FileUtilities;
import java.io.*;
public class Foo {
public static void main(String[] args) throws IOException {
Bar bar = new Bar();
bar.start();
while (true) {
FileUtilities.writeObjectToFile("C:\\test.html", "foo");
}
}
}
class Bar extends Thread {
#Override
public void run() {
while (true) {
try {
FileUtilities.writeObjectToFile("C:\\test.html", "bar");
} catch (IOException ignored) {}
}
}
}
class FileUtilitiess {
private static final Object _lock = new Object();
public static <T> T readObjectFromFile(File file) throws IOException, ClassNotFoundException {
synchronized (_lock) {
final byte[] bytes = FileUtils.readFileToByteArray(file);
ByteArrayInputStream bis = null;
ObjectInputStream ois = null;
try {
ois = new ObjectInputStream(bis = new ByteArrayInputStream(bytes));
return (T) ois.readObject();
} finally {
IOUtils.closeQuietly(ois);
IOUtils.closeQuietly(bis);
}
}
}
public static void writeObjectToFile(File file, Object object) throws IOException {
System.out.println("Sent object: " + object.toString());
synchronized (_lock) {
System.out.println("Writing object: " + object.toString());
ByteArrayOutputStream bos = null;
ObjectOutputStream oos = null;
try {
oos = new ObjectOutputStream(bos = new ByteArrayOutputStream());
oos.writeObject(object);
FileUtils.writeByteArrayToFile(file, bos.toByteArray());
// - Start: For testing lock.
try {
Thread.sleep(10000);
} catch (InterruptedException ignored) {}
// - End: For testing lock.
} finally {
IOUtils.closeQuietly(oos);
IOUtils.closeQuietly(bos);
}
}
}
}
See FileLock javadoc:
File locks are held on behalf of the entire Java virtual machine.
That means that on the OS level different threads of your application will have the same right to access the locked region.
To lock the file access from different threads you have to encapsulate your file IO code and to enforce synchronized execution.
I have two threads running, userInputThread waits for user input from the command line and interrupterThread tries to interrupt userInputThread 1 sec after starting. Obviously you cannot interrupt a thread that is blocked by the System.in. Another answer suggests to close System.in with System.in.close() before interrupting a thread. But when I run the following code, the userInputThread never gets interrupted and the app just hangs without closing.
class InputInterruptionExample {
private Thread userInputThread;
private Thread interrupterThread;
InputInterruptionExample() {
this.userInputThread = new Thread(new UserInputThread());
this.interrupterThread = new Thread(new InterrupterThread());
}
void startThreads() {
this.userInputThread.start();
this.interrupterThread.start();
}
private class UserInputThread implements Runnable {
public void run() {
try {
System.out.println("enter your name: ");
String userInput = (new BufferedReader(new InputStreamReader(System.in))).readLine();
} catch (IOException e) {
System.out.println("Oops..somethign went wrong.");
System.exit(1);
}
}
}
private class InterrupterThread implements Runnable {
public void run() {
try {
sleep(1000);
System.out.println("about to interrupt UserInputThread");
System.in.close();
userInputThread.interrupt();
userInputThread.join();
System.out.println("Successfully interrupted");
} catch (InterruptedException e) {
} catch (IOException ex) {
System.out.println("Oops..somethign went wrong.");
System.exit(1);
}
}
}
public static void main(String[] args) {
InputInterruptionExample exampleApp = new InputInterruptionExample();
exampleApp.startThreads();
}
}
There's already a similar question, but there aren't any definite answers.
This has solved the problem:
class InputInterruptionExample {
private UserInputThread userInputRunnable;
private Thread userInputThread;
private Thread interrupterThread;
InputInterruptionExample() {
this.userInputRunnable = new UserInputThread();
this.userInputThread = new Thread(userInputRunnable);
this.interrupterThread = new Thread(new InterrupterThread());
}
void startThreads() {
this.userInputThread.start();
this.interrupterThread.start();
}
private class UserInputThread implements Runnable {
private InputStreamReader isr;
private BufferedReader br;
UserInputThread() {
InputStreamReader isr = new InputStreamReader(System.in);
BufferedReader br = new BufferedReader(isr);
}
public void run() {
try {
System.out.println("enter your name: ");
try{
String userInput = br.readLine();
} catch(NullPointerException e) {}
} catch (IOException e) {
System.out.println("Oops..somethign went wrong.");
System.exit(1);
}
}
public void closeBufferdReader() {
try {
System.in.close();
} catch (IOException e) {
System.out.println("Oops..somethign went wrong in closeBufferdReader() method");
System.exit(1);
}
}
}
private class InterrupterThread implements Runnable {
public void run() {
try {
sleep(1000);
userInputRunnable.closeBufferdReader();
userInputThread.interrupt();
userInputThread.join();
System.out.println("Successfully interrupted");
} catch (InterruptedException e) {}
}
}
public static void main(String[] args) {
InputInterruptionExample exampleApp = new InputInterruptionExample();
exampleApp.startThreads();
}
}
Update: This only works when BufferedReader is split up this way:
InputStreamReader isr = new InputStreamReader(System.in);
BufferedReader br = new BufferedReader(isr);
String userInput = br.readLine();
For some reason the interruption does not seem to work when the readLine() structure is written as a oneliner:
this.userInput = (new BufferedReader(new InputStreamReader(System.in))).readLine();
So while it is possible to interrupt the thread in the split-up BufferedReader structure, it is now impossible to read user's input.
If someone could show a way to be able to get user input as well as interrupt the UserInputThread when the user doesn't provide any input in time (while interrupter is sleeping), please do.
All of my research leads me to believe that the underlying .read() in the .readLine() call cannot be interrupted (Without destroying the Process that System.in is attached to, at least). The only other choices at that point is to use a polling IO scheme or switch to NIO.
Here's a quick (and very dirty/ugly) adaptation of your code into a polling IO scheme. It's not an interupt solution so it's not directly answering your question, but rather hopefully getting you the behavior you desire.
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.concurrent.atomic.AtomicBoolean;
public class InputInterruptionExample {
private UserInputThread uiThreadObj = null;
private Thread inputThread = null;
private Thread interrupThread = null;
public InputInterruptionExample() {
this.uiThreadObj = new UserInputThread();
this.inputThread = new Thread(this.uiThreadObj);
this.interrupThread = new Thread(new InterrupterThread());
}
void startThreads() {
this.inputThread.start();
this.interrupThread.start();
}
private class UserInputThread implements Runnable {
private final AtomicBoolean runCmd = new AtomicBoolean(true);
public void run() {
try {
final BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
System.out.println("enter your name: ");
while (this.runCmd.get()) {
if (System.in.available() > 0) {
String userInput = br.readLine();
System.out.println("You typed: " + userInput);
System.out.println("enter your name: ");
} else {
Thread.sleep(5); //minimal sleep to prevent CPU peg
}
}
System.out.println("Finishing normally.");
} catch (IOException e) {
System.out.println("Oops..somethign went wrong.");
System.exit(1);
} catch (Exception e) {
System.out.println("What'd you do?!");
e.printStackTrace();
}
}
public final void requestStop() {
this.runCmd.set(false);
}
}
private class InterrupterThread implements Runnable {
public void run() {
try {
Thread.sleep(1000 * 10);
System.out.println("Requesting that UserInputThread stop.");
uiThreadObj.requestStop();
System.out.println("Request made.");
} catch (InterruptedException e) {
System.out.println("Oops..somethign went wrong.");
System.exit(1);
}
}
}
public static void main(String[] args) {
InputInterruptionExample exampleApp = new InputInterruptionExample();
exampleApp.startThreads();
}
}