How can I synchornize two Java processes running on Windows ?
I am looking for something like the Win32 Named Mutex object which allows two processes to use the same locking object.
Thanks
Java cross process lock:
// Tester
try {
if (crossProcessLockAcquire(SomeClassInYourApp.class, 3000)) {
// Success - This process now has the lock. (Don't keep it too long.)
}
else {
// Fail (Timeout) - Another process still had the lock after 3 seconds.
}
} finally {
crossProcessLockRelease(); // try/finally is very important.
}
// Acquire - Returns success ( true/false )
private static boolean crossProcessLockAcquire(final Class<?> c, final long waitMS) {
if (fileLock == null && c != null && waitMS > 0) {
try {
long dropDeadTime = System.currentTimeMillis() + waitMS;
File file = new File(lockTempDir, c.getName() + ".lock");
RandomAccessFile randomAccessFile = new RandomAccessFile(file, "rw");
FileChannel fileChannel = randomAccessFile.getChannel();
while (System.currentTimeMillis() < dropDeadTime) {
fileLock = fileChannel.tryLock();
if (fileLock != null) {
break;
}
Thread.sleep(250); // 4 attempts/sec
}
} catch (Exception e) {
e.printStackTrace();
}
}
return fileLock == null ? false : true;
}
// Release
private static void crossProcessLockRelease() {
if (fileLock != null) {
try {
fileLock.release();
fileLock = null;
} catch (IOException e) {
e.printStackTrace();
}
}
}
// Some class vars and a failsafe lock release.
private static File lockTempDir = new File(System.getProperty("java.io.tmpdir") + File.separator + "locks");
private static FileLock fileLock = null;
static {
Runtime.getRuntime().addShutdownHook(new Thread() {
public void run(){
crossProcessLockRelease();
}
});
}
I simplified Java42 answer
Usage
ProcessLock lock = new ProcessLock("lockKey");
lock.run(successLockRunnable, timeOutLockRunnable);
The code in successLockRunnable will lock any other process on the same machine using this implementation.
Source
/**
* Created by Ilya Gazman on 13/06/2016.
* Based on https://stackoverflow.com/a/9577667/1129332
*/
public class ProcessLock {
// Some class vars and a fail safe lock release.
private File lockTempDir = new File(System.getProperty("java.io.tmpdir") + File.separator + "locks");
private FileLock fileLock = null;
private String key;
public ProcessLock() {
this("lock");
}
public ProcessLock(String key) {
this.key = key;
Runtime.getRuntime().addShutdownHook(new Thread() {
public void run() {
crossProcessLockRelease();
}
});
}
public void run(Runnable successCallback) {
run(successCallback, null);
}
public void run(Runnable successCallback, Runnable timeOutCallback) {
try {
if (crossProcessLockAcquire(3000)) {
successCallback.run();
} else if (timeOutCallback != null) {
timeOutCallback.run();
}
} finally {
crossProcessLockRelease(); // try/finally is very important.
}
}
// Acquire - Returns success ( true/false )
private boolean crossProcessLockAcquire(final long waitMS) {
if (fileLock == null && waitMS > 0) {
try {
long dropDeadTime = System.currentTimeMillis() + waitMS;
File file = new File(lockTempDir, "_" + key + ".lock");
RandomAccessFile randomAccessFile = new RandomAccessFile(file, "rw");
FileChannel fileChannel = randomAccessFile.getChannel();
while (System.currentTimeMillis() < dropDeadTime) {
fileLock = fileChannel.tryLock();
if (fileLock != null) {
break;
}
Thread.sleep(250); // 4 attempts/sec
}
} catch (Exception e) {
e.printStackTrace();
}
}
return fileLock != null;
}
// Release
private void crossProcessLockRelease() {
if (fileLock != null) {
try {
fileLock.release();
fileLock = null;
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
It is not possible to do something like you want in Java. Different Java applications will use different JVM's fully separating themselves into different 'blackbox'es. However, you have 2 options:
Use sockets (or channels). Basically one application will open the listening socket and start waiting until it receives some signal. The other application will connect there, and send signals when it had completed something. I'd say this is a preferred way used in 99.9% of applications.
You can call winapi from Java. I do not remember the specifics, but you can find a plethora of example if you google "java winapi".
We use these kinds of statements to make sure only one process can do a block of code keyed by "myLockKey":
new LocalFileLock("myLockKey").doWithLock(() -> {
doSomething();
return null; // must return something
});
Here, we make use of this class:
import java.io.File;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.channels.FileChannel;
import java.nio.channels.FileLock;
import java.util.function.Supplier;
import com.headlandstech.utils.FileUtils;
import com.headlandstech.utils.Log;
public class LocalFileLock {
private final File lockFile;
public LocalFileLock(String name) {
this.lockFile = new File(FileUtils.TEMP_DIR, name + ".lock");
if (!lockFile.isFile()) {
FileUtils.writeStringToFile("", lockFile);
}
}
public <T> T doWithLock(Supplier<T> f) {
Log.log.info("Waiting on lock " + lockFile);
try (FileChannel channel = new RandomAccessFile(lockFile, "rw").getChannel()) {
final FileLock fileLock = channel.lock();
Log.log.info("Lock " + lockFile + " obtained");
T result = f.get();
fileLock.release();
Log.log.info("Lock " + lockFile + " released");
return result;
} catch (IOException e) {
throw new RuntimeException(e);
}
}
}
I don't think there are native methods in the java platform for this. However, there are several ways to go about obtaining the same type of effect depending on what synchronization you are trying to accomplish. In addition to having the processes communicate over network connections (direct sockets, multicast with an election, etc.) or dropping to platform specific calls, you can also explore obtaining a file lock to a shared file (see activemq passive stand-by with a shared file system for an example) or using a database either with something like a select for update or an optimistic update of a table row.
Not sure what you are trying to do, I'd possibly do this by exposing something via JMX and having the separate processes set a status flag which then programmatically revives your thread from a wait state. Instead of JMX you could of course use a socket/RMI.
using sockets for cross processes synchronizations is common practice . not only for java applications because in most *nix environments we have not system-wide mutexes as we have in Windows.
Related
This question already has answers here:
Threads and file writing
(6 answers)
Closed 9 years ago.
i have 20 threads that write with the println() function on a file called results.txt. How can i synchronize them all?
I note every time my program run i have different number of lines of text in results.txt.
Thank you.
Access the file through a class that contains a synchronized method to write to the file. Only one thread at a time will be able to execute the method.
I think that Singleton pattern would fit for your problem:
package com.test.singleton;
public class Singleton {
private static final Singleton inst= new Singleton();
private Singleton() {
super();
}
public synchronized void writeToFile(String str) {
// Do whatever
}
public static Singleton getInstance() {
return inst;
}
}
Every time you need to write to your file, you only would have to call:
Singleton.getInstance().writeToFile("Hello!!");
Duplicate question ... duplicate answer. As I said here:
If you can hold your file as a FileOutputStream you can lock it like this:
FileOutputStream file = ...
....
// Thread safe version.
void write(byte[] bytes) {
try {
boolean written = false;
do {
try {
// Lock it!
FileLock lock = file.getChannel().lock();
try {
// Write the bytes.
file.write(bytes);
written = true;
} finally {
// Release the lock.
lock.release();
}
} catch ( OverlappingFileLockException ofle ) {
try {
// Wait a bit
Thread.sleep(0);
} catch (InterruptedException ex) {
throw new InterruptedIOException ("Interrupted waiting for a file lock.");
}
}
} while (!written);
} catch (IOException ex) {
log.warn("Failed to lock " + fileName, ex);
}
}
You are intend to write data into one file. So if you try to lock the whole file, it'd better to use a single thread to do this job. Although you spawn 20 threads, but there is only one of them is running every time you call the method, the others are just waiting for the lock.
I recommend you use RandomAccessFile to write data to your file. Then each thread can write some unique data into to the file without locking the whole file.
Some demo code as following
try {
final RandomAccessFile file = new RandomAccessFile("/path/to/your/result.txt", "rw");
final int numberOfThread = 20;
final int bufferSize = 512;
ExecutorService pool = Executors.newFixedThreadPool(numberOfThread);
final AtomicInteger byteCounter = new AtomicInteger(0);
final byte[] yourText = "Your data".getBytes();
for (int i = 0; i < yourText.length; i++) {
pool.submit(new Runnable() {
#Override
public void run() {
int start = byteCounter.getAndAdd(bufferSize);
int chunkSize = bufferSize;
if (start + bufferSize > yourText.length) {
chunkSize = yourText.length - start;
}
byte[] chunkData = new byte[chunkSize];
System.arraycopy(yourText, start, chunkData, 0, chunkSize);
try {
file.write(chunkData);
} catch (IOException e) {
//exception handle
}
}
});
}
file.close();
} catch (Exception e) {
//clean up
}
I'm new to Java concurrent API and I've searched but didn't find an answer to my question.
Well, I have a code that look for every file inside directories and their subdirectories and another that copy every found file that match a specified pattern.
I separate this codes in one Runnable implementation called DirSearch and one Callable implementation called FileSearch and submit them using an ExecutorService.
That's the code:
private boolean execute() {
ExecutorService executor = Executors.newFixedThreadPool(threadsNumber);
BlockingQueue<File> dirQueue = new LinkedBlockingQueue<>();
BlockingQueue<File> fileQueue = new LinkedBlockingQueue<>(10000);
boolean isFinished = false;
try {
for(int i = 0; i < dirThreads; i++) {
executor.submit(new DirSearch(dirQueue, fileQueue, count, dirThreads);
}
count.incrementAndGet();
dirQueue.add(baseDir);
Future<Boolean> future = executor.submit(new FileSearch(filequeue, outputDirectory, filename));
isFinished = future.get();
} catch(ExecutionException | InterruptedException | RuntimeException ex) {
ex.printStackTrace();
} finally {
executor.shutdownNow();
}
return isFinished;
}
...
private void copyFile(File in, File out) {
Path inPath = Paths.get(in.getAbsolutePath());
Path outPath = Paths.get(out.getAbsolutePath(), in.getName());
try {
main.updateCurrentLabel(outPath.toString());
switch(mode) {
case "1":
Files.copy(inPath, outPath, StandardCopyOption.REPLACE_EXISTING);
break;
case "2":
Files.move(inPath, outPath, StandardCopyOption.REPLACE_EXISTING);
break;
default:
break;
}
main.updateCopiedLabel(String.valueOf(countCpFiles.incrementAndGet()));
} catch(IOException ex) {
ex.printStackTrace();
}
}
...
private class DirSearch implements Runnable {
...
#Override
public void run() {
try {
File dir = dirQueue.take();
while(dir != new File("")) {
File[] elements = dir.listFiles();
if(elements != null) {
for(File element : elements) {
if(element.isDirectory()) {
count.incrementAndGet();
dirQueue.put(element);
} else {
fileQueue.put(element);
}
}
}
if(count.decrementAndGet() == 0) {
end();
}
dir = dirQueue.take();
}
} catch(InterruptedException ex) {
ex.printStackTrace();
}
}
...
}
...
private class FileSearch implements Callable<Boolean> {
...
#Override
public Boolean call() {
boolean isFinished = false;
try {
File file = fileQueue.take();
while(file != new File("")) {
incrementAnalyzed();
String foundFile = file.getName().toLowerCase();
if(foundFile.matches(filename.replace("?", ".?").replace("*", ".*?"))) {
copyFile(file, outputDirectory);
}
file = fileQueue.take();
}
isFinished = true;
} catch(InterruptedException ex) {
ex.printStackTrace();
}
return isFinished;
}
}
The problem is: when the FileSearch start to copy files, the other threads (DirSearch) stop and don't look for any new file until the copy is completed. Why this is happening? Am I doing anything wrong or this is not the correct approach?
Two possible answers which came to my mind and which i cant guarantee they apply to your specific situation:
1. Java VM gets only one core from your CPU which means it can only run one thread at a time.
2. your threads both use the same variable which means only one is allowed to really manipulate it at a time. For this specific problem look up java keyword "synchronized".
I guess the root of the problem tends to be #1
So I'm trying to create a client/server program. I want to know when my client disconnects of his own accord, so I've setup a heartbeat system. Every 6 seconds my client sends a ping to my server, if the client doesn't send a ping for a total of 30 seconds the client is considered disconnected and removed from the current connections list (for which I plan to implement a GUI). Or at least, that's the plan.
ConnectionManager.java
public class ConnectionManager implements Runnable{
static Socket connection;
private ArrayList<Thread> allConnections;
private ArrayList<Connection> allConnectionList;
private ServerSocket server;
private int id = 0;
public ConnectionManager() {
allConnections = new ArrayList<Thread>();
allConnectionList = new ArrayList<Connection>();
}
#Override
public void run() {
try {
server = new ServerSocket(5555);
System.out.println("Server is running!");
while(true) {
connection = server.accept();
Connection a = new Connection(connection, id);
Runnable runnable = a;
allConnectionList.add(a);
allConnections.add(new Thread(runnable));
allConnections.get(allConnections.size() - 1).start();
id++;
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public void removeConnection(int id) {
allConnections.remove(id);
allConnectionList.remove(id);
}
Connection.java
public class Connection implements Runnable {
private Socket a;
public boolean amIActive;
private int id;
public Connection(Socket a, int id) {
amIActive = true;
this.a = a;
this.id = id;
}
public void onConnect() {
try {
String TimeStamp = new java.util.Date().toString();
String formattedAddress = a.getInetAddress().toString().replace("/", "");
System.out.println("Received connection from: " + formattedAddress + " at " + TimeStamp);
Runnable runnable = new ConnectionListener(this);
Thread connectionThread = new Thread(runnable);
connectionThread.start();
String returnCode = "Server repsonded to " + a.getInetAddress().toString().replace("/", "") + " at "+ TimeStamp + (char) 13;
BufferedOutputStream os = new BufferedOutputStream(a.getOutputStream());
OutputStreamWriter osw = new OutputStreamWriter(os, "US-ASCII");
osw.write(returnCode);
osw.flush();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
#Override
public void run() {
onConnect();
System.out.println("We got this far!");
while(amIActive) {
whileTrue();
}
System.out.println("This code never gets run because we get stuck in the while loop above");
Main.b.removeConnection(id);
System.out.println("Connection was closed from " + a.getInetAddress());
}
public void setOffline(boolean state) {
this.amIActive = state;
}
public void whileTrue() {
}
public Socket getSocket() {
return a;
}
ConnectionListener.java
public class ConnectionListener implements Runnable{
public Connection myConnection;
public boolean receivedHeartbeat;
public int missedHeartbeats = 0;
public ConnectionListener(Connection a) {
this.myConnection = a;
}
#Override
public void run() {
Runnable runnable = new Heartbeat(this);
Thread thread = new Thread(runnable);
thread.start();
while(myConnection.amIActive) {
try {
BufferedInputStream is;
is = new BufferedInputStream(myConnection.getSocket().getInputStream());
InputStreamReader isr = new InputStreamReader(is);
StringBuffer process = new StringBuffer();
int character;
while((character = isr.read()) != 13) { //GETTING STUCK HERE BECAUSE STUPID.
if(character == -1) {
myConnection.setOffline(true);
} else {
process.append((char)character);
}
}
handleInput(process);
} catch (Exception e) {
e.printStackTrace();
}
}
}
public void handleInput(StringBuffer process) {
String messageSent = process.toString();
if(messageSent.equals("Ping!")) {
receivedHeartbeat = true;
}
}
Heartbeat.java
public class Heartbeat implements Runnable{
private ConnectionListener b;
public Heartbeat(ConnectionListener a) {
b = a;
}
#Override
public void run() {
while(true) {
try {
Thread.sleep(1000);
if(b.missedHeartbeats > 5) {
b.myConnection.amIActive = false;
System.out.println("Setting amIActiveToFalse!");
}
if(b.receivedHeartbeat) {
b.receivedHeartbeat = false;
} else {
b.missedHeartbeats++;
}
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
My console is spammed with System.out.println("Setting amIActiveToFalse!"); from Heartbeat.java. But the while loop in Connection.java keeps running. I believe this might be something to do with my threading, but I can't figure it out.
When you have a non-volatile variable, there is no guarentee of visability of a change in one thread to another. In particular, if the JVM detects that a thread doesn't alter a boolean it can inline it, meaning you will never see the value change.
The simple solution is to make the boolean volatile and it will not be inlined and one thread will see when another changes it.
For more details http://vanillajava.blogspot.com/2012/01/demonstrating-when-volatile-is-required.html
The trivial answer to this is: make the variable volatile.
Without this, it is allowed for the thread changing the value to basically keep its updates in cache, committing them to main memory some time later.
This allows threaded code to run much faster, since it can keep its variables in cache rather than having to fetch from main memory. However, the consequence of this is that other threads don't see the update.
Making the variable volatile prevents this from happening: a thread always reads the value from main memory, and writes are immediately committed.
I say that this is the trivial answer because it doesn't necessarily fix all of your problems. There may also be an atomicity issue: in between one thread reading the variable and writing it again, another thread might sneak in and change its value, which may or may not put the first thread into an undefined state from the perspective of its invariants.
Specifically:
if(b.receivedHeartbeat) { b.receivedHeartbeat = false;
It is possible that some other thread can change b.receivedHeartbeat to false after this thread evaluates it to true, so this iteration is erroneously counted as a "non-missed" heartbeat.
This can be fixed by making the variable a (non-volatile) AtomicBoolean, on which there is an atomic compare-and-set method, which avoids such race conditions.
Java Concurrency In Practice is a great reference on these issues, I wholeheartedly recommend it. Look for the topics "visibility" and "atomicity".
Also read the advanced chapter on the Java Memory Model. That made me doubt myself at first, but made me a much stronger programmer after I digested it.
There are a couple issues I saw while debugging the code you posted, but I was able to successfully get the heartbeat functionality working.
In the Connection Listener class I don't think the if statement with .equals("Ping!") will match, because of the newline character at the end of each line.
In the Connection Listener class I would probably put the socket's Input Stream at the top of the loop not inside the loop. (I don't think this will break it but it's probably nicer this way)
ConnectionListener Updates:
public void run() {
Runnable runnable = new Heartbeat(this);
Thread thread = new Thread(runnable);
thread.start();
BufferedReader br = null;
try {
//is = new BufferedInputStream(myConnection.getSocket().getInputStream());
br = new BufferedReader(new InputStreamReader(myConnection.getSocket().getInputStream()));
} catch (IOException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
while(myConnection.amIActive) {
try {
String processLine = br.readLine();
System.out.println("handleInput:" + processLine);
handleInput(processLine);
} catch (Exception e) {
System.out.println("Exception!");
e.printStackTrace();
}
}
}
public void handleInput(String messageSent) {
if(messageSent.startsWith("Ping!")) { //Need to use startsWith, or add newline character
receivedHeartbeat = true;
System.out.println("receivedHeartbeat!");
}
}
Also, in your Heartbeat class make sure you reset the missedHeartbeats counter to 0 on true:
if(b.receivedHeartbeat) {
b.receivedHeartbeat = false;
b.missedHeartbeats = 0;
} else {
b.missedHeartbeats++;
}
I am working on a Java process that contains 2 threads: one for reading a file's contents and adding them in one shared blocking queue; and one for retrieving the lines from the blocking queue and sending them through the network (under a specified send rate). The two classes I have are the following:
Updated Code below
Producer Thread:
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.util.concurrent.ArrayBlockingQueue;
public class SourceFileProducer implements Runnable {
private File file;
private BufferedReader reader;
private ArrayBlockingQueue<String> buffer;
private String fileName;
private String endMarker;
public SourceFileProducer(ArrayBlockingQueue<String> buffer,
String endMarker, String fileName) {
this.buffer = buffer;
this.endMarker = endMarker;
file = new File(fileName);
if(file.exists()) {
try {
reader = new BufferedReader(new FileReader(file));
} catch (FileNotFoundException e) {
e.printStackTrace();
}
}
this.fileName = fileName;
}
#Override
public void run() {
System.out.println("SourceFileProducer thread-" + Thread.currentThread().getId() + " initiating with source file: " + fileName);
String line = "";
try {
while((line = reader.readLine()) != null) {
try {
buffer.put(line);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
try {
buffer.put(endMarker);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("SourceFileProducer thread-" + Thread.currentThread().getId() + " scanned and buffered the whole file.");
} catch (IOException e) {
e.printStackTrace();
}
}
}
and the Consumer thread:
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.PrintWriter;
import java.net.ServerSocket;
import java.net.Socket;
import java.util.concurrent.ArrayBlockingQueue;
public class SourceFileConsumer implements Runnable {
private ArrayBlockingQueue<String> buffer;
private BufferedReader socketInput;
private PrintWriter socketOutput;
private Socket client;
private ServerSocket serverSocket;
private long checkpoint[] = null;
private int rate[] = null;
private String endMarker;
public SourceFileConsumer(ArrayBlockingQueue<String> buffer, String endMarker,
ServerSocket serverSocket, Socket client, long checkpoint[], int rate[]) {
this.buffer = buffer;
this.endMarker = endMarker;
this.client = client;
try {
socketOutput = new PrintWriter(client.getOutputStream(), true);
socketInput = new BufferedReader(new InputStreamReader(client.getInputStream()));
} catch (IOException e) {
e.printStackTrace();
}
this.checkpoint = new long[checkpoint.length];
this.rate = new int[rate.length];
for(int i = 0; i < checkpoint.length; i++) {
this.checkpoint[i] = checkpoint[i];
this.rate[i] = rate[i];
}
this.serverSocket = serverSocket;
}
#Override
public void run() {
String line = null;
long start = System.currentTimeMillis();
int index = 0;
boolean fileScanFlag = true;
while(fileScanFlag) {
long startTimestamp = System.currentTimeMillis();
long interval = (startTimestamp - start) / 1000L;
if(interval >= checkpoint[index]) {
if(index < checkpoint.length - 1) {
if(interval >= checkpoint[index + 1]) {
index += 1;
System.out.println("SourceFileConsumer thread-" + Thread.currentThread().getId() +
" progressed to checkpoint " + index + " with rate: " + rate[index]);
}
}
}
int counter = 0;
while(counter < rate[index]) {
try {
line = buffer.take();
} catch (InterruptedException e1) {
e1.printStackTrace();
}
if(line == endMarker) {
fileScanFlag = false;
break;
}
if(socketOutput != null && socketOutput.checkError()) {
System.out.println("SourceFileConsumer Thread-" + Thread.currentThread().getId() + " detected broken link...");
try {
client = serverSocket.accept();
socketOutput = new PrintWriter(client.getOutputStream(), true);
socketInput = new BufferedReader(new InputStreamReader(client.getInputStream()));
} catch(IOException e) {
e.printStackTrace();
}
System.out.println("SourceFileConsumer Thread-" + Thread.currentThread().getId() + " re-established connection...");
}
if(socketOutput != null)
socketOutput.println(line);
counter += 1;
}
long endTimestamp = System.currentTimeMillis();
if(endTimestamp - startTimestamp <= 1000) {
System.out.println("thread-" + Thread.currentThread().getId() + " input rate: " + counter +
", wait time: " + (1000 - (endTimestamp - startTimestamp)));
try {
Thread.sleep((1000 - (endTimestamp - startTimestamp)));
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
if(socketInput != null && socketOutput != null && client != null) {
try {
socketInput.close();
socketOutput.close();
client.close();
} catch(IOException e) {
e.printStackTrace();
}
}
System.out.println("SourceFileConsumer Thread-" + Thread.currentThread().getId() + " transfer complete.");
}
}
The problem is that, after a while, both threads hang and no tuples are sent. When I run a top command in my Linux machine, I see that the Java process, in which the two threads are running in, uses a really small amount of CPU time. Why is this happening? Is this a problem with starvation? I think that starvation can be avoided by using the LinkedBlockingQueue.
Any hints?
Thanks,
Nick
That’s quite a lot of code, especially within your consumer. So it’s not possible to preclude that there are multiple errors. I recommend to simplify your code to narrow the problem, e.g. test your producer-consumer hand-off and the network operations independently.
One obvious problem is that you are trying to signal the end of a file via an AtomicBoolean but your consumer isn’t actually testing it before taking items. If you look at the place where it takes items, there is an inner loop:
while(counter < rate[index]) {
try {
line = buffer.take();
…
Since the producer has no influence on the counter < rate[index] condition, it has no control over how many lines the consumer will attempt to take before checking the state of the fileScanFlag.
But even if you try to fix this by checking the boolean flag right before take, the result is broken due to possible race conditions. The atomic boolean and the blocking queue are both thread-safe on their own but your combination of the two is not.
Putting the last item on the queue and setting the flag are two distinct operations. Right in-between these two actions, the consumer can take the last item, recheck the flag and find it being false and go to the next attempt to take while the producer is about to set it to true.
One solution is to change the order of the operations on the consumer side, which requires resorting to polling:
polling: for(;;) {
line = buffer.poll(timeout, timeOutUnit); // control the cpu consumption via timeout
if(line!=null) break polling;
if(fileScanFlag.get()) break outerLoop;
}
An alternative is not to use two different communication constructs. Instead of maintaining a boolean flag, place an end marker object to the queue once the file reached an end. This is one of the rare cases, where using the identity of a String rather than equals is appropriate:
public class SourceFileProducer implements Runnable {
private String endMarker;
…
public SourceFileProducer(LinkedBlockingQueue<String> buffer,
String endMarker, String fileName) {
this.buffer = buffer;
this.endMarker = endMarker;
…
#Override
public void run() {
System.out.println("SourceFileProducer thread-" + Thread.currentThread().getId()
+ " initiating with source file: " + fileName);
String line;
try {
while((line = reader.readLine()) != null) buffer.put(line);
} catch (IOException|InterruptedException e) {
e.printStackTrace();
}
buffer.put(endMarker);
}
public class SourceFileConsumer implements Runnable {
private String endMarker;
…
public SourceFileConsumer(LinkedBlockingQueue<String> buffer, String endMarker,
ServerSocket serverSocket, Socket client, long checkpoint[], int rate[]) {
this.buffer = buffer;
this.endMarker = endMarker;
…
line = buffer.take();
if(line==endMarker) break;
The value of the end marker doesn’t matter but it’s object identity. Hence, the code which creates the two threads must contain something like:
// using new to ensure unique identity
private static final String EOF = new String("end of file");
…
new SourceFileProducer(queue, EOF, …)
new SourceFileConsumer(queue, EOF, …)
The new operator guarantees to produce an object with a unique identity, therefore, comparing that marker object with any other String, i.e. the lines returned by BufferedReader, via == will always evaluate to false. Care must be taken not to let the marker object escape to code not knowing about its special role.
This question already has answers here:
Threads and file writing
(6 answers)
Closed 9 years ago.
i have 20 threads that write with the println() function on a file called results.txt. How can i synchronize them all?
I note every time my program run i have different number of lines of text in results.txt.
Thank you.
Access the file through a class that contains a synchronized method to write to the file. Only one thread at a time will be able to execute the method.
I think that Singleton pattern would fit for your problem:
package com.test.singleton;
public class Singleton {
private static final Singleton inst= new Singleton();
private Singleton() {
super();
}
public synchronized void writeToFile(String str) {
// Do whatever
}
public static Singleton getInstance() {
return inst;
}
}
Every time you need to write to your file, you only would have to call:
Singleton.getInstance().writeToFile("Hello!!");
Duplicate question ... duplicate answer. As I said here:
If you can hold your file as a FileOutputStream you can lock it like this:
FileOutputStream file = ...
....
// Thread safe version.
void write(byte[] bytes) {
try {
boolean written = false;
do {
try {
// Lock it!
FileLock lock = file.getChannel().lock();
try {
// Write the bytes.
file.write(bytes);
written = true;
} finally {
// Release the lock.
lock.release();
}
} catch ( OverlappingFileLockException ofle ) {
try {
// Wait a bit
Thread.sleep(0);
} catch (InterruptedException ex) {
throw new InterruptedIOException ("Interrupted waiting for a file lock.");
}
}
} while (!written);
} catch (IOException ex) {
log.warn("Failed to lock " + fileName, ex);
}
}
You are intend to write data into one file. So if you try to lock the whole file, it'd better to use a single thread to do this job. Although you spawn 20 threads, but there is only one of them is running every time you call the method, the others are just waiting for the lock.
I recommend you use RandomAccessFile to write data to your file. Then each thread can write some unique data into to the file without locking the whole file.
Some demo code as following
try {
final RandomAccessFile file = new RandomAccessFile("/path/to/your/result.txt", "rw");
final int numberOfThread = 20;
final int bufferSize = 512;
ExecutorService pool = Executors.newFixedThreadPool(numberOfThread);
final AtomicInteger byteCounter = new AtomicInteger(0);
final byte[] yourText = "Your data".getBytes();
for (int i = 0; i < yourText.length; i++) {
pool.submit(new Runnable() {
#Override
public void run() {
int start = byteCounter.getAndAdd(bufferSize);
int chunkSize = bufferSize;
if (start + bufferSize > yourText.length) {
chunkSize = yourText.length - start;
}
byte[] chunkData = new byte[chunkSize];
System.arraycopy(yourText, start, chunkData, 0, chunkSize);
try {
file.write(chunkData);
} catch (IOException e) {
//exception handle
}
}
});
}
file.close();
} catch (Exception e) {
//clean up
}