I'm writing to a file via multiple threads so I am using the following code:
synchronized (this) {
BufferedWriter bw = new BufferedWriter(
new FileWriter(path, true));
bw.write(data);
}
and I'm wondering for educational purposes how I can use some synchronizer (Semaphore, CountDownLatch, CyclicBarrier, Phaser or Exchanger) to essentially achieve the same thing: safe multithreaded writing.
Thanks
Honestly, the much better nearly tautological way to write from multiple sources to the same writer is to use a (blocking) queue. This way, all threads can drop their messages on the queue, and the writer thread can pick the messages off the queue and write them to the file. It's generally easier to implement, as well as more efficient
-- edit --
Example:
public class MyQueueableWriter implements Runnable {
private BlockingQueue<Msg> q = new BlockingQueue();
private FileOutputStream fis = ...;
private volatile boolean running = true;
public MyQueueableWriter(FileOutputStream fis) {
this.fis = fis;
}
public void run() {
try {
while (running) {
Message m = q.take();
fis.write(...);
}
fis.close();
} catch (IOException iox) {
...
}
}
public void addMsg(Msg m) {
q.put(m);
}
public void stop() {
running = false;
}
}
Then adding to the queue:
public class EnqueueMsgRunnable implements Runnable {
MyQueueableWriter q = ...;
q.put(myMessage);
q.put(myMessage2);
}
Then just
for (int i =0; i < numSources; i++) {
EnqueueMsgRunnable r = new EnqueueMsgRunnable(...);
new Thread(r).start();
}
Related
I have the following work queue implementation, which I use to limit the number of threads in use. It works by me initially adding a number of Runnable objects to the queue, and when I am ready to begin, I run "begin()". At this point I do not add any more to the queue.
public class WorkQueue {
private final int nThreads;
private final PoolWorker[] threads;
private final LinkedList queue;
Integer runCounter;
boolean hasBegun;
public WorkQueue(int nThreads) {
runCounter = 0;
this.nThreads = nThreads;
queue = new LinkedList();
threads = new PoolWorker[nThreads];
hasBegun = false;
for (int i = 0; i < nThreads; i++) {
threads[i] = new PoolWorker();
threads[i].start();
}
}
public boolean isQueueEmpty() {
synchronized (queue) {
if (queue.isEmpty() && runCounter == 0) {
return true;
} else {
return false;
}
}
}
public void begin() {
hasBegun = true;
synchronized (queue) {
queue.notify();
}
}
public void add(Runnable r) {
if (!hasBegun) {
synchronized (queue) {
queue.addLast(r);
runCounter++;
}
} else {
System.out.println("has begun executing. Cannot add more jobs ");
}
}
private class PoolWorker extends Thread {
public void run() {
Runnable r;
while (true) {
synchronized (queue) {
while (queue.isEmpty()) {
try {
queue.wait();
} catch (InterruptedException ignored) {
}
}
r = (Runnable) queue.removeFirst();
}
// If we don't catch RuntimeException,
// the pool could leak threads
try {
r.run();
synchronized (runCounter) {
runCounter--;
}
} catch (RuntimeException e) {
// You might want to log something here
}
}
}
}
}
This is a runnable I use to keep track of when all the jobs on the work queue have finished:
public class QueueWatcher implements Runnable {
private Thread t;
private String threadName;
private WorkQueue wq;
public QueueWatcher(WorkQueue wq) {
this.threadName = "QueueWatcher";
this.wq = wq;
}
#Override
public void run() {
while (true) {
if (wq.isQueueEmpty()) {
java.util.Date date = new java.util.Date();
System.out.println("Finishing and quiting at:" + date.toString());
System.exit(0);
break;
} else {
try {
Thread.sleep(1000);
} catch (InterruptedException ex) {
Logger.getLogger(PlaneGenerator.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
}
public void start() {
wq.begin();
System.out.println("Starting " + threadName);
if (t == null) {
t = new Thread(this, threadName);
t.setDaemon(false);
t.start();
}
}
}
This is how I use them:
Workqueue wq = new WorkQueue(9); //Get same results regardless of 1,2,3,8,9
QueueWatcher qw = new QueueWatcher(wq);
SomeRunnable1 sm1 = new SomeRunnable1();
SomeRunnable2 sm2 = new SomeRunnable2();
SomeRunnable3 sm3 = new SomeRunnable3();
SomeRunnable4 sm4 = new SomeRunnable4();
SomeRunnable5 sm5 = new SomeRunnable5();
wq.add(sm1);
wq.add(sm2);
wq.add(sm3);
wq.add(sm4);
wq.add(sm5);
qw.start();
But regardless of how many threads I use, the result is always the same - it always takes about 1m 10seconds to complete. This is about the same as when I just did a single threaded version (when everything ran in main()).
If I set wq to (1,2,3--9) threads it is always between 1m8s-1m10s. What is the problem ? The jobs (someRunnable) have nothing to do with each other and cannot block each other.
EDIT: Each of the runnables just read some image files from the filesystems and create new files in a separate directory. The new directory eventually contains about 400 output files.
EDIT: It seems that only one thread is always doing work. I made the following changes:
I let the Woolworker store an Id
PoolWorker(int id){
this.threadId = id;
}
Before running I print the id of the worker.
System.out.println(this.threadId + " got new task");
r.run();
In WorkQueue constructor when creating the poolworkers I do:
for (int i = 0; i < nThreads; i++) {
threads[i] = new PoolWorker(i);
threads[i].start();
}
But it seems that that only thread 0 does any work, as the output is always:
0 got new task
Use queue.notifyAll() to start processing.
Currently you're using queue.notify(), which will only wake a single thread. (The big clue that pointed me to this was when you mentioned only a single thread was running.)
Also, synchronizing on Integer runCounter isn't doing what you think it's doing - runCounter++ is actually assigning a new value to the Integer each time, so you're synchronizing on a lot of different Integer objects.
On a side note, using raw threads and wait/notify paradigms is complicated and error-prone even for the best programmers - it's why Java introduced the java.util.concurrent package, which provide threadsafe BlockingQueue implementations and Executors for easily managing multithreaded apps.
So I'm trying to create a client/server program. I want to know when my client disconnects of his own accord, so I've setup a heartbeat system. Every 6 seconds my client sends a ping to my server, if the client doesn't send a ping for a total of 30 seconds the client is considered disconnected and removed from the current connections list (for which I plan to implement a GUI). Or at least, that's the plan.
ConnectionManager.java
public class ConnectionManager implements Runnable{
static Socket connection;
private ArrayList<Thread> allConnections;
private ArrayList<Connection> allConnectionList;
private ServerSocket server;
private int id = 0;
public ConnectionManager() {
allConnections = new ArrayList<Thread>();
allConnectionList = new ArrayList<Connection>();
}
#Override
public void run() {
try {
server = new ServerSocket(5555);
System.out.println("Server is running!");
while(true) {
connection = server.accept();
Connection a = new Connection(connection, id);
Runnable runnable = a;
allConnectionList.add(a);
allConnections.add(new Thread(runnable));
allConnections.get(allConnections.size() - 1).start();
id++;
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public void removeConnection(int id) {
allConnections.remove(id);
allConnectionList.remove(id);
}
Connection.java
public class Connection implements Runnable {
private Socket a;
public boolean amIActive;
private int id;
public Connection(Socket a, int id) {
amIActive = true;
this.a = a;
this.id = id;
}
public void onConnect() {
try {
String TimeStamp = new java.util.Date().toString();
String formattedAddress = a.getInetAddress().toString().replace("/", "");
System.out.println("Received connection from: " + formattedAddress + " at " + TimeStamp);
Runnable runnable = new ConnectionListener(this);
Thread connectionThread = new Thread(runnable);
connectionThread.start();
String returnCode = "Server repsonded to " + a.getInetAddress().toString().replace("/", "") + " at "+ TimeStamp + (char) 13;
BufferedOutputStream os = new BufferedOutputStream(a.getOutputStream());
OutputStreamWriter osw = new OutputStreamWriter(os, "US-ASCII");
osw.write(returnCode);
osw.flush();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
#Override
public void run() {
onConnect();
System.out.println("We got this far!");
while(amIActive) {
whileTrue();
}
System.out.println("This code never gets run because we get stuck in the while loop above");
Main.b.removeConnection(id);
System.out.println("Connection was closed from " + a.getInetAddress());
}
public void setOffline(boolean state) {
this.amIActive = state;
}
public void whileTrue() {
}
public Socket getSocket() {
return a;
}
ConnectionListener.java
public class ConnectionListener implements Runnable{
public Connection myConnection;
public boolean receivedHeartbeat;
public int missedHeartbeats = 0;
public ConnectionListener(Connection a) {
this.myConnection = a;
}
#Override
public void run() {
Runnable runnable = new Heartbeat(this);
Thread thread = new Thread(runnable);
thread.start();
while(myConnection.amIActive) {
try {
BufferedInputStream is;
is = new BufferedInputStream(myConnection.getSocket().getInputStream());
InputStreamReader isr = new InputStreamReader(is);
StringBuffer process = new StringBuffer();
int character;
while((character = isr.read()) != 13) { //GETTING STUCK HERE BECAUSE STUPID.
if(character == -1) {
myConnection.setOffline(true);
} else {
process.append((char)character);
}
}
handleInput(process);
} catch (Exception e) {
e.printStackTrace();
}
}
}
public void handleInput(StringBuffer process) {
String messageSent = process.toString();
if(messageSent.equals("Ping!")) {
receivedHeartbeat = true;
}
}
Heartbeat.java
public class Heartbeat implements Runnable{
private ConnectionListener b;
public Heartbeat(ConnectionListener a) {
b = a;
}
#Override
public void run() {
while(true) {
try {
Thread.sleep(1000);
if(b.missedHeartbeats > 5) {
b.myConnection.amIActive = false;
System.out.println("Setting amIActiveToFalse!");
}
if(b.receivedHeartbeat) {
b.receivedHeartbeat = false;
} else {
b.missedHeartbeats++;
}
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
My console is spammed with System.out.println("Setting amIActiveToFalse!"); from Heartbeat.java. But the while loop in Connection.java keeps running. I believe this might be something to do with my threading, but I can't figure it out.
When you have a non-volatile variable, there is no guarentee of visability of a change in one thread to another. In particular, if the JVM detects that a thread doesn't alter a boolean it can inline it, meaning you will never see the value change.
The simple solution is to make the boolean volatile and it will not be inlined and one thread will see when another changes it.
For more details http://vanillajava.blogspot.com/2012/01/demonstrating-when-volatile-is-required.html
The trivial answer to this is: make the variable volatile.
Without this, it is allowed for the thread changing the value to basically keep its updates in cache, committing them to main memory some time later.
This allows threaded code to run much faster, since it can keep its variables in cache rather than having to fetch from main memory. However, the consequence of this is that other threads don't see the update.
Making the variable volatile prevents this from happening: a thread always reads the value from main memory, and writes are immediately committed.
I say that this is the trivial answer because it doesn't necessarily fix all of your problems. There may also be an atomicity issue: in between one thread reading the variable and writing it again, another thread might sneak in and change its value, which may or may not put the first thread into an undefined state from the perspective of its invariants.
Specifically:
if(b.receivedHeartbeat) { b.receivedHeartbeat = false;
It is possible that some other thread can change b.receivedHeartbeat to false after this thread evaluates it to true, so this iteration is erroneously counted as a "non-missed" heartbeat.
This can be fixed by making the variable a (non-volatile) AtomicBoolean, on which there is an atomic compare-and-set method, which avoids such race conditions.
Java Concurrency In Practice is a great reference on these issues, I wholeheartedly recommend it. Look for the topics "visibility" and "atomicity".
Also read the advanced chapter on the Java Memory Model. That made me doubt myself at first, but made me a much stronger programmer after I digested it.
There are a couple issues I saw while debugging the code you posted, but I was able to successfully get the heartbeat functionality working.
In the Connection Listener class I don't think the if statement with .equals("Ping!") will match, because of the newline character at the end of each line.
In the Connection Listener class I would probably put the socket's Input Stream at the top of the loop not inside the loop. (I don't think this will break it but it's probably nicer this way)
ConnectionListener Updates:
public void run() {
Runnable runnable = new Heartbeat(this);
Thread thread = new Thread(runnable);
thread.start();
BufferedReader br = null;
try {
//is = new BufferedInputStream(myConnection.getSocket().getInputStream());
br = new BufferedReader(new InputStreamReader(myConnection.getSocket().getInputStream()));
} catch (IOException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
while(myConnection.amIActive) {
try {
String processLine = br.readLine();
System.out.println("handleInput:" + processLine);
handleInput(processLine);
} catch (Exception e) {
System.out.println("Exception!");
e.printStackTrace();
}
}
}
public void handleInput(String messageSent) {
if(messageSent.startsWith("Ping!")) { //Need to use startsWith, or add newline character
receivedHeartbeat = true;
System.out.println("receivedHeartbeat!");
}
}
Also, in your Heartbeat class make sure you reset the missedHeartbeats counter to 0 on true:
if(b.receivedHeartbeat) {
b.receivedHeartbeat = false;
b.missedHeartbeats = 0;
} else {
b.missedHeartbeats++;
}
This question already has answers here:
How to wait for all threads to finish, using ExecutorService?
(27 answers)
Closed 8 years ago.
Please have a look at the following code.
public class BigFileWholeProcessor {
private static final int NUMBER_OF_THREADS = 2;
public void processFile(String fileName) {
BlockingQueue<String> fileContent = new LinkedBlockingQueue<String>();
BigFileReader bigFileReader = new BigFileReader(fileName, fileContent);
BigFileProcessor bigFileProcessor = new BigFileProcessor(fileContent);
ExecutorService es = Executors.newFixedThreadPool(NUMBER_OF_THREADS);
es.execute(bigFileReader);
es.execute(bigFileProcessor);
es.shutdown();
if(es.isTerminated())
{
System.out.println("Completed Work");
}
}
}
public class BigFileReader implements Runnable {
private final String fileName;
int a = 0;
public static final String SENTINEL = "SENTINEL";
private final BlockingQueue<String> linesRead;
public BigFileReader(String fileName, BlockingQueue<String> linesRead) {
this.fileName = fileName;
this.linesRead = linesRead;
}
#Override
public void run() {
try {
//since it is a sample, I avoid the manage of how many lines you have read
//and that stuff, but it should not be complicated to accomplish
BufferedReader br = new BufferedReader(new FileReader(new File("E:/Amazon HashFile/Hash.txt")));
String str = "";
while((str=br.readLine())!=null)
{
linesRead.put(str);
System.out.println(a);
a++;
}
linesRead.put(SENTINEL);
} catch (Exception ex) {
ex.printStackTrace();
}
System.out.println("Completed");
}
}
public class BigFileProcessor implements Runnable {
private final BlockingQueue<String> linesToProcess;
public BigFileProcessor (BlockingQueue<String> linesToProcess) {
this.linesToProcess = linesToProcess;
}
#Override
public void run() {
String line = "";
try {
while ( (line = linesToProcess.take()) != null) {
//do what you want/need to process this line...
if(line==BigFileReader.SENTINEL)
{
break;
}
String [] pieces = line.split("(...)/g");
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
I want to print the text "completed work" in BigFileWholeProcessor once all the thread work is done. But instead, it is not getting printed. Why is this? How to identify that all the threads are done and need printing?
shutdown() only signal ES to shutdown, you need
awaitTermination(long timeout, TimeUnit unit)
before print message
Use submit() method instead of execute(). The get() method can be used if you want to wait for the thread to finish at any point of time. Read documentation on use of Future object for further details.
ExecutorService es = Executors.newFixedThreadPool(2);
Future<?> f = es.submit(new Thread(new TestRun()));
f.get(); // Wait for result... (i.e similar to `join()` in this case)
es.shutdown(); // Shutdown ExecutorService
System.out.println("Done.");
I have defined a TestRun class implementing Runnable, not shown here. The Future object makes more sense in other scenarios.
Im facing one problem in streaming data capture for reading the broadcast data during multithreading, pls help or suggest,
Actually there is one class which is reading data from one of the udp socket. Another class accepts the tcp connection from every client request, creates a thread for every client and request the same udp class for data. The thing is working with 1st thread which gets created. But when i request with another client from another pc/ip the packets get losted to the 2nd client/thread
I have made a workaround by creating a list where im storing the Threads outputstream object
and looping it to send the data to all the client. But this is just temporary as it ll delay the packets if clients/connections gets increased.
code for reading UDP Data
public class EventNotifier
{
private InterestingEvent ie;
public DatagramSocket clientSocket;
public String[] split_str;
byte[] receiveData;
HashMap<String, String> secMap = new HashMap<String, String>();
public EventNotifier(InterestingEvent event)
{
ie = event;
clientSocket = new DatagramSocket(9050);
receiveData = new byte[500];
}
public String getDataFeed(String client_id)
{
try
{
DatagramPacket receivePacket = new DatagramPacket(receiveData, receiveData.length);
clientSocket.receive(receivePacket);
String s = new String(receivePacket.getData());
String split_str = s.split(",");
if(secMap.containsValue(split_str[0]))
return s;
else
return "";
} catch(Exception e3) {}
}
}// end of eventNotifier class
code for multithreading handling client requests
public class multiServer
{
static protected List<PrintWriter> writers = new ArrayList<PrintWriter>();
static String client_id = "";
public static void main(String[] args)
{
try
{
ServerSocket servsock = new ServerSocket(8858);
Socket incoming;
while(true)
{
incoming = servsock.accept();
multiServerThread connection = new multiServerThread(incoming);
Thread t1 = new Thread(connection);
t1.start();
}
}
catch(IOException e)
{
System.out.println("couldnt make socket");
}
}
}
class multiServerThread extends Thread implements InterestingEvent
{
Socket incoming;
PrintWriter out=null;
PrintWriter broad=null;
BufferedReader in = null;
String cliString=null;
private EventNotifier en;
int id;
public static String udp_data;
public void interestingEvent(String str1)
{
this.udp_data = str1;
}
public String getUdpData()
{
String _udp_data = this.udp_data;
return _udp_data;
}
multiServerThread(Socket incoming)
{
this.incoming=incoming;
en = new EventNotifier(this);
}
public void run()
{
try
{
out = new PrintWriter(incoming.getOutputStream(), true);
in = new BufferedReader(new InputStreamReader(incoming.getInputStream()));
cliString = in.readLine();
multiServer.writers.add(out);
while(true)
{
try
{
udp_data = en.getDataFeed(cliString);
if(udp_data!=null && udp_data.length()>0)
{
//workaround for serving the data to all cleints who are connected
for (int i=0; i<multiServer.writers.size();i++)
{
broad=multiServer.writers.get(i);
broad.println(udp_data.trim());
}
//else will directly write to the outputstream object for every thread which is connected
// out.println(udp_data.trim());
}
}
catch (Exception e)
{
System.out.println("exception "+e);
}
Thread.sleep(1);
}
} catch(IOException e)
{
System.out.print("IO Exception :: "+ e);
}
catch(InterruptedException e)
{
System.out.print("exception "+ e);
}
}
}
You need mutual exclusion (or a different design).
For example, what will happen if two threads call multiServer.writers.add(out); concurrently?
From the ArrayList Javadocs
Note that this implementation is not synchronized. If multiple threads access an ArrayList instance concurrently, and at least one of the threads modifies the list structurally, it must be synchronized externally. (A structural modification is any operation that adds or deletes one or more elements, or [...])
Another problem is two calling udp_data = en.getDataFeed(cliString); concurrently. The second thread might overwrite the result of the first. You'll loose data!
What happens if one thread calls for (int i=0; i<multiServer.writers.size();i++) while another thread is busy doing multiServer.writers.add(out);? The size may have increased, before out has actually been added to the list!
public class multiServer
{
private List<PrintWriter> writers = new ArrayList<PrintWriter>();
public synchronized void addWriter(PrintWrite out) {
writers.add(out);
}
public synchronized void serveAllWriters(String data) {
for (int i=0; i<multiServer.writers.size();i++)
{
broad=multiServer.writers.get(i);
broad.println(data);
}
}
}
Now when a thread tries to add a writer, the synchronizeds will make sure no other thread is adding or printing. So multiServerThread should be fixed to use the new methods:
class multiServerThread extends Thread implements InterestingEvent
{
//...
private String udp_data;
//...
myMultiServer.addWriter(out);
//...
udp_data = en.getDataFeed(cliString);
if(udp_data!=null && udp_data.length()>0)
myMultiServer.serveAllWriters(udp_data.trim());
//...
}
There might be more problems, not sure I don't fully understand your code. The question you must ask yourself is, can another thread read and/or write the same data or object? Yes? Then you'll need proper synchronization.
I would like to know if multiple threads try to access a single txt file, how to restrict it?
If thread A tries to access the file till it completes the reading and writing part, other threads must wait. Here is what I tried.
package singleton;
/**
*
* #author Admin
*/
import java.io.*;
class ReadFileUsingThread
{
public synchronized void readFromFile(final String f, Thread thread) {
Runnable readRun = new Runnable() {
public void run() {
FileInputStream in=null;
FileOutputStream out=null;
String text = null;
try{
Thread.sleep(5000);
File inputFile = new File(f);
in = new FileInputStream(inputFile);
byte bt[] = new byte[(int)inputFile.length()];
in.read(bt);
text = new String(bt);
//String file_name = "E:/sumi.txt";
//File file = new File(file_name);
// FileWriter fstream = new FileWriter("E:/sumi.txt");
out = new FileOutputStream("E:/sumi.txt");
out.write(bt);
System.out.println(text);
} catch(Exception ex) {
}
}
};
thread = new Thread(readRun);
thread.start();
}
public static void main(String[] args)
{
ReadFileUsingThread files=new ReadFileUsingThread();
Thread thread1=new Thread();
Thread thread2=new Thread();
Thread thread3=new Thread();
String f1="C:/Users/Admin/Documents/links.txt";//,f2="C:/employee.txt",f3="C:/hello.txt";
thread1.start();
files.readFromFile(f1,thread1);
thread2.start();
files.readFromFile(f1,thread2);
thread3.start();
files.readFromFile(f1,thread3);
}
}
An interesting way to do it would be to intern the string value of the file's FQN, and then synchronize on it. The more 'traditional' way is to use the FileChannel object and lock on it, with other processes simply waiting on the lock, to take their turns.
Caveat : None of these solutions will solve contention between JVM's, or contention between a JVM and other external programs.
You can use a ReentrantReadWriteLock.
ReadWriteLock lock = new ReentrantReadWriteLock();
...
lock.readLock().lock();
try {
//do reading stuff in here
} finally {
lock.readLock().unlock();
}
...
lock.writeLock().lock();
try {
//do writing stuff in here
} finally {
lock.writeLock().unlock();
}
Or, for something simpler, you could synchronize on the interned (interning ensures that the String object is shared) String object that represents that File's full path name:
synchronized(file.getAbsolutePath().intern()) {
//do operations on that file here
}
The ReadWriteLock approach will have better performance as Threads will be allowed to read the file at the same time while manually synchronizing does not allow this.