I am writing data to file using a queue on a separate thread, but the process consumes around 25% of CPU, as shown in this test main.
Is there something I can do to resolve this issue?
Perhaps I should be using flush() somewhere?
The test shows the main method start and run the queue thread and then send created data to it. The queue thread writes the data to a BufferedWriter which handles writing the data to a file.
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.logging.Level;
import java.util.logging.Logger;
import uk.co.moonsit.utils.timing.Time;
public class OutputFloatQueueReceiver extends Thread {
private static final Logger LOG = Logger.getLogger(OutputFloatQueueReceiver.class.getName());
private ConcurrentLinkedQueue<List<Float>> queue = null;
private boolean running = true;
private final BufferedWriter outputWriter;
private int ctr = 0;
private final int LIMIT = 1000;
public OutputFloatQueueReceiver(String outputFile, String header, ConcurrentLinkedQueue<List<Float>> q) throws IOException {
queue = q;
File f = new File(outputFile);
FileWriter fstream = null;
if (!f.exists()) {
try {
f.getParentFile().mkdirs();
if (!f.createNewFile()) {
throw new IOException("Exception when trying to create file " + f.getAbsolutePath());
}
fstream = new FileWriter(outputFile, false);
} catch (IOException ex) {
//Logger.getLogger(ControlHierarchy.class.getName()).log(Level.SEVERE, null, ex);
throw new IOException("Exception when trying to create file " + f.getAbsolutePath());
}
}
fstream = new FileWriter(outputFile, true);
outputWriter = new BufferedWriter(fstream);
outputWriter.append(header);
}
public synchronized void setRunning(boolean running) {
this.running = running;
}
#Override
public void run() {
while (running) {
while (queue.peek() != null) {
if (ctr++ % LIMIT == 0) {
LOG.log(Level.INFO, "Output Queue size = {0} '{'ctr={1}'}'", new Object[]{queue.size(), ctr});
}
List<Float> list = queue.poll();
if (list == null) {
continue;
}
try {
StringBuilder sbline = new StringBuilder();
Time t = new Time(list.get(0));
sbline.append(t.HMSS()).append(",");
for (Float f : list) {
sbline.append(f).append(",");
}
sbline.append("\n");
outputWriter.write(sbline.toString());
} catch (IOException ex) {
LOG.info(ex.toString());
break;
}
}
}
if (outputWriter != null) {
try {
outputWriter.close();
LOG.info("Closed outputWriter");
} catch (IOException ex) {
Logger.getLogger(OutputFloatQueueReceiver.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
public static void main(String[] args) {
try {
String outputFile = "c:\\tmp\\qtest.csv";
File f = new File(outputFile);
f.delete();
StringBuilder header = new StringBuilder();
header.append("1,2,3,4,5,6,7,8,9");
header.append("\n");
ConcurrentLinkedQueue<List<Float>> outputQueue = null;
OutputFloatQueueReceiver outputQueueReceiver = null;
outputQueue = new ConcurrentLinkedQueue<>();
outputQueueReceiver = new OutputFloatQueueReceiver(outputFile, header.toString(), outputQueue);
outputQueueReceiver.start();
for (int i = 1; i < 100000; i++) {
List<Float> list = new ArrayList<>();
//list.set(0, (float) i); // causes exception
list.add((float) i);
for (int j = 1; j < 10; j++) {
list.add((float) j);
}
outputQueue.add(list);
}
try {
Thread.sleep(5000);
} catch (InterruptedException ex) {
Logger.getLogger(OutputFloatQueueReceiver.class.getName()).log(Level.SEVERE, null, ex);
}
outputQueueReceiver.setRunning(false);
} catch (IOException ex) {
Logger.getLogger(OutputFloatQueueReceiver.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
This code is the reason while your code is using so much CPU:
while (running) {
while (queue.peek() != null) {
// logging
List<Float> list = queue.poll();
if (list == null) {
continue;
}
// do stuff with list
}
}
Basically, your code is busy-waiting, repeatedly "peeking" until a queue entry becomes available. It is probably spinning there in a tight loop.
You should replace your queue class with a BlockingQueue, and simply use take() ... like this:
while (running) {
List<Float> list = queue.take();
// do stuff with list
}
The take() call block indefinitely, only returning once there is an element available, and returning that element as the result. If blocking indefinitely is a problem, you could either use poll(...) with a timeout, or you could arrange that some other thread interrupts the thread that is blocked.
Related
so what I'm trying to accomplish with my code is to read tasks off of a document with an end goal of sorting them by a priority (given from a separate class) into a LinkedList using a ListIterator. I have all of my tasks going into a LinkedList, however I've made a mistake somewhere in the organizing part:
public static void main(String[] args) {
Task newTask = null;
LinkedList<Task> list = new LinkedList();
TaskReader reader = new TaskReader("tasks.txt");
list.add(reader.provideTask());
ListIterator<Task> iter = list.listIterator();
for (int i = 1; i < 20; i++) {
newTask = reader.provideTask();
while (iter.hasNext()) {
if (newTask.getPriority() <= iter.next().getPriority()) {
iter.add(newTask);
} else {
iter.previous();
iter.add(newTask);
iter.next();
}
}
iter = list.listIterator();
}
while (iter.hasNext()) {
System.out.println(iter.next().getPriority());
}
}
Right now I am get a huge amount of output, where I should only be getting 20 lines (and they should be organized from 10-1 by priority given from separate class). Any help would be awesome! Thank you!
Below is the TaskReader class
package versionTwo;
import java.io.File;
import java.io.FileNotFoundException;
import java.util.Scanner;
public class TaskReader implements TaskProvider {
private Scanner source = null;
public TaskReader(String fileName) {
try {
source = new Scanner(new File(fileName));
} catch (FileNotFoundException er) {
System.out.println("Error opening input file " + fileName + " for reading.");
}
}
public Task provideTask() {
Task newTask = null;
if (source != null) {
try {
newTask = new Task(source.nextInt(), source.nextInt());
} catch (Exception ex) {
System.out.println("Error while trying to read task.");
}
}
return newTask;
}
}
I am working on a Java process that contains 2 threads: one for reading a file's contents and adding them in one shared blocking queue; and one for retrieving the lines from the blocking queue and sending them through the network (under a specified send rate). The two classes I have are the following:
Updated Code below
Producer Thread:
import java.io.BufferedReader;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.util.concurrent.ArrayBlockingQueue;
public class SourceFileProducer implements Runnable {
private File file;
private BufferedReader reader;
private ArrayBlockingQueue<String> buffer;
private String fileName;
private String endMarker;
public SourceFileProducer(ArrayBlockingQueue<String> buffer,
String endMarker, String fileName) {
this.buffer = buffer;
this.endMarker = endMarker;
file = new File(fileName);
if(file.exists()) {
try {
reader = new BufferedReader(new FileReader(file));
} catch (FileNotFoundException e) {
e.printStackTrace();
}
}
this.fileName = fileName;
}
#Override
public void run() {
System.out.println("SourceFileProducer thread-" + Thread.currentThread().getId() + " initiating with source file: " + fileName);
String line = "";
try {
while((line = reader.readLine()) != null) {
try {
buffer.put(line);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
try {
buffer.put(endMarker);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println("SourceFileProducer thread-" + Thread.currentThread().getId() + " scanned and buffered the whole file.");
} catch (IOException e) {
e.printStackTrace();
}
}
}
and the Consumer thread:
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.PrintWriter;
import java.net.ServerSocket;
import java.net.Socket;
import java.util.concurrent.ArrayBlockingQueue;
public class SourceFileConsumer implements Runnable {
private ArrayBlockingQueue<String> buffer;
private BufferedReader socketInput;
private PrintWriter socketOutput;
private Socket client;
private ServerSocket serverSocket;
private long checkpoint[] = null;
private int rate[] = null;
private String endMarker;
public SourceFileConsumer(ArrayBlockingQueue<String> buffer, String endMarker,
ServerSocket serverSocket, Socket client, long checkpoint[], int rate[]) {
this.buffer = buffer;
this.endMarker = endMarker;
this.client = client;
try {
socketOutput = new PrintWriter(client.getOutputStream(), true);
socketInput = new BufferedReader(new InputStreamReader(client.getInputStream()));
} catch (IOException e) {
e.printStackTrace();
}
this.checkpoint = new long[checkpoint.length];
this.rate = new int[rate.length];
for(int i = 0; i < checkpoint.length; i++) {
this.checkpoint[i] = checkpoint[i];
this.rate[i] = rate[i];
}
this.serverSocket = serverSocket;
}
#Override
public void run() {
String line = null;
long start = System.currentTimeMillis();
int index = 0;
boolean fileScanFlag = true;
while(fileScanFlag) {
long startTimestamp = System.currentTimeMillis();
long interval = (startTimestamp - start) / 1000L;
if(interval >= checkpoint[index]) {
if(index < checkpoint.length - 1) {
if(interval >= checkpoint[index + 1]) {
index += 1;
System.out.println("SourceFileConsumer thread-" + Thread.currentThread().getId() +
" progressed to checkpoint " + index + " with rate: " + rate[index]);
}
}
}
int counter = 0;
while(counter < rate[index]) {
try {
line = buffer.take();
} catch (InterruptedException e1) {
e1.printStackTrace();
}
if(line == endMarker) {
fileScanFlag = false;
break;
}
if(socketOutput != null && socketOutput.checkError()) {
System.out.println("SourceFileConsumer Thread-" + Thread.currentThread().getId() + " detected broken link...");
try {
client = serverSocket.accept();
socketOutput = new PrintWriter(client.getOutputStream(), true);
socketInput = new BufferedReader(new InputStreamReader(client.getInputStream()));
} catch(IOException e) {
e.printStackTrace();
}
System.out.println("SourceFileConsumer Thread-" + Thread.currentThread().getId() + " re-established connection...");
}
if(socketOutput != null)
socketOutput.println(line);
counter += 1;
}
long endTimestamp = System.currentTimeMillis();
if(endTimestamp - startTimestamp <= 1000) {
System.out.println("thread-" + Thread.currentThread().getId() + " input rate: " + counter +
", wait time: " + (1000 - (endTimestamp - startTimestamp)));
try {
Thread.sleep((1000 - (endTimestamp - startTimestamp)));
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
if(socketInput != null && socketOutput != null && client != null) {
try {
socketInput.close();
socketOutput.close();
client.close();
} catch(IOException e) {
e.printStackTrace();
}
}
System.out.println("SourceFileConsumer Thread-" + Thread.currentThread().getId() + " transfer complete.");
}
}
The problem is that, after a while, both threads hang and no tuples are sent. When I run a top command in my Linux machine, I see that the Java process, in which the two threads are running in, uses a really small amount of CPU time. Why is this happening? Is this a problem with starvation? I think that starvation can be avoided by using the LinkedBlockingQueue.
Any hints?
Thanks,
Nick
That’s quite a lot of code, especially within your consumer. So it’s not possible to preclude that there are multiple errors. I recommend to simplify your code to narrow the problem, e.g. test your producer-consumer hand-off and the network operations independently.
One obvious problem is that you are trying to signal the end of a file via an AtomicBoolean but your consumer isn’t actually testing it before taking items. If you look at the place where it takes items, there is an inner loop:
while(counter < rate[index]) {
try {
line = buffer.take();
…
Since the producer has no influence on the counter < rate[index] condition, it has no control over how many lines the consumer will attempt to take before checking the state of the fileScanFlag.
But even if you try to fix this by checking the boolean flag right before take, the result is broken due to possible race conditions. The atomic boolean and the blocking queue are both thread-safe on their own but your combination of the two is not.
Putting the last item on the queue and setting the flag are two distinct operations. Right in-between these two actions, the consumer can take the last item, recheck the flag and find it being false and go to the next attempt to take while the producer is about to set it to true.
One solution is to change the order of the operations on the consumer side, which requires resorting to polling:
polling: for(;;) {
line = buffer.poll(timeout, timeOutUnit); // control the cpu consumption via timeout
if(line!=null) break polling;
if(fileScanFlag.get()) break outerLoop;
}
An alternative is not to use two different communication constructs. Instead of maintaining a boolean flag, place an end marker object to the queue once the file reached an end. This is one of the rare cases, where using the identity of a String rather than equals is appropriate:
public class SourceFileProducer implements Runnable {
private String endMarker;
…
public SourceFileProducer(LinkedBlockingQueue<String> buffer,
String endMarker, String fileName) {
this.buffer = buffer;
this.endMarker = endMarker;
…
#Override
public void run() {
System.out.println("SourceFileProducer thread-" + Thread.currentThread().getId()
+ " initiating with source file: " + fileName);
String line;
try {
while((line = reader.readLine()) != null) buffer.put(line);
} catch (IOException|InterruptedException e) {
e.printStackTrace();
}
buffer.put(endMarker);
}
public class SourceFileConsumer implements Runnable {
private String endMarker;
…
public SourceFileConsumer(LinkedBlockingQueue<String> buffer, String endMarker,
ServerSocket serverSocket, Socket client, long checkpoint[], int rate[]) {
this.buffer = buffer;
this.endMarker = endMarker;
…
line = buffer.take();
if(line==endMarker) break;
The value of the end marker doesn’t matter but it’s object identity. Hence, the code which creates the two threads must contain something like:
// using new to ensure unique identity
private static final String EOF = new String("end of file");
…
new SourceFileProducer(queue, EOF, …)
new SourceFileConsumer(queue, EOF, …)
The new operator guarantees to produce an object with a unique identity, therefore, comparing that marker object with any other String, i.e. the lines returned by BufferedReader, via == will always evaluate to false. Care must be taken not to let the marker object escape to code not knowing about its special role.
I am working on a project trying to make several people be able to control a robot arm. For this they have to connect to a Java server that then sends the commands to a robot screen for video conferencing.
I am trying to have a thread for each client and then I want to be able to switch between the different clients based on sound, because I want the speaker to be able to control the robot.
The clients all provide positional data and the level of sound taken by the kinect, and sent to the server in the form of a string.
I am having problems with performing the switch. Currently they seem to be switching back and forth and it makes the robot go haywire.
Is there a good way of comparing the threads to each other, find the appropriate one, switch to that, all the while checking the other threads to see if or when they become the most appropriate one? While also checking in case other clients try to connect to the server?
Thank you for your help.
I also include my code in case you want to look through it and get a better idea.
This is the server class:
import java.io.IOException;
import java.net.InetAddress;
import java.net.ServerSocket;
import java.net.Socket;
import java.net.SocketTimeoutException;
import java.util.ArrayList;
import java.util.Hashtable;
public class MultiThreadedServer implements Runnable {
protected int serverPort = 8888;
protected ServerSocket serverSocket = null;
protected boolean isStopped = false;
protected Thread runningThread = null;
protected Thread clientThread = null;
protected Thread threadThread = null;
private Hashtable<Long, WorkerRunnable> Users = new Hashtable<Long, WorkerRunnable>();
private ArrayList<Thread> ClientThreads = new ArrayList<Thread>();
private WorkerRunnable client = null;
private ThreadHandler threadHandler = null;
private int sound_max = 0;
private boolean once = true;
public MultiThreadedServer (int port) {
this.serverPort = port;
}
public void run() {
synchronized(this) {
this.runningThread = Thread.currentThread();
}
openServerSocket();
threadHandler = new ThreadHandler();
while( !isStopped() ) {
Socket clientSocket = null;
try {
System.out.println(InetAddress.getLocalHost());
clientSocket = this.serverSocket.accept(); // Connect to clients
} catch (SocketTimeoutException e) {
} catch (IOException e) {
if( isStopped() ) {
System.out.println("Server Stopped");
return;
}
throw new RuntimeException("Error accepting client connection", e);
}
client = new WorkerRunnable(clientSocket, "Multithreaded Server");//Class does client work
clientThread = new Thread(client); // Make a thread for each client
clientThread.start(); // start thread
threadHandler.setUp(client, clientThread); // Set up the thread handler
if ( once == true) { // make sure the threadHandler thread is only created once
threadThread = new Thread(threadHandler);
threadThread.start();
once = false;
}
}
System.out.println("Server Stopped");
}
/**
* Check if the socket is stopped
* #return true if the socket is stopped
*/
private synchronized boolean isStopped() {
return this.isStopped;
}
/**
* Stop and close the socket
*/
public synchronized void stop() {
this.isStopped = true;
try {
this.serverSocket.close();
} catch (IOException e) {
throw new RuntimeException("Error closing server", e);
}
}
/**
* Open server socket
*/
private void openServerSocket() {
try {
this.serverSocket = new ServerSocket(this.serverPort);
} catch (IOException e) {
throw new RuntimeException("Cannot open port 8888", e);
}
}
}
This is the Worker class, that handles the data from the clients:
import gnu.io.NoSuchPortException;
import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.net.Socket;
public class WorkerRunnable implements Runnable {
protected Socket clientSocket = null;
protected String serverText = null;
private BufferedReader inFromClient;
private DataOutputStream outToClient;
private int[] currentPos = new int[6];
private boolean connected = false;
static TwoWaySerialComm serialCom = null;
static MultiServoState mState;
static int sound_average;
int[] degrees = new int[7];
int count = 0;
public WorkerRunnable(Socket clientSocket, String serverText) {
this.clientSocket = clientSocket;
this.serverText = serverText;
initCurrentPos();
if (serialCom == null) {
serialCom = new TwoWaySerialComm();
}
try {
if (!serialCom.isConnected("COM5")) {
try {
serialCom.connect("COM5");
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
mState = new MultiServoState(serialCom);
}
} catch (NoSuchPortException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public void run() {
try {
work();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
}
}
public void work() throws InterruptedException {
try {
InputStream input = clientSocket.getInputStream();
OutputStream output = clientSocket.getOutputStream();
inFromClient = new BufferedReader(new InputStreamReader(input));
outToClient = new DataOutputStream(output);
long time = System.currentTimeMillis();
updateData();
String message = null;
long endTime = System.currentTimeMillis() + 2000;
while ((message = (String) inFromClient.readLine()) != null) {
System.out.println("Message Received: " + message);
parse(message);
sound_average = degrees[6];
//
// Send the positional data to the robot
//
mState.runServo(degrees[0], degrees[1], degrees[2],
degrees[3], degrees[4], degrees[5]);
//
// Send a response information to the client application
//
currentPos[0] = mState.getCurrentPos(0);
currentPos[1] = mState.getCurrentPos(1);
currentPos[2] = mState.getCurrentPos(2);
currentPos[3] = mState.getCurrentPos(3);
currentPos[4] = mState.getCurrentPos(4);
try {
updateData();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
System.out.println("Request processed: " + time);
} catch (IOException e) {
// report exception somewhere
e.printStackTrace();
}
}
/**
* Initiate the robot's starting position.
*/
public void initCurrentPos()
{
currentPos[0] = 100;
currentPos[1] = 100;
currentPos[2] = 100;
currentPos[3] = 100;
currentPos[4] = 100;
currentPos[5] = 0;
}
/**
* Send the data to the client
*
* #throws IOException
*/
public void updateData() throws IOException {
String sentence = Integer.toString(currentPos[0]) + ", " +
Integer.toString(currentPos[1]) + ", " +
Integer.toString(currentPos[2]) + ", " +
Integer.toString(currentPos[3]) + ", " +
Integer.toString(currentPos[4]) + "." + "\n";
outToClient.flush();
outToClient.writeBytes(sentence);
}
/**
* Get the clients sound average
* #param message
*/
public int getSoundAverage() {
return sound_average;
}
public void parse(String message) {
if (message != null) {
char c;
StringBuilder sb = new StringBuilder(4);
int j = 0;
boolean help = false;
for (int i = 0; i < message.length(); i++) {
c = message.charAt(i);
if (Character.isDigit(c)) {
sb.append(c);
help = true;
}
if (!Character.isDigit(c) && help == true) {
degrees[j] = Integer.parseInt(sb.toString());
j++;
help = false;
sb.delete(0, sb.length());
}
}
}
System.out.println("Waiting for client message...");
}
/**
* Close all connections
*/
public void close() {
if (connected) {
synchronized (this) {
connected = false;
}
if (outToClient != null) {
try {
outToClient.close();
synchronized (this) {
outToClient = null;
}
} catch (IOException e) {
// there is nothing we can do: ignore it
}
}
if (inFromClient != null) {
try {
inFromClient.close();
synchronized (this) {
inFromClient = null;
}
} catch (IOException e) {
// there is nothing we can do: ignore it
}
}
if (clientSocket != null) {
try {
clientSocket.close();
synchronized (this) {
clientSocket = null;
}
} catch (IOException e) {
// there is nothing we can do: ignore it
}
}
}
}
public void returnThread() {
return;
}
}
The final class is the thread handler where I try to compare sound levels and yield all threads except the loudest one:
import java.net.ServerSocket;
import java.util.ArrayList;
import java.util.Hashtable;
import com.research.aserver.WorkerRunnable;
public class ThreadHandler implements Runnable {
protected boolean isStopped = false;
protected Thread runningThread = null;
protected Thread clientThread = null;
private Hashtable<Long, WorkerRunnable> Users = new Hashtable<Long, WorkerRunnable>();
private ArrayList<Thread> ClientThreads = new ArrayList<Thread>();
private WorkerRunnable client = null;
private int sound_max = 0;
private int index = 0;
public ThreadHandler() {
}
public void setUp(WorkerRunnable client, Thread clientThread) {
this.client = client;
this.clientThread = clientThread;
Users.put(clientThread.getId(), this.client); // Place clients in a list with its thread ID as key
ClientThreads.add(this.clientThread); // List of client threads
}
#Override
public void run() {
long endTime = System.currentTimeMillis() + 2000; // Help variable to check every 2 sec
while (!Users.isEmpty() && !ClientThreads.isEmpty()) {
for (int i = 0; i < ClientThreads.size(); i++) { // Remove clients and threads if no longer active
if (!ClientThreads.get(i).isAlive()) {
Users.remove(ClientThreads.get(i).getId());
ClientThreads.get(i).interrupt();
ClientThreads.remove(i);
}
}
if(System.currentTimeMillis() >= endTime) { // Do work every 2 sec
for (int i = 0; i < ClientThreads.size(); i++) { // Get the client with the loudest sound
if (sound_max < Users.get(ClientThreads.get(i).getId()).getSoundAverage()) {
sound_max = Users.get(ClientThreads.get(i).getId()).getSoundAverage();
index = i;
}
}
for (int i = 0; i < ClientThreads.size(); i++) { // yield all threads that are not the loudest
if (Users.get(ClientThreads.get(index).getId()) != Users.get(ClientThreads.get(i).getId())){
ClientThreads.get(i).yield();
index = 0;
}
}
endTime = System.currentTimeMillis() + 2000; // update time
}
sound_max = 0;
}
}
}
One idea might be to use a PriorityBlockingQueue and define a quality value for each input, which is then sort by quality automatically inside the list.
Using this your consumer thread can simply fetch the first one in line and process it, knowing that it is the most appropriate one, while the generator threads can simply throw all input in the Queue.
LinkedList throws exception when trying to poll data. But I think i correctly use read/write lock concept. What is wrong with that code?
package sample;
import java.util.ArrayList;
import java.util.LinkedList;
import java.util.List;
import java.util.Queue;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReadWriteLock;
import java.util.concurrent.locks.ReentrantReadWriteLock;
public class PingPong extends Thread {
boolean read = false;
Queue<String> queue;
static ReadWriteLock lock = new ReentrantReadWriteLock();
final static Lock readLock = lock.readLock();
final static Lock writeLock = lock.writeLock();
boolean stop;
public PingPong(boolean read, Queue<String> queue) {
this.read = read;
this.queue = queue;
}
int count = 0;
#Override
public String toString() {
return "PingPong{" +
"read=" + read +
", count=" + count +
'}';
}
#Override
public void run() {
if (read) {
while (!stop) {
readLock.lock();
// synchronized (queue) {
try {
String string = queue.poll();
if (string != null) {
count++;
}
} finally {
readLock.unlock();
}
// }
inform();
}
} else {
while (!stop) {
writeLock.lock();
// synchronized (queue) {
try {
if (queue.add("some str" + count)) {
count++;
}
} finally {
writeLock.unlock();
}
// }
inform();
}
}
}
private void inform() {
// Thread.yield();
// synchronized (queue) {
// queue.notify();
// try {
// queue.wait(1);
// } catch (InterruptedException e) {
// e.printStackTrace(); //To change body of catch statement use File | Settings | File Templates.
// }
// }
}
public static void main(String[] args) throws InterruptedException {
Queue<String> queue = new LinkedList();
// queue = new ArrayBlockingQueue<String>(100);
// queue = new ConcurrentLinkedQueue<String>();
List<PingPong> pongs = new ArrayList<PingPong>();
for (int i = 0; i < 10; ++i) {
PingPong pingPong = new PingPong(i % 2 == 0, queue);
pingPong.start();
pongs.add(pingPong);
}
Thread.sleep(1000);
int sum = 0;
int read = 0;
int write = 0;
for (PingPong pp : pongs) {
pp.stop = true;
pp.join();
}
for (PingPong pp : pongs) {
System.out.println(pp);
sum += pp.count;
if (pp.read) read += pp.count;
else write += pp.count;
}
System.out.println(sum);
System.out.println("write = " + write);
System.out.println("read = " + read);
System.out.println("queue.size() = " + queue.size());
System.out.println("balance (must be zero) = " + (write - read - queue.size()));
}
}
It's because this call mutates the queue collection:
String string = queue.poll();
From Queue JavaDoc:
Retrieves and removes the head of this queue, or returns null if this queue is empty.
Read locks are meant to be used in situations where multiple threads can safely read, while writes have to be performed exclusively (no other reads and writes). Because you are using read lock to poll the queue (write operation!), you are effectively allowing multiple threads to modify non thread-safe LinkedList concurrently.
Read-write lock isn't the correct synchronization mechanism in this case.
I have a program that performs lots of calculations and reports them to a file frequently. I know that frequent write operations can slow a program down a lot, so to avoid it I'd like to have a second thread dedicated to the writing operations.
Right now I'm doing it with this class I wrote (the impatient can skip to the end of the question):
public class ParallelWriter implements Runnable {
private File file;
private BlockingQueue<Item> q;
private int indentation;
public ParallelWriter( File f ){
file = f;
q = new LinkedBlockingQueue<Item>();
indentation = 0;
}
public ParallelWriter append( CharSequence str ){
try {
CharSeqItem item = new CharSeqItem();
item.content = str;
item.type = ItemType.CHARSEQ;
q.put(item);
return this;
} catch (InterruptedException ex) {
throw new RuntimeException( ex );
}
}
public ParallelWriter newLine(){
try {
Item item = new Item();
item.type = ItemType.NEWLINE;
q.put(item);
return this;
} catch (InterruptedException ex) {
throw new RuntimeException( ex );
}
}
public void setIndent(int indentation) {
try{
IndentCommand item = new IndentCommand();
item.type = ItemType.INDENT;
item.indent = indentation;
q.put(item);
} catch (InterruptedException ex) {
throw new RuntimeException( ex );
}
}
public void end(){
try {
Item item = new Item();
item.type = ItemType.POISON;
q.put(item);
} catch (InterruptedException ex) {
throw new RuntimeException( ex );
}
}
public void run() {
BufferedWriter out = null;
Item item = null;
try{
out = new BufferedWriter( new FileWriter( file ) );
while( (item = q.take()).type != ItemType.POISON ){
switch( item.type ){
case NEWLINE:
out.newLine();
for( int i = 0; i < indentation; i++ )
out.append(" ");
break;
case INDENT:
indentation = ((IndentCommand)item).indent;
break;
case CHARSEQ:
out.append( ((CharSeqItem)item).content );
}
}
} catch (InterruptedException ex){
throw new RuntimeException( ex );
} catch (IOException ex) {
throw new RuntimeException( ex );
} finally {
if( out != null ) try {
out.close();
} catch (IOException ex) {
throw new RuntimeException( ex );
}
}
}
private enum ItemType {
CHARSEQ, NEWLINE, INDENT, POISON;
}
private static class Item {
ItemType type;
}
private static class CharSeqItem extends Item {
CharSequence content;
}
private static class IndentCommand extends Item {
int indent;
}
}
And then I use it by doing:
ParallelWriter w = new ParallelWriter( myFile );
new Thread(w).start();
/// Lots of
w.append(" things ").newLine();
w.setIndent(2);
w.newLine().append(" more things ");
/// and finally
w.end();
While this works perfectly well, I'm wondering:
Is there a better way to accomplish this?
Your basic approach looks fine. I would structure the code as follows:
import java.io.BufferedWriter;
import java.io.File;
import java.io.IOException;
import java.io.Writer;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.TimeUnit;
public interface FileWriter {
FileWriter append(CharSequence seq);
FileWriter indent(int indent);
void close();
}
class AsyncFileWriter implements FileWriter, Runnable {
private final File file;
private final Writer out;
private final BlockingQueue<Item> queue = new LinkedBlockingQueue<Item>();
private volatile boolean started = false;
private volatile boolean stopped = false;
public AsyncFileWriter(File file) throws IOException {
this.file = file;
this.out = new BufferedWriter(new java.io.FileWriter(file));
}
public FileWriter append(CharSequence seq) {
if (!started) {
throw new IllegalStateException("open() call expected before append()");
}
try {
queue.put(new CharSeqItem(seq));
} catch (InterruptedException ignored) {
}
return this;
}
public FileWriter indent(int indent) {
if (!started) {
throw new IllegalStateException("open() call expected before append()");
}
try {
queue.put(new IndentItem(indent));
} catch (InterruptedException ignored) {
}
return this;
}
public void open() {
this.started = true;
new Thread(this).start();
}
public void run() {
while (!stopped) {
try {
Item item = queue.poll(100, TimeUnit.MICROSECONDS);
if (item != null) {
try {
item.write(out);
} catch (IOException logme) {
}
}
} catch (InterruptedException e) {
}
}
try {
out.close();
} catch (IOException ignore) {
}
}
public void close() {
this.stopped = true;
}
private static interface Item {
void write(Writer out) throws IOException;
}
private static class CharSeqItem implements Item {
private final CharSequence sequence;
public CharSeqItem(CharSequence sequence) {
this.sequence = sequence;
}
public void write(Writer out) throws IOException {
out.append(sequence);
}
}
private static class IndentItem implements Item {
private final int indent;
public IndentItem(int indent) {
this.indent = indent;
}
public void write(Writer out) throws IOException {
for (int i = 0; i < indent; i++) {
out.append(" ");
}
}
}
}
If you do not want to write in a separate thread (maybe in a test?), you can have an implementation of FileWriter which calls append on the Writer in the caller thread.
One good way to exchange data with a single consumer thread is to use an Exchanger.
You could use a StringBuilder or ByteBuffer as the buffer to exchange with the background thread. The latency incurred can be around 1 micro-second, doesn't involve creating any objects and which is lower using a BlockingQueue.
From the example which I think is worth repeating here.
class FillAndEmpty {
Exchanger<DataBuffer> exchanger = new Exchanger<DataBuffer>();
DataBuffer initialEmptyBuffer = ... a made-up type
DataBuffer initialFullBuffer = ...
class FillingLoop implements Runnable {
public void run() {
DataBuffer currentBuffer = initialEmptyBuffer;
try {
while (currentBuffer != null) {
addToBuffer(currentBuffer);
if (currentBuffer.isFull())
currentBuffer = exchanger.exchange(currentBuffer);
}
} catch (InterruptedException ex) { ... handle ... }
}
}
class EmptyingLoop implements Runnable {
public void run() {
DataBuffer currentBuffer = initialFullBuffer;
try {
while (currentBuffer != null) {
takeFromBuffer(currentBuffer);
if (currentBuffer.isEmpty())
currentBuffer = exchanger.exchange(currentBuffer);
}
} catch (InterruptedException ex) { ... handle ...}
}
}
void start() {
new Thread(new FillingLoop()).start();
new Thread(new EmptyingLoop()).start();
}
}
Using a LinkedBlockingQueue is a pretty good idea. Not sure I like some of the style of the code... but the principle seems sound.
I would maybe add a capacity to the LinkedBlockingQueue equal to a certain % of your total memory.. say 10,000 items.. this way if your writing is going too slow, your worker threads won't keep adding more work until the heap is blown.
I know that frequent write operations
can slow a program down a lot
Probably not as much as you think, provided you use buffering.