I made a producer-consumer program. It's just a program in core java without any GUI(Swing or SWT). It has one producer who put objects into the queue.
Also there is a few consumers who must add some staff(for example String) into Every object in that shared queue. So, every consumer must handle every object in a shared queue.
In this case - every BookShelf must have items from All consumers in "books" ArrayList. consumers.
Question: What condition should I use in consumers to finish their threads correctly?
Here are the code fragments of the program. Maybe I implemented it in wrong way.
Here is an object for the queue:
public class BookShelf {
private int id;
private String name;
private int height;
private int weigh;
List<String> books = Collections.synchronizedList(new ArrayList<String>());
public BookShelf(int id, String name) {
this.id = id;
this.name = name;
}
public void addBook(String book) {
books.add(book);
}
public boolean eq(String book) {
synchronized (books) {
for (String b: books) {
if (b.equalsIgnoreCase(book)) {
return true;
}
}
}
return false;
}
other setters and getters..
}
Here is the producer class:
public class Producer implements Runnable {
private BlockingQueue myQueue;
public Producer(BlockingQueue myQueue) {
this.myQueue = myQueue;
}
public void run() {
for(int i=0; i<7; i++){
try {
System.out.println("Produced: " + i);
BookShelf myBookShelf = new BookShelf(i, "book #" + i);
myQueue.put(myBookShelf);
} catch (InterruptedException ex) {
//Proper handle
}
}
}
}
Here is one of consumers class:
public class Consumer implements Runnable {
private BlockingQueue myQueue;
public Consumer(BlockingQueue myQueue) {
this.myQueue = myQueue; }
public void run() {
while(true){
try {
BookShelf tempBookShelf = (BookShelf) myQueue.take();
//eq() is my method to check if ArraList has a book.
if (tempBookShelf.eq("Abc book")) {
System.out.println("It already has book");
myQueue.put(tempBookShelf);
Thread.sleep(2000);
} else {
tempBookShelf.addBook("Abc book");
myQueue.put(tempBookShelf);
Thread.sleep(2000);
}
} catch (InterruptedException ex) {
//Proper handle
}
}
}
}
Here is main class:
public class ProducerConsumerTest {
public static void main(String[] args) {
BlockingQueue sharedQueue = new LinkedBlockingQueue();
Thread prodThread = new Thread(new Producer(sharedQueue));
Thread consThread = new Thread(new Consumer(sharedQueue));
Thread consThread2 = new Thread(new Consumer2(sharedQueue));
prodThread.start();
consThread.start();
consThread2.start();
}
}
Register each consumer with the producer. Each consumer has its own queue and the producer puts the object into all the queues. Each consumer then process on the same instance of the object.
public interface Consumer{
public void process(BookShelf bs);
}
public class Producer implements Runnable{
private final List<Consumer> consumers = new CopyOnWriteArrayList<Consumer>(); // thread safe but not efficient with lots of changes
public void register(Consumer c){
consumers.add(c); // thread safe
}
public void run(){
for(;;){
BookShelf bs = generateBookShelfByWhateverMeans();
for (Consumer c : consumers){
c.process(bs);
}
}
}
}
public class BookShelfConsumer implements Runnable, Consumer{
private final BlockingQueue<BookShelf> queue = new LinkedTransferQueue<BookShelf>(); // unbounded & thread safe
public void process(BookShelf bs){
queue.offer(bs); // non-blocking
}
public void run(){
for(;;){
BookShelf bs = queue.take(); // blocks until got object or interrupted
// catch InterruptedException
// do whatever this consumer is supposed to do with the object
}
}
}
I would try using SwingWorker instead. It has a done() method that is executed when it's finished. See this page for some code examples.
If it's not Swing you are using, there is a similar function in Swt called Jobs. Check this page for examples. It also has a done() method being executed when the job is done.
Also there is a few(N number) consumers who must add some staff(for example String) into Every object in that shared queue
I assume you mean every consumer must add their thing to every object which ever enters the queue. In that case, this is not a producer-consumer problem, this is more like an observer-observable problem. Basically, when a new BookShelf is created, that is the Observable. All of the Observers should be notified about the BookShelf and given the opportunity to add their own Book.
I recommend using a ConcurrentLinkedQueue in Bookshelf instead of a synchronized list - it's lock free (doesn't need to be synchronized) and will probably be more efficient.
To end your consumers, change their while(true) loops to while(!cancel) loops. Give each consumer a cancel boolean as an instance variable that initializes to false, and give them a cancel() method that sets cancel to true. Call cancel() on your consumers when you're done with them. If you will always be canceling all of your consumers at once (instead of selectively canceling some but not others), then you can use a static cancel instead of an instance cancel.
Related
I am trying to access variables from multiple running threads.
I have a main class that starts up 2 threads, a producer and a consumer.
the PRODUCER thread reads a binary file. For each line in that binary file the producer thread creates an object from it, and passes that object to the consumer thread through a blockingqueue.
the CONSUMER then takes that object passed in through a blocking queue and outputs the values of the fields in that object to a text file.
sometimes there are errors in the binary file that the producer thread is reading from.
when there are too many errors in the binary file, I want the consumer thread to change the extension of the txt file that it outputs to .err
My problem: I dont know how to modify a value from a producer thread in a consumer thread. Ive been reading that I could use a volatile field. But I dont know what is the proper way to use it between threads.
Here is a very shortened and less complex example of my code :
public class Main
{
private volatile static boolean tooManyErrors= false;
public static void main(String[] args)
{
BlockingQueue<binaryObject> queue = new LinkedBlockingQueue<>(null);
binaryObject poison = null;
new Thread(new Producer(tooManyErrors, queue, poison)).start();
new Thread(new Consumer(tooManyErrors, queue, poison)).start();
}
}
public class Producer implements Runnable
{
private final BlockingQueue<binaryObject> queue;
private final binaryObject POISON;
private boolean tooManyErrors;
private int errorsCounter = 0;
public Producer(boolean tooManyErrors, BlockingQueue<binaryObject> queue,
binaryObject POISON)
{
this.tooManyErrors = tooManyErrors;
this.queue = queue;
this.POISON = POISON;
}
#Override
public void run()
{
try
{
process();
}
catch (InterruptedException e)
{
Thread.currentThread().interrupt();
}
finally
{
while (true)
{
try
{
queue.put(POISON);
break;
}
catch (InterruptedException e)
{
//...
}
}
}
}
private void process() throws InterruptedException
{
//here is where all the logic to read the file and create
//the object goes in. counts the number of errors in the file
//if too many errors, want to change the tooManyErrors to true
if(errorsCounter > 100)
{
tooManyErrors = true;
}
}
}
public class Consumer implements Runnable
{
private final BlockingQueue<binaryObject> queue;
private final binaryObject POISON;
private boolean tooManyErrors;
//variable with extension name
private String extension;
public Consumer(boolean tooManyErrors, BlockingQueue<Integer> queue,
binaryObject POISON)
{
this.tooManyErrors = tooManyErrors;
this.queue = queue;
this.POISON = POISON;
}
#Override
public void run()
{
try
{
while (true)
{
binaryObject take = queue.take();
process(take);
// if this is a poison pill, break, exit
if (take == POISON)
{
break;
}
}
}
catch (InterruptedException e)
{
Thread.currentThread().interrupt();
}
}
private void process(Integer take) throws InterruptedException
{
//this is where all the logic goes that takes the binaryObject
//grabs all the fields from it such as binaryObject.getFileName,
//happens. It then outputs a text file with all the fields grabbed
//from the object. If the producer thread found too many errors
// I want the extension changed to .err instead of .txt
// I dont know how to do that
if(tooManyErrors == false)
{
extension = ".txt";
createFile(extension);
}
else
{
extension = ".err";
createFile(extension);
}
}
private void createFile(String extension)
{
//...
}
}
OK, a couple of things here. I assume there is only one producer and one consumer. If that is the case you can mark the classes producer and consumer as static. If you mark the class as static, only one instance of its fields will exist - it will be a singleton. You can mark any of the fields of the producer and consumer as not private, and the simply access the fields. somethingINeed = Producer.fieldThatIsNotPrivate from inside the consumer and vice versa.
Another alternative would be to keep a handle to the object you want and pass that into the constructor.
Producer p = new Producer(tooManyErrors, queue, poison);
Consumer c = new Consumer(tooManyErrors, queue, poison);
p.setConsumer(c);
c.setProducer(p);
new Thread(p).start();
new Thread(c).start();
And you could create accessors for whatever fields need to share information.
I'm implementing a program which contains different tasks and all have implemented Runnable. e.g. there is a task which works on a database and sends some of the tuples to a synchronized shared memory and subsequently, there is another thread which checks the shared memory and sends messages to a queue. Moreover, these two threads iterate over an infinite while loop.
Already, I have used the fixedThreadPool to execute these threads.
The problem is that sometimes program control remained in the first running thread and the second one never gets the chance to go to its running state.
Here is a similar sample code to mine:
public class A implements Runnable {
#Override
public void run() {
while(true) {
//do something
}
}
}
public class B implements Runnable {
#Override
public void run() {
while(true) {
//do something
}
}
}
public class Driver {
public static void main(String[] args) {
ExecutorService executorService = Executors.newFixedThreadPool(2);
A a = new A();
executorService.execute(a);
B b = new B();
executorService.execute(b);
}
}
I'd also done something tricky, make the first thread to sleep once for a second after a short period of running. As a result, it makes the second thread to find the chance for running. But is there any well-formed solution to this problem? where is the problem in your opinion?
This is a good example of Producer/Consumer pattern. There are many ways of implementing this. Here's one naive implementation using wait/notify pattern.
public class A implements Runnable {
private Queue<Integer> queue;
private int maxSize;
public A(Queue<Integer> queue, int maxSize) {
super();
this.queue = queue;
this.maxSize = maxSize;
}
#Override
public void run() {
while (true) {
synchronized (queue) {
while (queue.size() == maxSize) {
try {
System.out.println("Queue is full, " + "Producer thread waiting for "
+ "consumer to take something from queue");
queue.wait();
} catch (Exception ex) {
ex.printStackTrace();
}
}
Random random = new Random();
int i = random.nextInt();
System.out.println("Producing value : " + i);
queue.add(i);
queue.notifyAll();
}
}
}
}
public class B implements Runnable {
private Queue<Integer> queue;
public B(Queue<Integer> queue) {
super();
this.queue = queue;
}
#Override
public void run() {
while (true) {
synchronized (queue) {
while (queue.isEmpty()) {
System.out.println("Queue is empty," + "Consumer thread is waiting"
+ " for producer thread to put something in queue");
try {
queue.wait();
} catch (Exception ex) {
ex.printStackTrace();
}
}
System.out.println("Consuming value : " + queue.remove());
queue.notifyAll();
}
}
}
}
And here's hot we set things up.
public class ProducerConsumerTest {
public static void main(String[] args) {
Queue<Integer> buffer = new LinkedList<>();
int maxSize = 10;
Thread producer = new Thread(new A(buffer, maxSize));
Thread consumer = new Thread(new B(buffer));
ExecutorService executorService = Executors.newFixedThreadPool(2);
executorService.submit(producer);
executorService.submit(consumer);
}
}
In this case the Queue acts as the shared memory. You may substitute it with any other data structure that suits your needs. The trick here is that you have to coordinate between threads carefully. That's what your implementation above lacks.
I know it may sound radical, but non-framework parts of asynchonous code base should try avoiding while(true) hand-coded loops and instead model it as a (potentially self-rescheduling) callback into an executor
This allows more fair resources utilization and most importantly per-iteration monitoring instrumentation.
When the code is not latency critical (or just while prototyping) the easiest way is to do it with Executors and possibly CompletableFutures.
class Participant implements Runnable {
final Executor context;
#Override
public void run() {
final Item work = workSource.next();
if (workSource.hasNext()) {
context.execute(this::run);
}
}
}
Thread A is a loop that performs calculations.
Thread B needs to read result produced by each iteration of loop enclosed in Thread A.
What would be the best approach to achieve this without blocking anything?
You need the Producer-Consumer pattern here. In Java, you can use BlockingQueues to implement it. Here's an example with an ArrayBlockingQueue that is used to deliver Double calculation results from a producer to a consumer:
Producer.java
class Producer implements Runnable {
private final BlockingQueue<Double> queue;
public Producer(BlockingQueue<Double> q) {
queue = q;
}
public void run() {
try {
while (true) {
Double result = calculateResult();
// This call will make the result available to the consumer:
queue.put(result);
}
} catch (InterruptedException ex) {
// Handle thread interruption here
}
}
}
Consumer.java
class Consumer implements Runnable {
private final BlockingQueue<Double> queue;
public Consumer(BlockingQueue<Double> q) {
queue = q;
}
public void run() {
try {
while (true) {
// This call Will wait until the next result is available:
Double result = queue.take();
// Process the result...
}
} catch (InterruptedException ex) {
// Handle thread interruption here
}
}
}
Program.java
class Program {
public static void main() {
BlockingQueue<Double> queue = new ArrayBlockingQueue<>();
Producer producer = new Producer(queue);
Consumer consumer = new Consumer(queue);
new Thread(producer).start();
new Thread(consumer).start();
}
}
If you need, you can create several consumers or several producers (or both) that share the same queue. That will allow you to balance the work between more than two threads.
Also have a look at the BlockingQueue's capabilities and on other implementations, there're plenty of them.
My main spawns 2 threads and they both need to access the same list. I am not sure what the best way to do this is. Here is what I have, but I still run into concurrentModificationException.
class Parent {
private List<String> data;
public List<String> getData() {
return data;
}
public static void main(String args[]) {
Parent p = new Parent();
p.start();
}
public void start() {
Thread a = new Thread(new A(this)).start();
Thread b = new Thread(new B(this)).start();
}
public A implements Runnable {
private Parent parent;
public A(Parent p) {
parent = p;
}
public void run() {
while (true) {
parent.getData().add("data");
}
}
}
public B implements Runnable {
private Parent parent;
public B(Parent p) {
parent = p;
}
public void run() {
Iterator<String> i = parent.getData().iterator();
while(i.hasNext()) {
// do more stuff with i
i.remove();
}
}
}
}
my A class is basically a producer of data and B is the consumer. I accept the possibility that I am going about this the wrong way. So all help is welcome. I just need to safely be able to add to a list from one thread and remove an item from the list from another thread. thanks in advance.
Well, for a producer/consumer, I'd recommend LinkedBlockingQueue or ConcurrentLinkedQueue. This will handle concurrent reads and writes (or pushes/polls in this case).
You'll probably want your consumer to run until some shutdown condition is sent to it. If you were using a blocking queue, this means you will want to send a queued item that indicates that the consumer should stop consuming. This would be a blocking queue implementation with a shutdown.
public enum QueueItemType {
CONSUMABLE,
SHUTDOWN
}
public class QueueItem {
public final QueueItemType type;
public final String payload;
public QueueItem(QueueItemType type, String payload) {
this.type = type;
this.payload = payload;
}
}
public class B implements Runnable {
private Parent parent;
public B(Parent p) {
parent = p;
}
public void run() {
while(true) {
QueueItem data = parent.getData().poll();
if (data.type == QueueItemType.SHUTDOWN) {
break;
} else {
// do more stuff with data.payload
}
}
}
}
Note that there is no null check for a blocking queue's poll result. This is because, by definition, blocking queues block the running thread until something is there.
If you prefer to have a consumer that does not contend with the producer, then you will need to poll periodically and sleep the consumer thread. Here's an example if you used the ConcurrentLinkedQueue:
public class B implements Runnable {
private Parent parent;
public B(Parent p) {
parent = p;
}
public void run() {
while(parent.isStillRunning()) {
String data = parent.getData().poll();
if (data != null) {
// do more stuff with data
} else {
Thread.sleep(10 /*10 ms, but you can make this whatever poll interval you want*/);
}
}
}
}
The least impactful change, could be to use a setter that is synchronized.
That way a thread would have to wait on the lock to be released before being able to add to the Collection.
I am new to using threads. In another class an instance of the ConnectionMaster class is created and started (extends thread). A Client object is given to the ConnectionMaster object which adds it to the list. The overridden run() method of the Thread class essentially listens for a client to be added to the list. Indeed it does listen and "hears" when a Client object is added to the list. However, although .hasNext() returns true .Next() causes an exception. What am I doing wrong?
The following methods are from class ConnectionMaster which extends Thread:
Constructor
public ConnectionMaster(){
clients = new Vector<>();
listIterator = clients.listIterator();
}
Public method for adding client objects to the list
#Override
public synchronized void addClient(Client client) {
listIterator.add(client);
}
This is the overridden thread method of the class Thread. It consistently checks for elements added to the list.
#Override
public void run(){
while(true){
while(listIterator.hasNext()){
processClient(listIterator.next()); //this is where error occurs
listIterator.remove();
}
while(listIterator.hasPrevious()){
processClient(listIterator.previous());
listIterator.remove();
}
}
}
////////////////////////////////UPDATE////////////////////////////////////
Thank You OldCurmudgeon and Stephen C.
Based on your feedback, my code has been modified thus:
Constructor
public ConnectionMaster(){
clients = new ArrayBlockingQueue<Client>(1024);
}
Method for receiving client objects
#Override
public synchronized void addClient(Client client) {
try {
clients.put(client);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
Listener
#Override
public void run(){
while(true){
try {
processClient((Client)clients.take());
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
This is a very strange way to implement Producer/Consumer. The usual way is to use a BlockingQueue.
public class TwoThreads {
public static void main(String args[]) throws InterruptedException {
System.out.println("TwoThreads:Test");
new TwoThreads().test();
}
// The end of the list.
private static final Integer End = -1;
static class Producer implements Runnable {
final BlockingQueue<Integer> queue;
public Producer(BlockingQueue<Integer> queue) {
this.queue = queue;
}
#Override
public void run() {
try {
for (int i = 0; i < 1000; i++) {
queue.add(i);
Thread.sleep(1);
}
// Finish the queue.
queue.add(End);
} catch (InterruptedException ex) {
// Just exit.
}
}
}
static class Consumer implements Runnable {
final BlockingQueue<Integer> queue;
public Consumer(BlockingQueue<Integer> queue) {
this.queue = queue;
}
#Override
public void run() {
boolean ended = false;
while (!ended) {
try {
Integer i = queue.take();
ended = i == End;
System.out.println(i);
} catch (InterruptedException ex) {
ended = true;
}
}
}
}
public void test() throws InterruptedException {
BlockingQueue<Integer> queue = new LinkedBlockingQueue<>();
Thread pt = new Thread(new Producer(queue));
Thread ct = new Thread(new Consumer(queue));
// Start it all going.
pt.start();
ct.start();
// Wait for it to finish.
pt.join();
ct.join();
}
}
What am I doing wrong?
Quite a lot actually.
First thing you are doing wrong is (apparently) using an ListIterator object in multiple threads. The ListIterator and Iterator implementations for Vector are not thread-safe1, so what you are doing is potentially hazardous.
The second thing is that even if the iterators / list iterators were thread-safe, you are performing a sequence of operations (e.g. hasNext, next, remove) without doing anything to ensure that the sequence of operations is performed in a way that is threadsafe. There is a distinct possibility that two threads could be performing the same sequence simultaneously on the shared iterator, and that one could interfere with the other.
I'm not sure what to suggest in order to fix your code. Two threads sharing an iterator is not going to work.
It would probably be better to ditch it, and use some kind of Queue as suggested by #OldCurmugeon.
Either problem 1 or problem 2 (as outlined above) could give rise to NoSuchElement exceptions.
1 - This is apparent from examining the source code - http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8u40-b25/java/util/Vector.java#Vector.ListItr .