We Keep Coding

Can my consumer become producer (apparently) for few cases

I am solving transaction polling case using producer-consumer problem using ExecutorService and BlockingQueue. I have a list of transactions, which I want to verify and take action. I am getting a new transaction to verify continuously.
Considering BlockingQueue, I have only one producer and I want to keep 3-5 parallel consumers to speed up the verification.
I might have to wait for a few transactions to get completed. (Say 30 secs).
So, I will verify and if it is false and time is greater than 30 sec, I will drop it. Basically, I want to consume only when the data item is consumable.
Firstly, is this approach good? Or I should try some other solutions (which I am not aware of as of now)
Here is the code that I have adapted from this question:
import java.util.concurrent.*;
public class ProducerConsumerWithES {
public static void main(String args[]){
BlockingQueue<Integer> sharedQueue = new LinkedBlockingQueue<Integer>();
ExecutorService pes = Executors.newFixedThreadPool(2);
ExecutorService ces = Executors.newFixedThreadPool(2);
pes.submit(new Producer(sharedQueue,1));
pes.submit(new Producer(sharedQueue,2));
ces.submit(new Consumer(sharedQueue,1));
ces.submit(new Consumer(sharedQueue,2));
// shutdown should happen somewhere along with awaitTermination
/* https://stackoverflow.com/questions/36644043/how-to-properly-shutdown-java-executorservice/36644320#36644320 */
pes.shutdown();
ces.shutdown();
}
}
class Producer implements Runnable {
private final BlockingQueue<Integer> sharedQueue;
private int threadNo;
public Producer(BlockingQueue<Integer> sharedQueue,int threadNo) {
this.threadNo = threadNo;
this.sharedQueue = sharedQueue;
}
#Override
public void run() {
for(int i=1; i<= 5; i++){
try {
int number = i+(10*threadNo);
System.out.println("Produced:" + number + ":by thread:"+ threadNo);
sharedQueue.put(number);
} catch (Exception err) {
err.printStackTrace();
}
}
}
}
class Consumer implements Runnable{
private final BlockingQueue<Integer> sharedQueue;
private int threadNo;
public Consumer (BlockingQueue<Integer> sharedQueue,int threadNo) {
this.sharedQueue = sharedQueue;
this.threadNo = threadNo;
}
#Override
public void run() {
while(true){
try {
int num = sharedQueue.take();
System.out.println("Consumed: "+ num + ":by thread:"+threadNo);
} catch (Exception err) {
err.printStackTrace();
}
}
}
}
I know I can do a peek() and then remove() if it is required.
But when I tried doing that, all other consumers get stuck to the same transaction. And the other transactions getting produced are never attended.
This is because of the storage being queue (FIFO).
This scenario never happens when I remove the element, do the verification instead of peeking because other consumers get access to the remaining elements.
My question is, Is doing a peek() followed by remove() or put() at the consumer side is okay?

Multi-threading with random thread

I try to realize Producer-Consumer pattern with several producers and consumers.
I try to make
CompletableFuture future = CompletableFuture.runAsync(() -> producer.run(), producerService)
.thenRunAsync(() -> consumer.run(), consumerService);
where producer.run() do something and return String but it is not necessary and consumer.run() do something like this
while (!queue.isEmpty()) {
try {
message = queue.poll();
if (message == null || !message.equals(thread)) {
queue.offer(message);
Thread.sleep(1000);
continue;
}
doWork(message);
} catch (InterruptedException e) {
e.printStackTrace();
My Thread has name equals the number of that like 1 or 2, or 3 if there are 3 Threads in consumerService.
message is a random number which I get with
String.valueOf(1 + new Random().nextInt(2)) for 2 Threads as I suppose.
So, my question is
What should I do instead of thenRunAsync() or something else that my consumer can be possible to change Thread to take message from the queue?
It needs to producer generates a list of numbers like 1,2,1,1,2,1,1,1 and Consumer with Thread which has name 1 get from the queue messages with number equals 1 but Thread with name 2 get with number equals 2.
I can't do every message and after that do CompletableFuture.allOf() because, if I'd have about 1_000_000 tasks, I'd have to wait while it generates and after that, I'd be able to call my consumers

CompletableFuture.run* methods are used to run multiple short-living tasks using a thread pool. Your tasks are not short-living, they are looping over queue and handle multiple values. As a result, they occupy threads from the thread pool, and the size of the thread pool decreases, which may lead to a thread starvation (a kind of dead lock).
So you should not use CompletableFuture.run* methods. Use explicit thread creation instead.
Then, make sure that producer puts messages into queue with queue.put() or queue.offer(), and consumer pulls messages with queue.get() or queue.poll(). In your code, consumer both puts and pulls messages, and producer does not interact with the queue at all.

I realized it like
class Stater {
public static boolean STOP = false;
private Producer producer;
private Consumer consumer;
private ExecutorService producerService= Executors.newFixedThreadPool(PRODUCER_NUMBER, taxiFactory);
private ExecutorService consumerService= Executors.newFixedThreadPool(CONSUMER_NUMBER, clientFactory);
private void working() {
for (int i = 0; i < PRODUCER_NUMBER; i++) {
producerService.execute(() -> producer.get());
consumerService.execute(() -> consumer.run());
}
Starter.STOP = true;
producerService.shutdown();
consumerService.shutdown();
}
}
class Common {
private Queue<Message> emergencyQueue;
private BlockingQueue<Message> blockingQueue;
public void insertOrder(Message message) {
if (!blockingQueue.offer(message)) {
emergencyQueue.add(message);
}
}
public Message getOrder() {
if (emergencyQueue.isEmpty()) {
if (!blockingQueue.isEmpty()) {
return blockingQueue.poll();
} else {
return null;
}
} else {
return emergencyQueue.poll();
}
}
public boolean shouldStop() {
return blockingQueue.isEmpty() && emergencyQueue.isEmpty() && Starter.STOP;
}
}
class Consumer implements Runnable{
private Common common;
public void run(){
common.insertOrder(new Message());
}
}
class Producer implements Runnable{
private Common common;
public void run(){
while (!common.shouldStop()) {
Message message=common.getOrder();
if (message == null) {
Thread.sleep(new Random().nextInt(TIME_TO_WAIT));
}
}
}
}

How to start a remote thread in java?

I have an apllication where there are three folders. I am trying to implement the producer consumer concept using LinkedBlockingQueue.
Folder 1:
Contains a class which has a shared queue
public static BlockingQueue sharedQueue = new LinkedBlockingQueue();
From a method in this class I try to call the Producer thread and the Consumer thread both of which reside in separate files.
Thread updateThread = new Thread(new Producer(sharedQueue));
Thread takeThread = new Thread(new Consumer(sharedQueue));
updateThread.start();
takeThread.start();
Folder 2:Contains the producer thread class as follows:
public class Producer implements Runnable {
private final BlockingQueue Queue;
public Producer(BlockingQueue sharedQueue){
Queue = sharedQueue;
}
public void run()
{
while (Thread.currentThread() != null) {
Random random = new Random();
int pos = random.nextInt(productList.size());
String query = "insert into tab1 values("+pos+")";
Queue.put(query);
}
}
Folder 3: Contains the consumer class as follows:
public class Consumer implements Runnable{
private final BlockingQueue queue;
Collection<String> joblist;
public Consumer (BlockingQueue sharedQueue) {
queue = sharedQueue;
MonitoringForm.txtInforamtion.append("hi"+sharedQueue.size());
joblist = new ArrayList<String>();
}
#Override
public void run() {
while(true){
try {
for(int i = 0; i < queue.size(); i++)
{
joblist.add(queue.take().toString());
MonitoringForm.txtInforamtion.append("What we got "+queue.take().toString());
}
} catch (InterruptedException ex) {
Logger.getLogger(Consumer.class.getName()).log(Level.SEVERE, null, ex);
}
try {
Thread.sleep(60*1000);
} catch (Exception e) {
}
}
}
}
Can somebody tell me how to make sure that the data added to the shared queue by Producer class can be taken by Consumer class. The consumer consumes the data every minute. Not necessary that data is consumed as soon as it is produced. Calling the Consumer and Producer threads is not working as I have them in separate directories. Do I have to use remote method invocation? Is it possible for threads?

Here's a sample code I wrote which can help you understand the concept:
import java.util.Random;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
/**
* We want a Producer thread to create random values, and the Consumer thread to
* consume it. One caveat is that if the Producer has already created a random
* value, which the Consumer thread hasn't consumed yet, the Producer thread
* blocks or waits. On the flip side, the Consumer thread waits for the Producer
* thread to produce some value if the Producer thread hasn't already.
*
* Write a program to simulate such a situation.
*/
public class ProducerConsumerCommunication
{
private volatile boolean running = true;
private ArrayBlockingQueue<Integer> buffer = new ArrayBlockingQueue<>(1);
private Random random = new Random(System.currentTimeMillis());
private class ProducerTask implements Runnable
{
public void run()
{
while (running)
{
try
{
Thread.sleep(random.nextInt(2000));
Integer value = random.nextInt();
buffer.put(value); // Blocks if buffer is full.
System.out.println("Value Put: " + value);
}
catch (InterruptedException e)
{
e.printStackTrace();
}
}
}
}
private class ConsumerTask implements Runnable
{
public void run()
{
while (running)
{
try
{
Thread.sleep(random.nextInt(2000));
Integer value = buffer.take(); // Blocks if buffer is empty.
System.out.println("Value Taken: " + value);
}
catch (InterruptedException e)
{
e.printStackTrace();
}
}
}
}
public ProducerConsumerCommunication()
{
ExecutorService service = Executors.newCachedThreadPool();
service.execute(new ProducerTask());
service.execute(new ConsumerTask());
service.shutdown();
}
public static void main(String[] args)
{
new ProducerConsumerCommunication();
}
}

In the traditional Consumer/Producer concept the Consumer waits on a resource. And whenever the Producer pushes anything on the Queue it notifies the Consumer via notify() / notifyAll()
Producer :
queue.put(query) ;
obj.notifyAll();
Consumer :
while(true)
{
try {
obj.wait();
}catch (InterruptedException e) {
}
// get data from Queue
data = queue.take();
}
Refer to the following link for more information : example

To run the Producer and Consumer in the fashion like when the Producer produces something in the queue he should Notify the Consumer, and when the Consumer consumed from the queue he should notify to the Producer to produce something in the queue,
To implement your problem in that way you have to use
wait and Notify method if you have one producer and one consumer.
and if you have multiple consumer then you have to use NotifyAll method as well of Object Class,
And in your Consumer if you get this line printed to your console "What we got" then you are sure that Consumer has consumed something from the queue

BlockingQueue: how can multiple producers stop a single consumer?

I wrote a producer/consumer based program using Java's BlockingQueue. I'm trying to find a way to stop the consumer if all producers are done. There are multiple producers, but only one consumer.
I found several solutions for the "one producer, many consumers" scenario, e.g. using a "done paket / poison pill" (see this discussion), but my scenario is just the opposite.
Are there any best practice solutions?

The best-practice system is to use a count-down latch. Whether this works for you is more interesting.....
Perhaps each producer should register and deregister with the consumer, and when all producers are deregistered (and the queue is empty) then the consumer can terminate too.

Presumably your producers are working in different threads in the same VM, and that they exit when done. I would make another thread that calls join() on all the producers in a loop, and when it exist that loop (because all the producer threads have ended) it then notifies the consumer that it's time to exit. This has to run in another thread because the join() calls will block. Incidentally, rolfl's suggestion of using a count down latch would have the problem, if I understand it correctly.
Alternately, if the producers are Callables, then the consumer can call isDone() and isCanceled() on their Futures in the loop, which won't bock, so it can be used right in the consumer thread.

You could use something like the following, i use registerProducer() and unregisterProducer() for keeping track of the producers. Another possible solution could make use of WeakReferences.
It's worth to mention that this solution will not consume the events that have already been queued when the consumer is shut down, so some events may be lost when shutting down.
You would have to drain the queue if the consumer gets interrupt and then process them.
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicInteger;
public class TestConsumerShutdown {
private static interface SomeEvent {
String getName();
}
private static class Consumer implements Runnable {
private final BlockingQueue<SomeEvent> queue = new ArrayBlockingQueue<>(10);
private final ExecutorService consumerExecutor = Executors.newSingleThreadExecutor();
private final AtomicBoolean isRunning = new AtomicBoolean();
private final AtomicInteger numberProducers = new AtomicInteger(0);
public void startConsumer() {
consumerExecutor.execute(this);
}
public void stopConsumer() {
consumerExecutor.shutdownNow();
try {
consumerExecutor.awaitTermination(Long.MAX_VALUE, TimeUnit.SECONDS);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
public void registerProducer() {
numberProducers.incrementAndGet();
}
public void unregisterProducer() {
if (numberProducers.decrementAndGet() < 1) {
stopConsumer();
}
}
public void produceEvent(SomeEvent event) throws InterruptedException {
queue.put(event);
}
#Override
public void run() {
if (isRunning.compareAndSet(false, true)) {
try {
while (!Thread.currentThread().isInterrupted()) {
SomeEvent event = queue.take();
System.out.println(event.getName());
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
} finally {
System.out.println("Consumer stopped.");
isRunning.set(false);
}
}
}
}
public static void main(String[] args) {
final Consumer consumer = new Consumer();
consumer.startConsumer();
final Runnable producerRunnable = new Runnable() {
#Override
public void run() {
final String name = Thread.currentThread().getName();
consumer.registerProducer();
try {
for (int i = 0; i < 10; i++) {
consumer.produceEvent(new SomeEvent() {
#Override
public String getName() {
return name;
}
});
}
System.out.println("Produver " + name + " stopped.");
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
} finally {
consumer.unregisterProducer();
}
}
};
List<Thread> producers = new ArrayList<>();
producers.add(new Thread(producerRunnable, "producer-1"));
producers.add(new Thread(producerRunnable, "producer-2"));
producers.add(new Thread(producerRunnable, "producer-3"));
for (Thread t : producers) {
t.start();
}
}
}

Java Multi thread messaging

I have an app with two threads, 1 that writes to a queue and the second one that read async from it.
I need to create a third one that generate 20 more.
the newly created threads will run till explicitly stopped. those 20 threads should get "live" data in order to analyze it.
each of the 20 has a unique ID/name. I need to send the relevant data (that the READ thread collect) to the correct thread (of the 20 threads). e.g. if the data include a string with id (in it) of 2 --> I need to send it to thread with the ID =2.
my question is: how should I hold a "pointer" to each of the 20 threads and send it the relevant data? (I can search the id in a runnable list (that will hold the threads)--> but then I need to call to a method "NewData(string)" in order to send the data to the running thread).
How should I do it?
TIA
Paz

You would probably be better to use a Queue to communicate with your threads. You could then put all of the queues in a map for easy access. I would recommend a BlockingQueue.
public class Test {
// Special stop message to tell the worker to stop.
public static final Message Stop = new Message("Stop!");
static class Message {
final String msg;
// A message to a worker.
public Message(String msg) {
this.msg = msg;
}
public String toString() {
return msg;
}
}
class Worker implements Runnable {
private volatile boolean stop = false;
private final BlockingQueue<Message> workQueue;
public Worker(BlockingQueue<Message> workQueue) {
this.workQueue = workQueue;
}
#Override
public void run() {
while (!stop) {
try {
Message msg = workQueue.poll(10, TimeUnit.SECONDS);
// Handle the message ...
System.out.println("Worker " + Thread.currentThread().getName() + " got message " + msg);
// Is it my special stop message.
if (msg == Stop) {
stop = true;
}
} catch (InterruptedException ex) {
// Just stop on interrupt.
stop = true;
}
}
}
}
Map<Integer, BlockingQueue<Message>> queues = new HashMap<>();
public void test() throws InterruptedException {
// Keep track of my threads.
List<Thread> threads = new ArrayList<>();
for (int i = 0; i < 20; i++) {
// Make the queue for it.
BlockingQueue<Message> queue = new ArrayBlockingQueue(10);
// Build its thread, handing it the queue to use.
Thread thread = new Thread(new Worker(queue), "Worker-" + i);
threads.add(thread);
// Store the queue in the map.
queues.put(i, queue);
// Start the process.
thread.start();
}
// Test one.
queues.get(5).put(new Message("Hello"));
// Close down.
for (BlockingQueue<Message> q : queues.values()) {
// Stop each queue.
q.put(Stop);
}
// Join all threads to wait for them to finish.
for (Thread t : threads) {
t.join();
}
}
public static void main(String args[]) {
try {
new Test().test();
} catch (Throwable t) {
t.printStackTrace(System.err);
}
}
}