Faced with the problem when the producer and consumer pattern has multiple consumers; i have an intention of creating single producer that pass data to consumers (as Workers) that do some staff with data-object. But the problem is that I have no idea how to pass data to producer;
Lets say that we have main function that gets data from somewhere:
public function Foo(){
dataobject = new DataObject();
Sting data = dataobject.get();
}
Then pass this data to Queue and process it and the final function should be like this:
public function Foo(){
QueueService queue = new QueueService();
dataObject = new DataObject();
Sting data = dataobject.get();
queue.send(data);
}
And there is my case:
Producer -> Queue <- Consumer_1, Consumer_n;
Instead of using Blocking Queue I went with ThreadPool and stuck with the problem that I don't have any idea how to pass data to consumer;
public static void main(String[] args) throws ExecutionException, InterruptedException {
ExecutorService service = null;
String threadName = Thread.currentThread().getName();
try {
service = Executors.newFixedThreadPool(6); // +1 thread for producer
service.submit(new Producer(service)).get(); // Wait until producer exits
} finally {
if (null != service) {
service.shutdown();
try {
service.awaitTermination(1, TimeUnit.HOURS);
} catch (InterruptedException e) {
//handle
}
}
}
System.out.println("Exit");
}
Thats okay, we spawn several threads and started to work with them. Also there is no problem even with Worker, here is the implementation:
class Worker implements Runnable {
private String message;
public Worker(String message) {
this.message = message;
}
#Override
public void run() {
String name = Thread.currentThread().getName();
ThreadLocalRandom random = ThreadLocalRandom.current();
try {
//Do staff
} catch (InterruptedException e) {
//handle
}
}
}
And finally - Producer
class Producer implements Runnable {
private ExecutorService service;
Producer(ExecutorService service) {
this.service = service;
}
#Override
public void run() {
String threadName = Thread.currentThread().getName();
try {
service.submit(new Worker(input));
} catch (IOException e) {
//handle
}
System.out.printf("[%s] Producer shutdown", threadName);
}
}
As i mentioned earlier I don't have an idea how to pass data to Producers and also I see two possible problems for me:
I don't quite understand the conception of Threads in Java;
This implementation is wrong architectural solution;
Related
I have to make three producers and one consumer thread. Producers threads is reading chars from file and using buffer of one char. I made Store like above, now i have to synchronize threads to consumer write whole word and give back control to other producer - patern like this
Producer1->Word1
Producer2->word1
Producer3->word1
Producer1->Word2
Producer2->word2
Producer3->word2
Producer1->Word3
Producer2->word3
Producer3->word3
Store (edited):
public class Store {
public static final char CONSUMER_FREE = '\0';
private volatile char consumer = CONSUMER_FREE;
private final ThreadPoolExecutor executor;
public Store(ThreadPoolExecutor executor) {
this.executor = executor;
}
public synchronized void produce(char c) {
while (isConsumerBussy()) {
try {
wait();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
consumer = c;
notify(); // single Consumer
}
public synchronized char consume() throws StoreProducersRip {
while (isConsumerFree()) {
try {
wait(250);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
try {
return consumer;
} finally {
freeConsumer();
notifyAll();
}
}
private void freeConsumer() {
consumer = CONSUMER_FREE;
}
private boolean isConsumerBussy() {
return consumer != CONSUMER_FREE;
}
private boolean isConsumerFree() throws StoreProducersRip {
if (executor.getActiveCount() == 0) throw new StoreProducersRip();
return consumer == CONSUMER_FREE;
}
}
You can use locks for this purpose. Below code demonstrates 3 producer synchronous and 1 consumer async to producers but it'd also be sync to multiple consumers here. Also you should add some logic to consume same amount that produced. If you want to get benefits of multithreading, you should buffer your producer input and create some pipeline before consuming.
public class Executor {
public static void main(String[] args) {
ThreadPoolExecutor executor = (ThreadPoolExecutor) Executors.newCachedThreadPool();
Store store = new Store(executor);
Runnable word1 = new Runnable() {
#Override
public void run() {
store.producerLock.lock();
store.produce('1');
store.produce('2');
store.produce('3');
store.producerLock.unlock();
}
};
Runnable consume = new Runnable() {
#Override
public void run() {
store.consumerLock.lock();
System.out.print(store.consume());
System.out.print(store.consume());
System.out.print(store.consume());
System.out.println();
store.consumerLock.unlock();
}
};
Future p1 = executor.submit(word1);
Future p2 = executor.submit(word1);
Future p3 = executor.submit(word1);
executor.submit(consume);
executor.submit(consume);
executor.submit(consume);
executor.shutdown();
}
}
Faced the fact that when the database is unavailable, the queue grows because tasks stop running. What is the best way to set some timeout for tasks executed in method run()? May be there is some good approach with using ExecutorService?
#Service
public class AsyncWriter implements Writer, Runnable {
private LinkedBlockingQueue<Entry> queue = new LinkedBlockingQueue<>();
private volatile boolean terminate = false;
private AtomicInteger completedCounter = new AtomicInteger();
#PostConstruct
private void runAsyncWriter() {
Thread async = new Thread(this);
async.setName("Writer Thread");
async.setPriority(2);
async.start();
}
#Override
public void run() {
while (!terminate) {
try {
Entry entry = queue.take();
dao.save(entry);
completedCounter.incrementAndGet();
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
#Override
public void write(Entry entry) {
queue.add(entry);
}
}
Maybe you can try RxJava
https://www.baeldung.com/rx-java
And you can set your aync funtions
Timeout in RxJava
I must design a simple server which receives messages from multiple nodes and stores them in a message repository.
The code for the server is :
public class CommunicationServer implements Runnable {
private List<String> messages;
private MessageRepository messageRepository;
private boolean serverBusy;
public CommunicationServer() {
messages = new ArrayList<String>();
messageRepository = new MessageRepository();
serverBusy = false;
}
#Override
public void run() {
try {
while (!Thread.interrupted()) {
synchronized (this) {
if (messages.size() > 10) {
serverBusy = true;
addMessageToRepository();
notifyAll();
}
else {
serverBusy = false;
wait();
}
}
}
}
catch (InterruptedException e) {
System.out.println(e.getMessage());
}
}
public synchronized void receiveMessage(String message) {
if (messages.size() < 10) {
messages.add(message);
}
}
private void addMessageToRepository() {
if (messages.size() != 0) {
messageRepository.addMessage(messages.remove(0));
}
}
public void showMessageRepository() {
messageRepository.showStoredMessages();
}
public synchronized boolean isServerBusy() {
return serverBusy;
}
}
The code for the node is:
public class Node implements Runnable {
private static final String CHARACTERS = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVXYZ0123456789";
private static final int MESSAGE_LENGHT = 5;
private Random random = new Random();
private CommunicationServer communicationServer;
public Node(CommunicationServer communicationServer) {
this.communicationServer = communicationServer;
}
#Override
public void run() {
try {
while (!Thread.interrupted()) {
while (communicationServer.isServerBusy()) {
wait();
}
communicationServer.receiveMessage(generateRandomString());
}
}
catch (InterruptedException e) {
System.out.println(e.getMessage());
}
}
private String generateRandomString() {
StringBuffer randomMessage = new StringBuffer();
for (int i = 0; i < MESSAGE_LENGHT; i++) {
randomMessage.append(CHARACTERS.charAt(random.nextInt(51)));
}
return randomMessage.toString();
}
}
In main I just create a thread for the server and 5 threads for the nodes and let them run for some time. The server sleeps until it receives 10 messages, after that it must wake up to process the messages. The problem is I can't figure it out where to call notifyAll() in order to wake the thread responsible for the server.
Making CommunicationServer implements Runnable doesn't really make sense and exposes your basic misunderstanding: you identify the actors in your process (server, client) with threads. A thread is not the actor; a thread is where actor's code gets executed.
So, when in your CommunicationServer you say wait(), you don't make the server wait for messages; you make that particular thread wait on the server object as its monitor. Likewise, then you say notifyAll(), you are not "notifying all servers"; you are notifying all threads waiting on that particular monitor. It should be some code in the client which notifies the threads that are currently waiting on the server's monitor, and some code in the server which notifies those waiting on the client monitor.
As a general rule of thumb, when you find yourself using both wait() and notify() within the same synchronized block, you can be pretty sure there's something wrong with your logic.
This is the first time I've ever created a multi-threaded application in Java that that will run continuously until canceled and I'm having trouble shutting down/interrupting my threads.
I have some threads that communicate with a Mediator which encapsulates a TransferQueue, an ExecutorService, and facilitates communication between producing and consuming threads.
I'm using this Mediator instead of Future because TransferQueue is a lot more block-friendly when it comes to a single consumer handling multiple producers (producer threads can mediator.put(E e) any time they want, and consumer threads can just wait on E e = mediator.take() for something to be available) and I do not want to waste CPU cycles polling.
The design is very clean, fast, and effective, but I'm having trouble interrupting blocking on queue.take(), serverSocket.accept(), and interrupting the threads as a whole.
The producers:
public class SocketProducer implements Colleague<Socket> {
private Mediator<Socket> mediator;
private ServerSocket serverSocket;
private Integer listeningPort;
private volatile boolean runnable = true;
public SocketProducer(Mediator<Socket> mediator) {
this.mediator = mediator;
}
public Colleague<Socket> setListeningPort(Integer listeningPort) {
this.listeningPort = listeningPort;
return this;
}
public Void call() throws Exception {
serverSocket = new ServerSocket(listeningPort, 10);
while (runnable) {
Socket socket = serverSocket.accept(); // blocks until connection
mediator.putIntoQueue(socket);
}
return null;
}
public void interrupt() {
// ?
runnable = false;
serverSocket.close();
// ?
}
}
and the consumer:
private class SocketConsumer implements Colleague<Socket> {
private Mediator<Socket> mediator;
private volatile boolean runnable = true;
public SomeConsumer(Mediator<Socket> mediator) {
this.mediator = mediator;
}
public Void call() throws Exception {
while (runnable) {
Socket socket = mediator.takeFromQueue(); // blocks until element is in queue
}
return null;
}
public void interrupt() {
// ?
runnable = false;
// ?
}
}
The Colleague interface just extends Callable, to give some additional capability to the Mediator in managing its producer/consumer colleagues (ie: calling for:each colleague.interrupt()).
I've tried a lot of methods, throwing InterruptedException in various places, catching InterruptedException in various places, letting threads return an instance of their Thread to the mediator for interruption. Everything I've tried has been so ineffective that it feels like I'm missing some crucial piece to this puzzle.
So far the most effective method I've seen is the poison pill (which would be great if the queues didn't throw NPE on a null insertion), and all the methods I've tried of introducing a poison generic have failed because of ClassCastException (trying to cast Object to Socket, trying to instantiate a generic Socket, etc.).
I'm really not sure where to go from here. I'd really like to be able to cleanly terminate these threads on demand.
Completed solutions:
public class SocketProducer implements Colleague<Socket> {
private static final Logger logger = LogManager.getLogger(SocketProducer.class.getName());
private Mediator<Socket> mediator;
private ServerSocket serverSocket;
private Integer listeningPort;
private volatile boolean runnable = true;
public SocketProducer(Mediator<Socket> mediator) {
this.mediator = mediator;
}
public Colleague<Socket> setListeningPort(Integer listeningPort) {
this.listeningPort = listeningPort;
return this;
}
public Void call() throws Exception {
serverSocket = new ServerSocket(listeningPort, 10);
logger.info("Listening on port " + listeningPort);
while (runnable) {
try {
Socket socket = serverSocket.accept();
logger.info("Connected on port " + socket.getLocalPort());
mediator.putIntoQueue(socket);
} catch (SocketException e) {
logger.info("Stopped listening on port " + listeningPort);
}
}
return null;
}
public void interrupt() {
try {
runnable = false;
serverSocket.close();
} catch (IOException e) {
logger.error(e);
}
}
}
public class SocketConsumer implements Colleague<Socket> {
private static final Logger logger = getLogger(SocketConsumer.class.getName());
private Mediator<Socket> socketMediator;
public SocketConsumer(Mediator<Socket> mediator) {
this.socketMediator = mediator;
}
public Void call() throws Exception {
while (!Thread.currentThread().isInterrupted()) {
try {
Socket socket = socketMediator.takeFromQueue();
logger.info("Received socket on port: " + socket.getLocalPort());
} catch (InterruptedException e) {
logger.info("Interrupted.");
Thread.currentThread().interrupt();
}
}
return null;
}
public void interrupt() {
Thread.currentThread().interrupt();
}
}
I think poison pills will only make things more complicated, so I'd keep it simple.
As for the ServerSocket, this answer suggests that calling close() should be enough to interrupt it.
As for BlockingQueue, consumer can look like this:
// you can use volatile flag instead if you like
while (!Thread.currentThread.isInterrupted()) {
try {
Object item = queue.take();
// do something with item
} catch (InterruptedException e) {
log.error("Consumer interrupted", e);
Thread.currentThread().interrupt(); // restore flag
}
}
Then in your Mediator you can just call interrupt() on a consumer thread.
A poison pill is straight forward.
private static Socket poisonPill = new Socket();
public Void call() throws Exception {
while (runnable) {
Socket socket = mediator.takeFromQueue(); // blocks until element is in queue
if (socket == poisonPill) {
// quit the thread...
}
}
return null;
}
Note the socket == poisonPill. This is an equality check that they're the exact same instance, so that's how the poisonPill works yet still being type safe.
I am trying to develop active object pattern in concurrent Java using java.util.concurrent classes.
I describe it using a Client and a Server. A sample Server is as:
class Server implements Runnable {
public final LinkedBlockingQueue que = new LinkedBlockingQueue();
private final ExecutorService es = Executors.newCachedThreadPool();
private Message currentMessage;
private boolean state = false;
public init() {
es.submit(this);
}
public void requestForServer() {
if (state) {
this.currentMessage.await();
}
state = true;
}
public void run() {
for(;;) {
Message m = que.take();
this.currentMessage = m;
this.es.submit(m);
}
}
}
And a sample Client:
class Client {
private Server server;
public Client(Server s) {
this.server = s;
}
public void doSomething() {
Message m = new Message(new Callable() {
public Object call() {
server.requestForServer();
}
});
this.server.que.add(m);
}
}
And a sample Message encapsulation is:
class Message<V> extends FutureTask<V> {
private Lock lock = new ReentrantLock();
private Condition condition = new Condition();
public Message(Callable<V> callable) {
super(callable);
}
public void run() {
try {
lock.lock();
super.run();
lock.unlock();
} catch(Exception e) {}
}
public void await() {
try {
condition.await();
} catch(Exception e) {}
}
public void signal() {
try {
condition.signalAll();
} catch(Exception e) {}
}
}
And a sample running code:
Server s = new Server();
Client c = new Client (s);
s.init();
c.doSomething();
I dropped some implementation details to get my message across.
Now, the problem is when in Server the state is true so the incoming message should wait and the await is called on the current message. However, I get IllegalMonitorStateException which means that the current message does not own the current thread to await on it. But, I believe this is strange since the current message gets called in the Server and its thread pool so the current message has also an access to the current thread of execution.
I'd be most thankful for any ideas or suggestions, or with a known working implementation of this pattern using java.util.concurrent. Thanks in advance.
UPDATE:
I discussed the solution I could deploy in this blog post. I hope it could help.
You have to actually acquire the lock when you await on its corresponding condition. Without that lock you cannot associate yourself to the condition directly. To demonstrate this:
public void await() {
lock.lock();
try {
condition.await();
} catch(Exception e) {}
finally{
lock.unlock();
}
}
That should resolve your IllegalMonitorStateException
On a side note of correctness you should always release a lock in a try{ } finally{ } manner, you can observe what I wrote as an example. The reason for this is if an exception occurs between lock().lock(); and super.run(); lock.unlock() will never be called.