I just want to test agrona.OneToOneRingBuffer. i have one Producer to produce message,one Consumer to consume.But My consumer class has no output because countdownlatch doesn't zero out.
public class Producer implements Runnable{
private final RingBuffer buffer;
public Producer(RingBuffer buffer) {
this.buffer = buffer;
}
#Override
public void run() {
for(int i=0;i<Config.SIZE;i++){
String s = String.format("i am %s",i);
System.out.println( "name -> " + s);
UnsafeBuffer unsafeBuffer = new UnsafeBuffer(s.getBytes());
unsafeBuffer.wrap(s.getBytes());
buffer.write(1, unsafeBuffer, 0, s.length());
}
}
}
public class Consumer implements Runnable {
private final RingBuffer buffer;
public Consumer(RingBuffer buffer) {
this.buffer = buffer;
}
#Override
public void run() {
long start = System.currentTimeMillis();
CountDownLatch countDownLatch = new CountDownLatch(Config.SIZE);
while (countDownLatch.getCount() > 0) {
buffer.read((msgTypeId, srcBuffer, index, length) -> {
byte[] message = new byte[length];
srcBuffer.getBytes(index, message);
System.out.println("Consumer <- " + new String(message));
countDownLatch.countDown();
});
}
long end = System.currentTimeMillis();
System.out.println("cost time " + (end - start));
}
}
public class App {
private static final OneToOneRingBuffer BUFFER = new OneToOneRingBuffer(new UnsafeBuffer(
ByteBuffer.allocate(1024 + RingBufferDescriptor.TRAILER_LENGTH)));
public static void main(String[] args) throws Exception {
ExecutorService executor = Executors.newCachedThreadPool();
executor.execute(new Producer(BUFFER));
executor.execute(new Consumer(BUFFER));
Thread.currentThread().join();
}
}
The Config.SIZE just set to 100_000 and when it was 10_000 ,the programe runs well.
Is the OneToOneRingBuffer class a thread-unsafe class?
Related
In the below code, restarting the tailer process is ok. However, restarting the appender process results in the tailer failing to receive any more messages. Is there a way to restart the appender and keep the channel open?
Edited: Below is a full class that I've used to recreate the issue consistently.
Environment:
Ubuntu 18
chronicle-queue-5.16.9.jar
1) java com.tradeplacer.util.IpcTest producer
2) java com.tradeplacer.util.IpcTest consumer
3) kill the producer
4) restart the producer
5) notice that the consumer is no longer reading anything
package com.tradeplacer.util;
import java.nio.ByteBuffer;
import net.openhft.chronicle.bytes.Bytes;
import net.openhft.chronicle.queue.ChronicleQueue;
import net.openhft.chronicle.queue.ChronicleQueueBuilder;
import net.openhft.chronicle.queue.ExcerptAppender;
import net.openhft.chronicle.queue.ExcerptTailer;
import net.openhft.chronicle.queue.RollCycles;
public class IpcTest {
private static final String DIR = "chronicle-test";
public static final void startProducer() {
new Thread() {
public void run() {
System.out.println("starting producer...");
ChronicleQueue queue = ChronicleQueueBuilder.single(DIR).blockSize(65536).rollCycle(RollCycles.MINUTELY).build();
ExcerptAppender appender = queue.acquireAppender();
ByteBuffer ipcBuffer = ByteBuffer.allocate(8192);
for (int i = 0; i < Integer.MAX_VALUE; i++) {
ipcBuffer.clear();
ipcBuffer.put(("data" + i).getBytes());
Bytes<ByteBuffer> bbb = Bytes.wrapForWrite(ipcBuffer);
appender.writeBytes(bbb);
try {
Thread.sleep(1);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}.start();
}
public static final void startConsumer() {
new Thread() {
public void run() {
System.out.println("starting consumer...");
ChronicleQueue queue = ChronicleQueueBuilder.single(DIR).blockSize(65536).rollCycle(RollCycles.MINUTELY).build();
ExcerptTailer tailer = queue.createTailer().toEnd(); // skip to end, don't read old messages
Bytes bytes = Bytes.allocateDirect(8192);
while (true) {
try {
long ipcIndex = tailer.index();
boolean read = tailer.readBytes(bytes);
int len = bytes.length();
byte[] data = new byte[len];
bytes.read(data);
if (read) {
System.out.println("read " + data);
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
}.start();
}
public static void main(final String[] args) {
if ("producer".equals(args[0]))
startProducer();
else
startConsumer();
}
}
I have modified the code a little to reduce object creation. On the latest version 5.17.1, I can restart the producer many times and the consumer keeps reading data.
NOTE: If you are going to write text, the writeText method might be a better choice.
If you want to write something more complex I suggest using Wire or each MethodReader/MethodWriters which allow you to make interface method calls.
package net.openhft.chronicle.queue;
import net.openhft.chronicle.bytes.Bytes;
import net.openhft.chronicle.core.Jvm;
import net.openhft.chronicle.queue.impl.single.SingleChronicleQueueBuilder;
import net.openhft.chronicle.threads.Pauser;
import java.nio.ByteBuffer;
public class IpcTest {
private static final String DIR = "chronicle-test";
public static final void startProducer() {
System.out.println("starting producer...");
ChronicleQueue queue = SingleChronicleQueueBuilder.single(DIR).blockSize(65536).rollCycle(RollCycles.MINUTELY).build();
ExcerptAppender appender = queue.acquireAppender();
Bytes<ByteBuffer> bytes = Bytes.elasticByteBuffer(8192);
ByteBuffer ipcBuffer = bytes.underlyingObject();
for (int i = 0; i < Integer.MAX_VALUE; i++) {
ipcBuffer.clear();
ipcBuffer.put(("data" + i).getBytes());
bytes.readPositionRemaining(0, ipcBuffer.position());
appender.writeBytes(bytes);
Jvm.pause(1);
}
}
public static final void startConsumer() {
System.out.println("starting consumer...");
ChronicleQueue queue = SingleChronicleQueueBuilder.single(DIR).blockSize(65536).rollCycle(RollCycles.MINUTELY).build();
ExcerptTailer tailer = queue.createTailer().toEnd(); // skip to end, don't read old messages
Bytes<ByteBuffer> bytes = Bytes.elasticHeapByteBuffer(8192);
Pauser pauser = Pauser.balanced();
while (true) {
try {
long ipcIndex = tailer.index();
bytes.clear();
boolean read = tailer.readBytes(bytes);
if (read) {
byte[] data = bytes.underlyingObject().array();
int len = (int) bytes.readRemaining();
System.out.println("read " + new String(data, 0, 0, len));
pauser.reset();
} else {
pauser.pause();
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
public static void main(final String[] args) {
if ("producer".equals(args[0]))
startProducer();
else
startConsumer();
}
}
Using MethodReader/MethodWriter
public class IpcTest {
interface Hello {
void hello(String text);
}
private static final String DIR = "chronicle-test";
public static final void startProducer() {
System.out.println("starting producer...");
ChronicleQueue queue = SingleChronicleQueueBuilder.single(DIR).blockSize(65536).rollCycle(RollCycles.MINUTELY).build();
Hello hello = queue.methodWriter(Hello.class);
for (int i = 0; i < Integer.MAX_VALUE; i++) {
hello.hello("data" + i);
Jvm.pause(1);
}
}
public static final void startConsumer() {
System.out.println("starting consumer...");
ChronicleQueue queue = SingleChronicleQueueBuilder.single(DIR).blockSize(65536).rollCycle(RollCycles.MINUTELY).build();
Hello hello = text -> System.out.println("read " + text);
MethodReader reader = queue.createTailer().methodReader(hello);
Pauser pauser = Pauser.balanced();
while (true) {
if (reader.readOne()) {
pauser.reset();
} else {
pauser.pause();
}
}
}
public static void main(final String[] args) {
if ("producer".equals(args[0]))
startProducer();
else
startConsumer();
}
}
You can use a DTO with is AbstractMarshallable to make it efficient to serialize and deserialize.
package net.openhft.chronicle.queue;
import net.openhft.chronicle.bytes.MethodReader;
import net.openhft.chronicle.core.Jvm;
import net.openhft.chronicle.core.pool.ClassAliasPool;
import net.openhft.chronicle.queue.impl.single.SingleChronicleQueueBuilder;
import net.openhft.chronicle.threads.Pauser;
import net.openhft.chronicle.wire.AbstractMarshallable;
public class IpcTest {
static class Hi extends AbstractMarshallable {
String text;
int value;
}
interface Hello {
void hi(Hi hi);
}
private static final String DIR = "chronicle-test";
public static final void startProducer() {
System.out.println("starting producer...");
ChronicleQueue queue = SingleChronicleQueueBuilder.single(DIR).blockSize(65536).rollCycle(RollCycles.MINUTELY).build();
Hello hello = queue.methodWriter(Hello.class);
Hi hi = new Hi();
for (int i = 0; i < Integer.MAX_VALUE; i++) {
hi.text = "data";
hi.value = i;
hello.hi(hi);
Jvm.pause(1);
}
}
public static final void startConsumer() {
System.out.println("starting consumer...");
ChronicleQueue queue = SingleChronicleQueueBuilder.single(DIR).blockSize(65536).rollCycle(RollCycles.MINUTELY).build();
Hello hello = text -> System.out.println("read " + text);
MethodReader reader = queue.createTailer().methodReader(hello);
Pauser pauser = Pauser.balanced();
while (true) {
if (reader.readOne()) {
pauser.reset();
} else {
pauser.pause();
}
}
}
public static void main(final String[] args) {
ClassAliasPool.CLASS_ALIASES.addAlias(Hi.class);
if ("producer".equals(args[0]))
startProducer();
else
startConsumer();
}
}
In this case, the consumer prints
....
read !Hi {
text: data,
value: 3862
}
read !Hi {
text: data,
value: 3863
}
read !Hi {
text: data,
value: 3864
}
....
I am trying to write and read from ArrayBlockingQueue via threads. I have thread factory and worker threads but for some reason I cannot read the data that I pass to to worker threads, within the worker threads. Any idea? Thanks.
public class CommunicationThreadFactory implements ThreadFactory {
#Override
public Thread newThread(Runnable runnable){
Thread thread=new Thread(runnable);
return thread;
}
}
public class ThreadEx implements Runnable {
private byte[] pack;
private final BlockingQueue writeBlockingQueue;
public Writer(BlockingQueue writeBlockingQueue, byte[] pack) {
this.writeBlockingQueue = writeBlockingQueue;
this.pack = pack;
}
#Override
public void run() {
TimeUnit.SECONDS.sleep(1);//Even this line block it wont process below this line. Even after time out.
writeBlockingQueue.put(pack);//Tried even with disabling put and takes from queue. still did not work
System.out.println("In Thread "+Thread.currentThread().getName() +" got "+ this.pack.length);// This line does not even produce output.
writeBlockingQueue.take();
}
}
public static void main(String[] args) throws UnsupportedEncodingException {
ExecutorService connectionThreadPool = Executors.newFixedThreadPool(15,new
CommunicationThreadFactory());
BlockingQueue<byte[]> blockingQueue = new ArrayBlockingQueue<>(10, true);
byte[] packet = new byte[]{0x63, 0x41, 0x35, 0x19};
for (int i = 0; i < 10; i++) {
connectionThreadPool.execute(new ThreadEx());
}
}
Your code does not compile. After correcting some synax errors, it works fine.
import java.io.UnsupportedEncodingException;
import java.util.concurrent.*;
class CommunicationThreadFactory implements ThreadFactory {
#Override
public Thread newThread(Runnable runnable) {
Thread thread = new Thread(runnable);
return thread;
}
}
public class ThreadEx implements Runnable {
private byte[] pack;
private final BlockingQueue writeBlockingQueue;
public ThreadEx(BlockingQueue writeBlockingQueue, byte[] pack) {
this.writeBlockingQueue = writeBlockingQueue;
this.pack = pack;
}
#Override
public void run() {
try {
TimeUnit.SECONDS.sleep(1);
writeBlockingQueue.put(pack);
// This line does not even produce output.
System.out.println("In Thread " + Thread.currentThread().getName() + " got "
+ this.pack.length);
writeBlockingQueue.take();
} catch (Exception e) {
}
}
public static void main(String[] args) throws UnsupportedEncodingException {
ExecutorService connectionThreadPool = Executors.newFixedThreadPool(15, new
CommunicationThreadFactory());
BlockingQueue<byte[]> blockingQueue = new ArrayBlockingQueue<>(10, true);
byte[] packet = new byte[]{0x63, 0x41, 0x35, 0x19};
for (int i = 0; i < 10; i++) {
connectionThreadPool.execute(new ThreadEx(blockingQueue, packet));
}
}
}
I'm trying to solve single consumer/producer problem using monitor in Java, and the code is as follows. When I run this code, it will finally get stucked. The most typical case is that the consumer calls wait(), and then the producer keeps producing but cannot notify the consumer (although it will call notify()). I don't know why it's happening. Java code:
import java.util.*;
class Monitor {
int length;
int size;
int begin, end;
int queue[];
private static Random randGenerator;
public Monitor() {}
public Monitor(int length) {
this.length = length;
this.size = 0;
begin = end = 0;
queue = new int[length];
randGenerator = new Random(10);
}
public synchronized void produce() throws InterruptedException {
while(size == length) {
System.out.println("Producer waiting");
wait();
}
int produced = randGenerator.nextInt();
size++;
queue[end] = produced;
end = (end + 1) % length;
System.out.println("Produce element " + produced + " size "+size);
// When size is not 1, no thread is blocked and therefore don't need to notify
if(size == 1) {
System.out.println("Notify consumer");
notify();
}
}
public synchronized void consume() throws InterruptedException {
while(size == 0) {
System.out.println("Consumer waiting, size " + size);
wait();
}
size--;
System.out.println("Consume element " + queue[begin] + " size " + size);
begin = (begin + 1) % length;
if(size == length - 1) {
System.out.println("Notify producer");
notify();
}
}
}
class Producer implements Runnable {
Monitor producer;
public Producer(Monitor m) {
producer = m;
}
#Override
public void run() {
producer = new Monitor();
System.out.println("Producer created");
try {
while(true) {
producer.produce();
}
}
catch (InterruptedException e) {
e.printStackTrace();
}
}
}
class Consumer implements Runnable {
Monitor consumer;
public Consumer(Monitor m) {
consumer = m;
}
#Override
public void run() {
System.out.println("Consumer created");
consumer = new Monitor();
try {
while(true) {
consumer.consume();
}
}
catch (InterruptedException e) {
e.printStackTrace();
}
}
}
public class monitorTest {
public static void main(String args[]) {
Monitor monitor = new Monitor(10);
Thread t1 = new Thread(new Producer(monitor));
Thread t2 = new Thread(new Consumer(monitor));
t1.start();
t2.start();
}
}
When the control of each thread enters the produce() or consume() methods, the size and length are both zero and hence both threads are waiting for the other to notify. Break this and your code will come out of the deadlock.
public synchronized void produce() throws InterruptedException {
while(size == length) { // size is 0 and length is 0; so wait
System.out.println("Producer waiting");
wait();
}
public synchronized void consume() throws InterruptedException {
while(size == 0) { // size is 0 so wait
System.out.println("Consumer waiting, size " + size);
wait();
}
This is happening because you have a default constructor which you are calling inside the run() method of your Producer and Consumer objects.
class Producer implements Runnable {
Monitor producer;
public Producer(Monitor m) {
producer = m;
}
#Override
public void run() {
producer = new Monitor(); // REMOVE THIS
class Consumer implements Runnable {
Monitor consumer;
public Consumer(Monitor m) {
consumer = m;
}
#Override
public void run() {
System.out.println("Consumer created");
consumer = new Monitor(); // AND REMOVE THIS
Hope this helps!
I study java concurency.
I am trying to estimate time execution depends on thread count(read and write)
my code:
public class Task5 {
public static int [] readerThreadCount = {1,10,100,1000};
public static int [] writerThreadCount = {10, 1000, 1000000};
public static void main(String[] args) throws InterruptedException {
for (int readCount : readerThreadCount) {
for (int writeCount : writerThreadCount) {
System.out.println(readCount + "/" + writeCount + " = " + test(readCount, writeCount, new ArrayHolderBySynchronized()));
}
}
}
private static long test(int readCount, int writeCount, ArrayHolder arrayHolder) throws InterruptedException {
CountDownLatch countDownLatch = new CountDownLatch(readCount + writeCount);
List<Thread> threads = new ArrayList<>();
for (int i = 0; i < readCount; i++) {
threads.add(new Thread(new ArrayReader(arrayHolder, countDownLatch)));
}
for (int i = 0; i < writeCount; i++) {
threads.add(new Thread(new ArrayWriter(arrayHolder, countDownLatch)));
}
for(Thread thread:threads){
thread.start();
}
countDownLatch.await();//all threads started
long start = System.currentTimeMillis();
for (Thread thread : threads) {
thread.join();
}
return System.currentTimeMillis() - start;
}
}
class ArrayHolderBySynchronized extends ArrayHolder {
#Override
public synchronized int get(int index) {
return arr[index];
}
#Override
public synchronized void write(int index, int value) {
arr[index] = value;
}
}
class ArrayReader implements Runnable {
ArrayHolder arrayHolder;
CountDownLatch countDownLatch;
ArrayReader(ArrayHolder arrayHolder, CountDownLatch countDownLatch) {
this.arrayHolder = arrayHolder;
this.countDownLatch = countDownLatch;
}
#Override
public void run() {
countDownLatch.countDown();
arrayHolder.get(new Random().nextInt(ArrayHolder.ARRAY_SIZE));
}
}
class ArrayWriter implements Runnable {
ArrayHolder arrayHolder;
CountDownLatch countDownLatch;
ArrayWriter(ArrayHolder arrayHolder, CountDownLatch countDownLatch) {
this.arrayHolder = arrayHolder;
this.countDownLatch = countDownLatch;
}
#Override
public void run() {
countDownLatch.countDown();
arrayHolder.write(new Random().nextInt(ArrayHolder.ARRAY_SIZE), -1);
}
}
abstract class ArrayHolder {
public static int ARRAY_SIZE = 1_000_000;
protected int[] arr = generateArray();
private int[] generateArray() {
int[] arr = new int[ARRAY_SIZE];
for (int i = 0; i < ARRAY_SIZE; i++) {
arr[i] = i + 1;
}
return arr;
}
public abstract int get(int index);
public abstract void write(int index, int value);
}
it outputs
1/10 = 0
1/1000 = 1
and hangs.
I have not ideas why.
please help.
It doesn't hang, starting 1000000 threads just takes 1000 times longer than starting 1000 threads (a couple minutes, on my machine):
> java Task5
1/10 = 0
1/1000 = 1
1/1000000 = 63
10/10 = 0
10/1000 = 0
10/1000000 = 60
100/10 = 0
100/1000 = 0
100/1000000 = 63
1000/10 = 0
1000/1000 = 0
1000/1000000 = 60
Your next question will likely be why your test reports a duration of 60 ms when it took minutes to execute. That's because starting a threads is far more expensive than counting down a latch, or reading from or writing to a single array element, and you only measure the latter.
I have the problem regarding the implementation of One Publisher - Multiple Subscribers pattern. The Publisher uses the fixed-size buffer and queue the messages. The messages are send to all subscribers. The ordering of messages get by subscribers must be the same as the ordering of publishing messages.
I use BlockingQueue to hold publisher messages (publisherQueue) and pass them to each subscriber BlockingQueue (subscriberQueue).
The issue is that the buffer and subscribers are working correctly, but the buffer size (publisherQueue.size()) always returns 1.
System.out.println("Actual number of messages in buffer: " + publisherQueue.size());
Here is my full code:
PublisherSubscriberService.java
package program;
import java.util.HashSet;
import java.util.Set;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
public class PublisherSubscriberService {
private int buffer;
private int subscribersNumber;
static Set<subscriber> subscribers = new HashSet<subscriber>();
public PublisherSubscriberService(int buffer, int subscribersNumber) {
this.buffer = buffer;
this.subscribersNumber = subscribersNumber;
}
public void addsubscriber(subscriber subscriber) {
subscribers.add(subscriber);
}
public void start() {
publisher publisher = new publisher(buffer);
System.out.println("publisher started the job");
for (int i = 0; i < subscribersNumber; i++) {
subscriber subscriber = new subscriber(buffer);
subscriber.setName(Integer.toString(i + 1));
subscribers.add(subscriber);
new Thread(subscriber).start();
System.out.println("Subscriber " + subscriber.getName() + " started the job");
}
new Thread(publisher).start();
}
public class Publisher implements Runnable {
private int buffer;
final BlockingQueue<Message> publisherQueue;
public Publisher(int buffer) {
this.buffer = buffer;
publisherQueue = new LinkedBlockingQueue<>(buffer);
}
#Override
public void run() {
for (int i = 1; i < 100; i++) {
Message messageObject = new Message("" + i);
try {
Thread.sleep(50);
publisherQueue.put(messageObject);
System.out.println("Queued message no " + messageObject.getMessage());
System.out.println("Actual number of messages in buffer: " + publisherQueue.size());
for (subscriber subscriber : subscribers) {
subscriber.subscriberQueue.put(messageObject);
}
publisherQueue.take();
} catch (InterruptedException e) {
System.out.println("Some error");
e.printStackTrace();
}
}
}
}
class Subscriber implements Runnable {
private String name;
private int buffer;
final BlockingQueue<Message> subscriberQueue;
public Subscriber(int buffer) {
this.buffer = buffer;
subscriberQueue = new LinkedBlockingQueue<>(buffer);
}
public void setName(String name) {
this.name = name;
}
public String getName() {
return name;
}
#Override
public void run() {
try {
Message messageObject;
while (true) {
Thread.sleep(100);
messageObject = subscriberQueue.take();
System.out.println(this.getName() + " got message: " + messageObject.getMessage());
}
} catch (InterruptedException e) {
System.out.println("Some error");
e.printStackTrace();
}
}
}
class Message {
private String message;
public Message(String str) {
this.message = str;
}
public String getMessage() {
return message;
}
}
}
PublisherSubscriberProgram.java
package program;
public class ProducerConsumerProgram {
public static void main(String[] args) {
ProducerConsumerService service = new ProducerConsumerService(10, 3);
service.start();
}
}
Your publisher never has more than 1 item in the queue. Each time through your loop you put and take a single item:
**publisherQueue.put(messageObject);**
System.out.println("Queued message no " + messageObject.getMessage());
System.out.println("Actual number of messages in buffer: " + publisherQueue.size());
for (subscriber subscriber : subscribers) {
subscriber.subscriberQueue.put(messageObject);
}
**publisherQueue.take();**
With the code you have provided, there is point in even having the publisher queue.