In the below code, restarting the tailer process is ok. However, restarting the appender process results in the tailer failing to receive any more messages. Is there a way to restart the appender and keep the channel open?
Edited: Below is a full class that I've used to recreate the issue consistently.
Environment:
Ubuntu 18
chronicle-queue-5.16.9.jar
1) java com.tradeplacer.util.IpcTest producer
2) java com.tradeplacer.util.IpcTest consumer
3) kill the producer
4) restart the producer
5) notice that the consumer is no longer reading anything
package com.tradeplacer.util;
import java.nio.ByteBuffer;
import net.openhft.chronicle.bytes.Bytes;
import net.openhft.chronicle.queue.ChronicleQueue;
import net.openhft.chronicle.queue.ChronicleQueueBuilder;
import net.openhft.chronicle.queue.ExcerptAppender;
import net.openhft.chronicle.queue.ExcerptTailer;
import net.openhft.chronicle.queue.RollCycles;
public class IpcTest {
private static final String DIR = "chronicle-test";
public static final void startProducer() {
new Thread() {
public void run() {
System.out.println("starting producer...");
ChronicleQueue queue = ChronicleQueueBuilder.single(DIR).blockSize(65536).rollCycle(RollCycles.MINUTELY).build();
ExcerptAppender appender = queue.acquireAppender();
ByteBuffer ipcBuffer = ByteBuffer.allocate(8192);
for (int i = 0; i < Integer.MAX_VALUE; i++) {
ipcBuffer.clear();
ipcBuffer.put(("data" + i).getBytes());
Bytes<ByteBuffer> bbb = Bytes.wrapForWrite(ipcBuffer);
appender.writeBytes(bbb);
try {
Thread.sleep(1);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}.start();
}
public static final void startConsumer() {
new Thread() {
public void run() {
System.out.println("starting consumer...");
ChronicleQueue queue = ChronicleQueueBuilder.single(DIR).blockSize(65536).rollCycle(RollCycles.MINUTELY).build();
ExcerptTailer tailer = queue.createTailer().toEnd(); // skip to end, don't read old messages
Bytes bytes = Bytes.allocateDirect(8192);
while (true) {
try {
long ipcIndex = tailer.index();
boolean read = tailer.readBytes(bytes);
int len = bytes.length();
byte[] data = new byte[len];
bytes.read(data);
if (read) {
System.out.println("read " + data);
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
}.start();
}
public static void main(final String[] args) {
if ("producer".equals(args[0]))
startProducer();
else
startConsumer();
}
}
I have modified the code a little to reduce object creation. On the latest version 5.17.1, I can restart the producer many times and the consumer keeps reading data.
NOTE: If you are going to write text, the writeText method might be a better choice.
If you want to write something more complex I suggest using Wire or each MethodReader/MethodWriters which allow you to make interface method calls.
package net.openhft.chronicle.queue;
import net.openhft.chronicle.bytes.Bytes;
import net.openhft.chronicle.core.Jvm;
import net.openhft.chronicle.queue.impl.single.SingleChronicleQueueBuilder;
import net.openhft.chronicle.threads.Pauser;
import java.nio.ByteBuffer;
public class IpcTest {
private static final String DIR = "chronicle-test";
public static final void startProducer() {
System.out.println("starting producer...");
ChronicleQueue queue = SingleChronicleQueueBuilder.single(DIR).blockSize(65536).rollCycle(RollCycles.MINUTELY).build();
ExcerptAppender appender = queue.acquireAppender();
Bytes<ByteBuffer> bytes = Bytes.elasticByteBuffer(8192);
ByteBuffer ipcBuffer = bytes.underlyingObject();
for (int i = 0; i < Integer.MAX_VALUE; i++) {
ipcBuffer.clear();
ipcBuffer.put(("data" + i).getBytes());
bytes.readPositionRemaining(0, ipcBuffer.position());
appender.writeBytes(bytes);
Jvm.pause(1);
}
}
public static final void startConsumer() {
System.out.println("starting consumer...");
ChronicleQueue queue = SingleChronicleQueueBuilder.single(DIR).blockSize(65536).rollCycle(RollCycles.MINUTELY).build();
ExcerptTailer tailer = queue.createTailer().toEnd(); // skip to end, don't read old messages
Bytes<ByteBuffer> bytes = Bytes.elasticHeapByteBuffer(8192);
Pauser pauser = Pauser.balanced();
while (true) {
try {
long ipcIndex = tailer.index();
bytes.clear();
boolean read = tailer.readBytes(bytes);
if (read) {
byte[] data = bytes.underlyingObject().array();
int len = (int) bytes.readRemaining();
System.out.println("read " + new String(data, 0, 0, len));
pauser.reset();
} else {
pauser.pause();
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
public static void main(final String[] args) {
if ("producer".equals(args[0]))
startProducer();
else
startConsumer();
}
}
Using MethodReader/MethodWriter
public class IpcTest {
interface Hello {
void hello(String text);
}
private static final String DIR = "chronicle-test";
public static final void startProducer() {
System.out.println("starting producer...");
ChronicleQueue queue = SingleChronicleQueueBuilder.single(DIR).blockSize(65536).rollCycle(RollCycles.MINUTELY).build();
Hello hello = queue.methodWriter(Hello.class);
for (int i = 0; i < Integer.MAX_VALUE; i++) {
hello.hello("data" + i);
Jvm.pause(1);
}
}
public static final void startConsumer() {
System.out.println("starting consumer...");
ChronicleQueue queue = SingleChronicleQueueBuilder.single(DIR).blockSize(65536).rollCycle(RollCycles.MINUTELY).build();
Hello hello = text -> System.out.println("read " + text);
MethodReader reader = queue.createTailer().methodReader(hello);
Pauser pauser = Pauser.balanced();
while (true) {
if (reader.readOne()) {
pauser.reset();
} else {
pauser.pause();
}
}
}
public static void main(final String[] args) {
if ("producer".equals(args[0]))
startProducer();
else
startConsumer();
}
}
You can use a DTO with is AbstractMarshallable to make it efficient to serialize and deserialize.
package net.openhft.chronicle.queue;
import net.openhft.chronicle.bytes.MethodReader;
import net.openhft.chronicle.core.Jvm;
import net.openhft.chronicle.core.pool.ClassAliasPool;
import net.openhft.chronicle.queue.impl.single.SingleChronicleQueueBuilder;
import net.openhft.chronicle.threads.Pauser;
import net.openhft.chronicle.wire.AbstractMarshallable;
public class IpcTest {
static class Hi extends AbstractMarshallable {
String text;
int value;
}
interface Hello {
void hi(Hi hi);
}
private static final String DIR = "chronicle-test";
public static final void startProducer() {
System.out.println("starting producer...");
ChronicleQueue queue = SingleChronicleQueueBuilder.single(DIR).blockSize(65536).rollCycle(RollCycles.MINUTELY).build();
Hello hello = queue.methodWriter(Hello.class);
Hi hi = new Hi();
for (int i = 0; i < Integer.MAX_VALUE; i++) {
hi.text = "data";
hi.value = i;
hello.hi(hi);
Jvm.pause(1);
}
}
public static final void startConsumer() {
System.out.println("starting consumer...");
ChronicleQueue queue = SingleChronicleQueueBuilder.single(DIR).blockSize(65536).rollCycle(RollCycles.MINUTELY).build();
Hello hello = text -> System.out.println("read " + text);
MethodReader reader = queue.createTailer().methodReader(hello);
Pauser pauser = Pauser.balanced();
while (true) {
if (reader.readOne()) {
pauser.reset();
} else {
pauser.pause();
}
}
}
public static void main(final String[] args) {
ClassAliasPool.CLASS_ALIASES.addAlias(Hi.class);
if ("producer".equals(args[0]))
startProducer();
else
startConsumer();
}
}
In this case, the consumer prints
....
read !Hi {
text: data,
value: 3862
}
read !Hi {
text: data,
value: 3863
}
read !Hi {
text: data,
value: 3864
}
....
Related
I just want to test agrona.OneToOneRingBuffer. i have one Producer to produce message,one Consumer to consume.But My consumer class has no output because countdownlatch doesn't zero out.
public class Producer implements Runnable{
private final RingBuffer buffer;
public Producer(RingBuffer buffer) {
this.buffer = buffer;
}
#Override
public void run() {
for(int i=0;i<Config.SIZE;i++){
String s = String.format("i am %s",i);
System.out.println( "name -> " + s);
UnsafeBuffer unsafeBuffer = new UnsafeBuffer(s.getBytes());
unsafeBuffer.wrap(s.getBytes());
buffer.write(1, unsafeBuffer, 0, s.length());
}
}
}
public class Consumer implements Runnable {
private final RingBuffer buffer;
public Consumer(RingBuffer buffer) {
this.buffer = buffer;
}
#Override
public void run() {
long start = System.currentTimeMillis();
CountDownLatch countDownLatch = new CountDownLatch(Config.SIZE);
while (countDownLatch.getCount() > 0) {
buffer.read((msgTypeId, srcBuffer, index, length) -> {
byte[] message = new byte[length];
srcBuffer.getBytes(index, message);
System.out.println("Consumer <- " + new String(message));
countDownLatch.countDown();
});
}
long end = System.currentTimeMillis();
System.out.println("cost time " + (end - start));
}
}
public class App {
private static final OneToOneRingBuffer BUFFER = new OneToOneRingBuffer(new UnsafeBuffer(
ByteBuffer.allocate(1024 + RingBufferDescriptor.TRAILER_LENGTH)));
public static void main(String[] args) throws Exception {
ExecutorService executor = Executors.newCachedThreadPool();
executor.execute(new Producer(BUFFER));
executor.execute(new Consumer(BUFFER));
Thread.currentThread().join();
}
}
The Config.SIZE just set to 100_000 and when it was 10_000 ,the programe runs well.
Is the OneToOneRingBuffer class a thread-unsafe class?
I'm doing some courses on MultiThreading in Java. I've tried (following instructor) to synchronise two consumers reading from ArrayList with one producer filling it with some basic input. Yet producer gives first number and consumer gets in infinite loop because it gets empty List. No idea how to force it to get proper values.
Producer:
class MyProducer implements Runnable {
private List<String> buffer;
private String color;
public MyProducer(List<String> buffer, String color) {
this.buffer = buffer;
this.color = color;
}
public void run() {
Random random = new Random();
String[] nums = {"1", "2", "3", "4", "5"};
for(String num: nums) {
try {
System.out.println(color + "Adding..." + num);
synchronized (buffer) {
buffer.add(num);
System.out.println(buffer);
}
Thread.sleep(random.nextInt(1000));
} catch (InterruptedException e) {
System.out.println("Producer was interrupted");
}
}
System.out.println(color + "Adding EOF and exiting...");
synchronized (buffer) {
buffer.add(EOF);
}
}
}
Consumer:
class MyConsumer implements Runnable {
private List<String> buffer;
private String color;
public MyConsumer(List<String> buffer, String color) {
this.buffer = buffer;
this.color = color;
}
public void run() {
synchronized (buffer) {
while(true) {
if (buffer.isEmpty()) {
continue;
}
System.out.println("Przejście między warunkami działa");
if (buffer.get(0).equals(EOF)) {
System.out.println(color + "Exiting");
break;
} else {
System.out.println(color + "Removed" + buffer.remove(0));
}
}
}
}
}
Main:
import java.util.ArrayList;
import java.util.List;
import java.util.Random;
import static com.siecz-k-.Main.EOF;
public class Main {
public static final String EOF = "EOF";
public static void main(String[] args) {
List<String> buffer = new ArrayList<>();
MyProducer producer = new MyProducer(buffer, ThreadColor.ANSI_CYAN);
MyConsumer consumer1 = new MyConsumer(buffer, ThreadColor.ANSI_PURPLE);
MyConsumer consumer2 = new MyConsumer(buffer, ThreadColor.ANSI_RED);
new Thread(producer).start();
new Thread(consumer1).start();
new Thread(consumer2).start();
}
}
I am trying to implement the observer pattern to a game i have made. When a villain is created in the battle-zone file using threads, I would like to use the observer pattern to create a hero using threads and add it to the same file. The villians and heroes are created using the factory method pattern. I am unsure of where to go with regards to linking my HeroCreationMain class to the observer pattern classes.
Villian Creation
public class VillianCreationMain {
private static Villian villian;
public static void main(String[] args, int userInput) throws IOException {
String fileName = null;
Random randomVillian = new Random();
int amountOfVillians = userInput;
if (amountOfVillians < 7) {
for (int x = 0; x < amountOfVillians; x++) {
int randomGenerator = randomVillian.nextInt(6);
for (int i = 0; i < 5; i++) {
if (randomGenerator == 0 ) {
setVillian(new FlyingVillian());
}
else if (randomGenerator == 1) {
setVillian(new StrongVillian());
}
else if (randomGenerator == 2) {
setVillian(new FastVillian());
}
else if (randomGenerator == 3) {
setVillian(new SmartVillian());
}
else if (randomGenerator == 4) {
setVillian(new FireVillian());
}
else if (randomGenerator == 5) {
setVillian(new IceVillian());
}
try {
writeToFile(getVillian(), i, fileName);
}
catch (IOException e) {
System.out.println(e.getMessage());
}
}
VillianThreads t1 = new VillianThreads(VillianCreationMain.getVillian());
t1.start();
}
}
else {
System.out.println("Please enter a value of less than 7");
}
}
public static void writeToFile(Villian villian, int amountOfVillians, String fileName) throws IOException {
for(int x = 0; x < amountOfVillians; x++) {
// String parsedInt = Integer.toString(x);
fileName = "battle-zone.ser";
FileOutputStream file = new FileOutputStream(fileName);
ObjectOutputStream oos = new ObjectOutputStream(file);
oos.writeObject(villian);
file.close();
oos.close();
}
}
public static Villian getVillian() {
return villian;
}
public static void setVillian(Villian villian) {
VillianCreationMain.villian = villian;
}
}
Hero Creation
public class HeroCreationMain {
private static Hero hero = null;
public static void main(String[] Hero) {
EnemyStatus enemyStatus = new EnemyStatus();
VillianObserver observer1 = new VillianObserver(enemyStatus);
}
public static void readFile() throws IOException, ClassNotFoundException {
#SuppressWarnings("resource")
ObjectInputStream ois = new ObjectInputStream (new FileInputStream("battle-zone.ser"));
Villian targetVillian = (Villian) ois.readObject();
System.out.println(targetVillian + " is being attacked by a hero!");
}
public static Hero getHero() {
return hero;
}
public static void setHero(Hero hero) {
HeroCreationMain.hero = hero;
}
}
Observer
public interface Observer {
public void update(boolean enemyPresent);
}
public interface Subject {
public void register(Observer o);
public void unregister(Observer o);
public void notifyObserver();
}
Observable
public class VillianObserver implements Observer {
private boolean enemyPresent;
private static int heroIDTracker;
private int heroID;
private Subject villianObserver;
public VillianObserver(Subject villianObserver) {
this.villianObserver = villianObserver;
this.heroID = ++heroIDTracker;
System.out.println("New Observer " + this.heroID);
villianObserver.register(this);
}
#Override
public void update(boolean enemyPresent) {
this.enemyPresent = enemyPresent;
printResult();
}
public void printResult() {
System.out.println(heroID + " " + enemyPresent);
}
}
Enemy Status
import java.util.ArrayList;
public class EnemyStatus implements Subject {
private ArrayList<Observer> observers;
private boolean enemyPresent;
public EnemyStatus() {
// Creates an ArrayList to hold all observers
observers = new ArrayList<Observer>();
}
#Override
public void register(Observer newObserver) {
observers.add(newObserver);
}
#Override
public void unregister(Observer deleteObserver) {
// Get the index of the observer to delete
int heroIndex = observers.indexOf(deleteObserver);
// Print out message (Have to increment index to match
System.out.println("Observer " + (heroIndex+1) + " deleted");
// Removes observer from the ArrayList
observers.remove(heroIndex);
}
#Override
public void notifyObserver() {
for(Observer observer : observers) {
observer.update(enemyPresent);
}
}
public void setEnemyStatus(boolean enemyPresent) {
this.enemyPresent = enemyPresent;
notifyObserver();
}
}
JNotify is the Java library to observe file changes on the file system.
One piece of advice: Object(Input/Output)Streams are easy when you're just getting started but they lead you down a path of ruin. Objects get so easily BadVersion'ed. Object files are also relatively hard to inspect using a text editor. I'd advise you to try using a different data format (like JSON) instead.
I have the problem regarding the implementation of One Publisher - Multiple Subscribers pattern. The Publisher uses the fixed-size buffer and queue the messages. The messages are send to all subscribers. The ordering of messages get by subscribers must be the same as the ordering of publishing messages.
I use BlockingQueue to hold publisher messages (publisherQueue) and pass them to each subscriber BlockingQueue (subscriberQueue).
The issue is that the buffer and subscribers are working correctly, but the buffer size (publisherQueue.size()) always returns 1.
System.out.println("Actual number of messages in buffer: " + publisherQueue.size());
Here is my full code:
PublisherSubscriberService.java
package program;
import java.util.HashSet;
import java.util.Set;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
public class PublisherSubscriberService {
private int buffer;
private int subscribersNumber;
static Set<subscriber> subscribers = new HashSet<subscriber>();
public PublisherSubscriberService(int buffer, int subscribersNumber) {
this.buffer = buffer;
this.subscribersNumber = subscribersNumber;
}
public void addsubscriber(subscriber subscriber) {
subscribers.add(subscriber);
}
public void start() {
publisher publisher = new publisher(buffer);
System.out.println("publisher started the job");
for (int i = 0; i < subscribersNumber; i++) {
subscriber subscriber = new subscriber(buffer);
subscriber.setName(Integer.toString(i + 1));
subscribers.add(subscriber);
new Thread(subscriber).start();
System.out.println("Subscriber " + subscriber.getName() + " started the job");
}
new Thread(publisher).start();
}
public class Publisher implements Runnable {
private int buffer;
final BlockingQueue<Message> publisherQueue;
public Publisher(int buffer) {
this.buffer = buffer;
publisherQueue = new LinkedBlockingQueue<>(buffer);
}
#Override
public void run() {
for (int i = 1; i < 100; i++) {
Message messageObject = new Message("" + i);
try {
Thread.sleep(50);
publisherQueue.put(messageObject);
System.out.println("Queued message no " + messageObject.getMessage());
System.out.println("Actual number of messages in buffer: " + publisherQueue.size());
for (subscriber subscriber : subscribers) {
subscriber.subscriberQueue.put(messageObject);
}
publisherQueue.take();
} catch (InterruptedException e) {
System.out.println("Some error");
e.printStackTrace();
}
}
}
}
class Subscriber implements Runnable {
private String name;
private int buffer;
final BlockingQueue<Message> subscriberQueue;
public Subscriber(int buffer) {
this.buffer = buffer;
subscriberQueue = new LinkedBlockingQueue<>(buffer);
}
public void setName(String name) {
this.name = name;
}
public String getName() {
return name;
}
#Override
public void run() {
try {
Message messageObject;
while (true) {
Thread.sleep(100);
messageObject = subscriberQueue.take();
System.out.println(this.getName() + " got message: " + messageObject.getMessage());
}
} catch (InterruptedException e) {
System.out.println("Some error");
e.printStackTrace();
}
}
}
class Message {
private String message;
public Message(String str) {
this.message = str;
}
public String getMessage() {
return message;
}
}
}
PublisherSubscriberProgram.java
package program;
public class ProducerConsumerProgram {
public static void main(String[] args) {
ProducerConsumerService service = new ProducerConsumerService(10, 3);
service.start();
}
}
Your publisher never has more than 1 item in the queue. Each time through your loop you put and take a single item:
**publisherQueue.put(messageObject);**
System.out.println("Queued message no " + messageObject.getMessage());
System.out.println("Actual number of messages in buffer: " + publisherQueue.size());
for (subscriber subscriber : subscribers) {
subscriber.subscriberQueue.put(messageObject);
}
**publisherQueue.take();**
With the code you have provided, there is point in even having the publisher queue.
LinkedList throws exception when trying to poll data. But I think i correctly use read/write lock concept. What is wrong with that code?
package sample;
import java.util.ArrayList;
import java.util.LinkedList;
import java.util.List;
import java.util.Queue;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReadWriteLock;
import java.util.concurrent.locks.ReentrantReadWriteLock;
public class PingPong extends Thread {
boolean read = false;
Queue<String> queue;
static ReadWriteLock lock = new ReentrantReadWriteLock();
final static Lock readLock = lock.readLock();
final static Lock writeLock = lock.writeLock();
boolean stop;
public PingPong(boolean read, Queue<String> queue) {
this.read = read;
this.queue = queue;
}
int count = 0;
#Override
public String toString() {
return "PingPong{" +
"read=" + read +
", count=" + count +
'}';
}
#Override
public void run() {
if (read) {
while (!stop) {
readLock.lock();
// synchronized (queue) {
try {
String string = queue.poll();
if (string != null) {
count++;
}
} finally {
readLock.unlock();
}
// }
inform();
}
} else {
while (!stop) {
writeLock.lock();
// synchronized (queue) {
try {
if (queue.add("some str" + count)) {
count++;
}
} finally {
writeLock.unlock();
}
// }
inform();
}
}
}
private void inform() {
// Thread.yield();
// synchronized (queue) {
// queue.notify();
// try {
// queue.wait(1);
// } catch (InterruptedException e) {
// e.printStackTrace(); //To change body of catch statement use File | Settings | File Templates.
// }
// }
}
public static void main(String[] args) throws InterruptedException {
Queue<String> queue = new LinkedList();
// queue = new ArrayBlockingQueue<String>(100);
// queue = new ConcurrentLinkedQueue<String>();
List<PingPong> pongs = new ArrayList<PingPong>();
for (int i = 0; i < 10; ++i) {
PingPong pingPong = new PingPong(i % 2 == 0, queue);
pingPong.start();
pongs.add(pingPong);
}
Thread.sleep(1000);
int sum = 0;
int read = 0;
int write = 0;
for (PingPong pp : pongs) {
pp.stop = true;
pp.join();
}
for (PingPong pp : pongs) {
System.out.println(pp);
sum += pp.count;
if (pp.read) read += pp.count;
else write += pp.count;
}
System.out.println(sum);
System.out.println("write = " + write);
System.out.println("read = " + read);
System.out.println("queue.size() = " + queue.size());
System.out.println("balance (must be zero) = " + (write - read - queue.size()));
}
}
It's because this call mutates the queue collection:
String string = queue.poll();
From Queue JavaDoc:
Retrieves and removes the head of this queue, or returns null if this queue is empty.
Read locks are meant to be used in situations where multiple threads can safely read, while writes have to be performed exclusively (no other reads and writes). Because you are using read lock to poll the queue (write operation!), you are effectively allowing multiple threads to modify non thread-safe LinkedList concurrently.
Read-write lock isn't the correct synchronization mechanism in this case.