We Keep Coding

Pulsar client thread balance

I'm trying to implement a Pulsar client with multiple producers that distributes the load among the threads, but regardless the value passed on ioThreads() and on listenerThreads(), it is always overloading the first thread (> 65% cpu while the other threads are completely idle)
I have tried a few things including this "dynamic rebalancing" every hour(last method) but closing it in the middle of the process certainly is not the best approach
This is the relevant code
...
// pulsar client
pulsarClient = PulsarClient.builder() //
.operationTimeout(config.getAppPulsarTimeout(), TimeUnit.SECONDS) //
.ioThreads(config.getAppPulsarClientThreads()) //
.listenerThreads(config.getAppPulsarClientThreads()) //
.serviceUrl(config.getPulsarServiceUrl()).build();
...
private createProducers() {
String strConsumerTopic = this.config.getPulsarTopicInput();
List<Integer> protCasesList = this.config.getEventProtoCaseList();
for (Integer e : protCasesList) {
String topicName = config.getPulsarTopicOutput().concat(String.valueOf(e));
LOG.info("Creating producer for topic: {}", topicName);
Producer<byte[]> protobufProducer = pulsarClient.newProducer().topic(topicName).enableBatching(false)
.blockIfQueueFull(true).compressionType(CompressionType.NONE)
.sendTimeout(config.getPulsarSendTimeout(), TimeUnit.SECONDS)
.maxPendingMessages(config.getPulsarMaxPendingMessages()).create();
this.mapLink.put(strConsumerTopic.concat(String.valueOf(e)), protobufProducer);
}
}
public void closeProducers() {
String strConsumerTopic = this.config.getPulsarTopicInput();
List<Integer> protCasesList = this.config.getEventProtoCaseList();
for (Integer e : protCasesList) {
try {
this.mapLink.get(strConsumerTopic.concat(String.valueOf(e))).close();
LOG.info("{} producer correctly closed...",
this.mapLink.get(strConsumerTopic.concat(String.valueOf(e))).getProducerName());
} catch (PulsarClientException e1) {
LOG.error("Producer: {} not closed cause: {}",
this.mapLink.get(strConsumerTopic.concat(String.valueOf(e))).getProducerName(),
e1.getMessage());
}
}
}
public void rebalancePulsarThreads(boolean firstRun) {
ThreadMXBean threadHandler = ManagementFactory.getThreadMXBean();
ThreadInfo[] threadsInfo = threadHandler.getThreadInfo(threadHandler.getAllThreadIds());
for (ThreadInfo threadInfo : threadsInfo) {
if (threadInfo.getThreadName().contains("pulsar-client-io")) {
// enable cpu time for all threads
threadHandler.setThreadCpuTimeEnabled(true);
// get cpu time for this specific thread
long threadCPUTime = threadHandler.getThreadCpuTime(threadInfo.getThreadId());
int thresholdCPUTime = 65;
if (threadCPUTime > thresholdCPUTime) {
LOG.warn("Pulsar client thread with CPU time greater than {}% - REBALANCING now", thresholdCPUTime);
try {
closeProducers();
} catch (Exception e) {
if (!firstRun) {
// producers will not be available in the first run
// therefore, the logging only happens when it is not the first run
LOG.warn("Unable to close Pulsar client threads on rebalancing: {}", e.getMessage());
}
}
try {
createPulsarProducers();
} catch (Exception e) {
LOG.warn("Unable to create Pulsar client threads on rebalancing: {}", e.getMessage());
}
}
}
}
}

From what you describe, the most likely scenario is that all the topics you're using are served by one single broker.
If that's indeed the case, and avoiding topic load balancing across brokers, it's normal that it's using a single thread because all these producers will be sharing a single, pooled, TCP connection and each connection is assigned to 1 IO thread (listener threads are used for consumer listeners).
If you want to force more threads, you can increase the "Max TCP connection per each broker" setting, in order to use all the configured IO threads.
eg:
PulsarClient client = PulsarClient.builder()
.serviceUrl("pulsar://localhost:6650")
.ioThreads(16)
.connectionsPerBroker(16)
.create();

How to properly use wait() and notify() in Java? (HiveMQ Client) [duplicate]

This question already has answers here:
java.lang.IllegalMonitorStateException: object not locked by thread before wait()?
(3 answers)
Closed 3 years ago.
I've writing a program using HiveMQ Client (an MQTT Open source implementation in Java) that involves using two multithreaded clients. One client is designated as the publisher and the other as the subscriber (I'm aware I could the same client can both publish and subscribe). I'm trying to design a test where the publisher sends 100 messages to the client. The goal is to time how long it takes to send and receive all the messages. I realized if I wanted to time how long it would take for the messages to be received, I would need to have the Subscribing thread wait until the publishing thread was ready to send the message. I decided to use wait() and notify() but I can't seem to implement it correctly. I'm aware that you need to use the same object which I tried to do but I can't get the design correct. I added snipers on code for both of the run methods of the two clients. CommonThread.java isn't actually a thread and I'm not running it but I tried to use it an in between class to be able to wait() and notify() but I'm missing something.
HiveMQ:
https://github.com/hivemq/hivemq-community-edition
https://github.com/hivemq/hivemq-mqtt-client
SubMainThread.java:
public void run() {
// Creates the client object using Blocking API
Mqtt5BlockingClient subscriber = Mqtt5Client.builder()
.identifier(UUID.randomUUID().toString()) // the unique identifier of the MQTT client. The ID is randomly generated between
.serverHost("localhost") // the host name or IP address of the MQTT server. Kept it localhost for testing. localhost is default if not specified.
.serverPort(1883) // specifies the port of the server
.addConnectedListener(context -> ClientConnectionRetreiver.printConnected("Subscriber1")) // prints a string that the client is connected
.addDisconnectedListener(context -> ClientConnectionRetreiver.printDisconnected("Subscriber1")) // prints a string that the client is disconnected
.buildBlocking(); // creates the client builder
subscriber.connect(); // connects the client
ClientConnectionRetreiver.getConnectionInfo(subscriber); // gets connection info
try {
Mqtt5Publishes receivingClient1 = subscriber.publishes(MqttGlobalPublishFilter.ALL); // creates a "publishes" instance thats used to queue incoming messages // .ALL - filters all incoming Publish messages
subscriber.subscribeWith()
.topicFilter(subscriberTopic)
.qos(MqttQos.EXACTLY_ONCE)
.send();
PubSubUtility.printSubscribing("Subscriber1");
System.out.println("Publisher ready to send: " + PubMainThread.readyToSend);
x.threadCondWait(); // <<<<< HOW TO MAKE THIS WORK
System.out.println("Back to the normal execution flow :P");
startTime = System.currentTimeMillis();
System.out.println("Timer started");
for (int i = 1; i <= messageNum; i++) {
Mqtt5Publish receivedMessage = receivingClient1.receive(MESSAGEWAITTIME,TimeUnit.SECONDS).get(); // receives the message using the "publishes" instance waiting up to 5 minutes // .get() returns the object if available or throws a NoSuchElementException
PubSubUtility.convertMessage(receivedMessage); // Converts a Mqtt5Publish instance to string and prints
}
endTime = System.currentTimeMillis();
finalTime = endTime - startTime;
System.out.println( finalTime + PubMainThread.finalTime + " milliseconds");
finalSecTime = TimeUnit.MILLISECONDS.toSeconds(finalTime);
System.out.println(finalSecTime + PubMainThread.finalSecTime);
}
catch (InterruptedException e) { // Catches interruptions in the thread
LOGGER.log(Level.SEVERE, "The thread was interrupted while waiting for a message to be received", e);
}
catch (NoSuchElementException e){
System.out.println("There are no received messages"); // Handles when a publish instance has no messages
}
subscriber.disconnect();
}
PubMainThread.java:
public void run() {
// Creates the client object using Blocking API
Mqtt5BlockingClient publisher = Mqtt5Client.builder()
.identifier(UUID.randomUUID().toString()) // the unique identifier of the MQTT client. The ID is randomly generated between
.serverHost("localhost") // the host name or IP address of the MQTT server. Kept it localhost for testing. localhost is default if not specified.
.serverPort(1883) // specifies the port of the server
.addConnectedListener(context -> ClientConnectionRetreiver.printConnected("Publisher1")) // prints a string that the client is connected
.addDisconnectedListener(context -> ClientConnectionRetreiver.printDisconnected("Publisher1")) // prints a string that the client is disconnected
.buildBlocking(); // creates the client builder
publisher.connect(); // connects the client
ClientConnectionRetreiver.getConnectionInfo(publisher); // gets connection info
PubSubUtility.printPublising("Publisher1");
readyToSend = true;
x.threadCondNotify(); <<<<< HOW TO MAKE THIS WORK
// Think about making the PubClient Thread sleep for a short while so its not too ahead of the client
startTime = System.currentTimeMillis();
for (int i = 1; i <= messageNum; i++) {
publisher.publishWith()
.topic(publisherTopic) // publishes to the specified topic
.qos(MqttQos.EXACTLY_ONCE)
.payload(convertedMessage) // the contents of the message
.send();
}
endTime = System.currentTimeMillis();
finalTime = endTime - startTime;
finalSecTime = TimeUnit.MILLISECONDS.toSeconds(finalTime);
PubSubUtility.printNumOfPublished("Publisher1", messageNum);
publisher.disconnect();
}
public class CommonThread {
private static final Logger LOGGER = Logger.getLogger(SubMainThread.class.getName()); // Creates a logger instance
public synchronized void threadCondNotify() {
notify();
System.out.println("Notified other thread");
}
public synchronized void threadCondWait() {
try {
while (PubMainThread.readyToSend != true) {
System.out.println("Waiting for another thread....");
wait();
}
}
catch (InterruptedException e) {
LOGGER.log(Level.SEVERE, "The thread was interrupted while waiting for another thread", e);
}
}
}

In Sender (rough Java code with some details omitted):
//package statement and imports here
class Sender extends Thread {
public static final Boolean x= new Boolean(true);
public void run() {
//initialize here
synchronized(x) {
x.notify();
}
//send messages here
}
}
In Receiver (start before Sender):
//package statement and imports here
class Receiver extends Thread {
public void run() {
//initialize here
synchronized(Sender.x) {
Sender.x.wait(); //blocks till Sender.x.notify()
}
Date start= new Date();
//receive messages here
Date end= new Date();
int duration_milliseconds= end.getTime()-start.getTime();
}
}
maybe you have to add
try{ /* code here */ } catch (InterruptedException e) {}
Feel free to discuss sense and nonsense of direct use of notify() and wait() especially in Java versions with extended concurrency libraries...

Send 35000 jms messages per minute

We have a spring boot application for performing load test on one other component. We need to send max 35000 JMS messages per minute and for that reason I am using a scheduler for running a task every one minute.
The problem is when I keep the intensity low it manages to send the messages within the specified time interval (one minute). But when the intensity is high it takes more than 1 minute to send the chunk of messages. Any suggestions on the below implementation?
Scheduler class
#Component
public class MessageScheduler {
private final Logger log = LoggerFactory.getLogger(getClass());
private static ScheduledExecutorService executorService = Executors.newScheduledThreadPool(16);
private final static int TIME_PERIOD = ConfigFactory.getConfig().getInt("messages.period").orElse(60000);
#Autowired
JmsSender sender;
public void startScheduler() {
Runnable runnableTask = sender::sendMessagesChunk;
executorService.scheduleAtFixedRate(runnableTask, 0, TIME_PERIOD,
TimeUnit.MILLISECONDS);
}
}
Class for sending the messages
#Component
public class JmsSender {
#Autowired
TrackingManager manager;
private final Logger log = LoggerFactory.getLogger(getClass());
private final static int TOTAL_MESSAGES = ConfigFactory.getConfig().getInt("total.tracking.messages").orElse(10);
private final static int TIME_PERIOD = ConfigFactory.getConfig().getInt("messages.period").orElse(60000);
private static int failedPerPeriod=0;
private static int totalFailed=0;
private static int totalMessageCounter=0;
public void sendMessagesChunk() {
log.info("Started at: {}", Instant.now());
log.info("Sending messages with intensity {} messages/minute", TOTAL_MESSAGES);
for (int i=0; i<TOTAL_MESSAGES; i++) {
try {
long start = System.currentTimeMillis();
MessageDTO msg = manager.createMessage();
send(msg);
long stop = System.currentTimeMillis();
if (timeOfDelay(stop-start)>=0L) {
Thread.sleep(timeOfDelay(stop-start));
}
} catch (Exception e) {
log.info("Error : " + e.getMessage());
failedPerPeriod++;
}
}
totalMessageCounter += TOTAL_MESSAGES;
totalFailed += failedPerPeriod;
log.info("Finished at: {}", Instant.now());
log.info("Success rate(of last minute): {} %, Succeeded: {}, Failed: {}, Success rate(in total): {} %, Succeeded: {}, Failed: {}"
,getSuccessRatePerPeriod(), getSuccededPerPeriod(), failedPerPeriod,
getTotalSuccessRate(), getTotalSucceded(), totalFailed);
failedPerPeriod =0;
}
private long timeOfDelay(Long elapsedTime){
return (TIME_PERIOD / TOTAL_MESSAGES) - elapsedTime;
}
private int getSuccededPerPeriod(){
return TOTAL_MESSAGES - failedPerPeriod;
}
private int getTotalSucceded(){
return totalMessageCounter - totalFailed;
}
private double getSuccessRatePerPeriod(){
return getSuccededPerPeriod()*100D / TOTAL_MESSAGES;
}
private double getTotalSuccessRate(){
return getTotalSucceded()*100D / totalMessageCounter;
}
private void send(MessageDTO messageDTO) throws Exception {
requestContextInitializator();
JmsClient client = JmsClientBuilder.newClient(UriScheme.JmsType.AMQ);
client.target(new URI("activemq:queue:" + messageDTO.getDestination()))
.msgTypeVersion(messageDTO.getMsgType(), messageDTO.getVersion())
.header(Header.MSG_VERSION, messageDTO.getVersion())
.header(Header.MSG_TYPE, messageDTO.getMsgType())
.header(Header.TRACKING_ID, UUID.randomUUID().toString())
.header(Header.CLIENT_ID, "TrackingJmsClient")
.post(messageDTO.getPayload());
}

You should solve two problems:
total send operation time must be under max time.
messages should be sent not as fast as possible, instead, they should be sent uniformly along all available time.
Obviously, if your send method is too slow, the max time will be exceeded.
The faster way to send messages is to use some sort of bulk operation. Never mind if your MQ API don't support bulk operation, you can't use it! because of the second restriction ("uniformly").
You can send messages asynchronously, but if your MQ API create threads for that instead of "non-blocking" async, you could have memory problems.
Using javax.jms.MessageProducer.send you can send messages asynchronously, but a new one thread will be created for each one (a lot of memory and server threads will be created).
Another speedup could be create only one JMS client (your send method).
To achieve the second requirement, you should fix your timeOfDelay function, it's wrong. Really, you should take in account the probability distribution of the send function to estimate the proper value but, you can simply do:
long accTime = 0L;
for (int i=0; i<TOTAL_MESSAGES; i++) {
try {
long start = System.currentTimeMillis();
MessageDTO msg = manager.createMessage();
send(msg);
long stop = System.currentTimeMillis();
accTime += stop - start;
if(accTime < TIME_PERIOD)
Thread.sleep((TIME_PERIOD - accTime) / (TOTAL_MESSAGES - i));
} catch (Exception e) {
log.info("Error : " + e.getMessage());
failedPerPeriod++;
}
}

35000 msg/min is a notch below 600 msg/sec. That is not considered "a lot" and should be relatively easy goal to clear. Primary idea is to "reuse" all heavy weight JMS objects: connection, session and destination. Single thread should be enough.
ConnectionFactory connFactory = .... // initialize connection factory
#Cleanup Connection conn = connFactory.createConnection();
#Cleanup Session session = conn.createSession(true, Session.SESSION_TRANSACTED);
Queue q = session.createQueue("example_destiation");
#Cleanup MessageProducer producer = session.createProducer(q);
for (String payload: messagesToSend) {
TextMessage message = session.createTextMessage(payload);
producer.send(msg);
session.commit();
}
Additional speedups are possible by:
commiting every n-th message
by using faster ACKNOWLEDGE modes
by using non-persistent messages
by using destination object created outside session
sending messages asynchronously
Example of NON_PERSISTENT, ACKOWLEDGE, ASYNC delivery:
#Cleanup Connection conn = connFactory.createConnection();
#Cleanup Session session = conn.createSession(false, Session.DUPS_OK_ACKNOWLEDGE);
Queue q = session.createQueue("example_destiation");
#Cleanup MessageProducer producer = session.createProducer(q);
producer.setDeliveryMode(DeliveryMode.NON_PERSISTENT);
producer.setAsync(new ExmpleSendListener());
for (String payload: messagesToSend) {
TextMessage message = session.createTextMessage(payload);
producer.send(msg);
}

Alert if Kafka nodes are down

I have been exposed a Kafka nodes and a topic-name. My web server receives a lot of http request data which I need to process them and then push them to kafka. Sometimes, if the kafka nodes are down, then my server still keeps pumping the data, which results in blowing up my memory and my server gets down.
I want is to stop publishing the data if the Kafka is down. My Java sample code is as follows:
static Producer producer;
Produce() {
Properties properties = new Properties();
properties.put("request.required.acks","1");
properties.put("bootstrap.servers","localhost:9092,localhost:9093,localhost:9094");
properties.put("enabled","true");
properties.put("value.serializer","org.apache.kafka.common.serialization.StringSerializer");
properties.put("kafka-topic","pixel-server");
properties.put("batch.size","1000");
properties.put("producer.type","async");
properties.put("key.serializer","org.apache.kafka.common.serialization.StringSerializer");
producer = new KafkaProducer<String, String>(properties);
}
public static void main(String[] args) {
Produce produce = new Produce();
produce.send(producer, "pixel-server", "Some time");
}
//This method is called lot of times
public void send(Producer<String, String> producer, String topic, String data) {
ProducerRecord<String, String> producerRecord = new ProducerRecord<>(topic, data);
Future<RecordMetadata> response = producer.send(producerRecord, (metadata, exception) -> {
if (null != exception) {
exception.printStackTrace();
} else {
System.out.println("Done");
}
});
I have just abstracted out some sample code. The send method is called numerous times. I just want to prevent send any message if the kafka is down. What is the efficient way to tackle this situation.

If I were you I'll try to implement a circuit breaker. When you hit a reasonable amount of failures while sending your records, circuit breaks and provides some fallback behavior. Once some condition is met (e.g.: some time passed) the circuit close and you'll send records again. Also vertx.io comes with it's own solution.

How to check that KafkaConsumer still has assigned partitions without reading more data with poll()

In my KafkaConsumer app I want to read a batch of messages with poll() and process them. But processing may fail. In this case I want to retry until I succeed but only retry if consumer still owns partitions. I don't want to constantly call poll() because I don't want to read more data.
This is a code snippet:
consumer = new KafkaConsumer<>(consumerConfig);
try {
consumer.subscribe(config.topics() /** Callback does not work as I do not call poll in between */ );
while (true) {
ConsumerRecords<byte[], Value> values = consumer.poll(10000);
while (/* I am still owner of partitions */) {
try {
process(values);
} catch (Exception e) {
log.error("I dont care, just retry while I own the partitions", e)
}
}
}
} catch (WakeupException e) {
// shutting down
} finally {
consumer.close();
}

There is a callback method that tells you when your consumers partition assignments are about to be revoked. Keep processing message unless you get an onPartitionRevoked() event.
https://kafka.apache.org/0110/javadoc/org/apache/kafka/clients/consumer/ConsumerRebalanceListener.html#onPartitionsRevoked(java.util.Collection)

What about simply calling assignment() ?
http://kafka.apache.org/0110/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#assignment()

I came to a conclusion that it is impossible to call poll() without reading messages with current kafka consumer 10.2.x. However, it is possible to update offset after a processing failure. So I update offset as if the messages were never read
while (!stopped) {
ConsumerRecords<byte[], Value> values = consumer.poll(timeout);
try {
process(values);
} catch (Exception e) {
rewind(records);
// Ensure a delay after errors to let dependencies recover
Thread.sleep(delay);
}
}
and rewind method is
private void rewind(ConsumerRecords<byte[], Value> records) {
records.partitions().forEach(partition -> {
long offset = records.records(partition).get(0).offset();
consumer.seek(partition, offset);
});
}
It solves the initial problem