We Keep Coding

Kafka consumer poll gets messages only the second time

I has consumer with configuration:
public static Consumer<String, TransactionDataAvro> createConsumer() {
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, BOOTSTRAP_SERVERS);
props.put(ConsumerConfig.GROUP_ID_CONFIG, "KafkaExampleAvroConsumer");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, KafkaAvroDeserializer.class.getName());
props.put(KafkaAvroDeserializerConfig.SPECIFIC_AVRO_READER_CONFIG, "true");
props.put(KafkaAvroDeserializerConfig.SCHEMA_REGISTRY_URL_CONFIG, "http://localhost:8000");
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
return new KafkaConsumer<>(props);
}
He subscribed on topic
this.consumer.subscribe(Collections.singletonList("fss-fsstransdata"));
Everytime I try to get messages (this.consumer.poll(Duration.ofMillis(500))), it stably returns only to the second method call
code
log.info("COUNT:" + this.consumer.poll(Duration.ofMillis(1000)).count());
log.info("COUNT:" + this.consumer.poll(Duration.ofMillis(1000)).count());
log.info("COUNT:" + this.consumer.poll(Duration.ofMillis(1000)).count());
log.info("COUNT:" + this.consumer.poll(Duration.ofMillis(1000)).count());
log.info("COUNT:" + this.consumer.poll(Duration.ofMillis(1000)).count());
logs:
| 17:47:40.688 | main | INFO | IaStepsDefinitions | COUNT:0
| 17:47:40.689 | main | INFO | IaStepsDefinitions | COUNT:1
| 17:47:41.690 | main | INFO | IaStepsDefinitions | COUNT:0
| 17:47:42.691 | main | INFO | IaStepsDefinitions | COUNT:0
| 17:47:43.692 | main | INFO | IaStepsDefinitions | COUNT:0
Please explain to me why so

Consumer.poll() has really lots of things going on under the hood except for actually polling the data.
Trying to find the group coordinator
Connecting to this group coordinator node
Starting heart beat thread
Joining the group and being assigned by partitions
Reset offsets to commited or earliest/latest if not found
Fetch the records
And all these steps are bound by the Duration object you pass to the method poll(),
you can see that things are going to work even worse if you have Duration = 1ms.
In my opinion it's misleading and incorrect to put this logic to the poll() method, give the poll do the polling, do the rest in the background threads and/or subscribe method.
When you do the poll, you don't expect the system to do the following:
if (!updateAssignmentMetadataIfNeeded(timer)) {
return ConsumerRecords.empty();
}
The poll is a client facing logic, if it gets 0 records it should mean
the broker is empty.
If you call REST service and get empty the response you know server's empty.
If you call PreparedStatement.execute() you get correct result or exception.
If you call RabbitMQ.basicGet() you get an empty response it means queue is empty.
So long story short, in your case, just increase timeout for the first poll and you should get the updates right away.

Firestore adding 2 documents execution time

Is there any difference between those 2 in terms of execution time?
collectionReference.add(testObject)
.addOnSuccessListener(new OnSuccessListener<DocumentReference>() {
#Override
public void onSuccess(DocumentReference documentReference) {
collectionReference.add(testObject2);
}
})
And
collectionReference.add(testObject);
collectionReference.add(testObject2);
In the first case second adding will be executed after first one is finished, is the same thing happening in second case? Is the second adding queried and is waiting for first to finish, or are they running in parallel?

Yes, there will be a difference between the execution time of these two.
In the first case you're waiting for the first write to be completed on the server, before sending the second write to the server. In a diagram:
Client Server
| |
|---- Send document to write ----->|
| |
| |
|<----- Response from server ------|
|---- Send document to write ----->|
| |
| |
|<----- Response from server ------|
| |
In the second case, the second write is sent to the server right after the first write was sent.
Client Server
| |
|---- Send document to write ----->|
|---- Send document to write ----->|
| |
| |
| |
|<----- Response from server ------|
|<----- Response from server ------|
| |
The difference in performance between these two is the latency of the connection between you and the server.
Note that this is just the theoretical difference, and likely there are many more factors influencing the performance.

python rabbitmqadmin list queues command doesn't obtain messages content although messages exist

I am learning rabbitMq and and now I want to know how to wath queue content.
First of all I want to day that I googled this question and know about command
python rabbitmqadmin list queues
I have written 2 separated applications.
sender:
#Autowired
private AmqpTemplate template;
...
for (int i = 0; i < 100; i++) {
template.convertAndSend("queue1", "message_" + i);
}
receiver:
#RabbitListener(queues = "queue1")
public void listenQueue1(String message, #Header(AmqpHeaders.DELIVERY_TAG) long tag) {
logger.info("Got message:[" + message + "]");
}
If I run these applications together - I see messages on receiver side.
To see messages in the queue I decided to stop receiver and run sender
I run sender
execute python rabbitmqadmin list queues
and see following result:
+-----------------+----------+
| name | messages |
+-----------------+----------+
| query-example-6 | |
| queue1 | |
| queue2 | |
| queue3 | |
| queue4 | |
| queue5 | |
| queue6 | |
| queue7 | |
| queue8 | |
| queue9 | |
+-----------------+----------+
3.Then I run receiver and see logs that receiver accepted messages
Can you clarify reason why I can't see messages in console?
How to see queue messages content.

I am not familiar with rabbitmq.
maybe the message is "unacknowledged"?
e.g. I found my queue has a message:
$ rabbitmqadmin list queues name node messages
+----------------------------+----------------+----------+
| name | node | messages |
+----------------------------+----------------+----------+
| my_queue_name | rabbit#xx-2 | 1 |
but when I run "get" command to show its content, rabbitmq tells me "there's no item"
so, I query it with this command:
$ rabbitmqadmin list queues name node messages messages_ready messages_unacknowledged
+----------------------------+----------------+----------+----------------+-------------------------+
| name | node | messages | messages_ready | messages_unacknowledged |
+----------------------------+----------------+----------+----------------+-------------------------+
| my_queue_name | rabbit#xxxxx-2 | 1 | 0 | 1 |
+----------------------------+----------------+----------+----------------+-------------------------+
I don't know why. just restart the rabbitmq server and everything seems goes fine.

Spring: Initialize multiple connection pools at startup

I want to initialize multiple connection pools at context startup by reading connection parameters from a database table. Basically I want to address following two things
Read connection properties from database not the properties file.
Their are multiple connection pool(rows in db) details.
So my question is How can I iterate over list of rows returned by database in spring-context file and create multiple data-source objects and store them (Lets say in a map) with a unique key?
Databse table structure are somewhat like following:
+--------------+----------------+---------------+
| DBSERVERNAME | DBDRIVERCLASS | DBMINPOOLSIZE |
+--------------+----------------+---------------+
| Server1 | Mysql-Driver | 10 |
| Server2 | Oracle-Driver | 20 |
| Server3 | DB2-Driver | 10 |
+--------------+----------------+---------------+
Let me know if more details are needed. Thanks.

POST data from one page to another

I am trying to implement some data flow as follows
______________ _____________ _____________
| myecom.com | submit | myecom.com | add additional | payment.com |
| product.html |------------>| process.php |----------------->| pay.php |
| myform | form data | | data & submit | |
-------------- ------------- -------------
|
______________ |
| myecom.com |<--------------------------------
| receiver.php | success or failure info
--------------
An ecommerce site receive some info from user
Submit the info to an internal processor
The internal processor processes the data
The processed data along with some additional data is submitted to an external processor
The external processor sends back some success/failure report
My goal is to perform step 4 without informing the user what data is being submitted to external processor. In another word, I want to POST some data to the external processor from the internal processor.
FYI, simply cURL will not do as it does not redirect. The page must redirect to payment.com/pay.php. Also, it need to to be PHP specific, any technology like Java, etc. will do. I am considering storing the data in session using cURL and then javascript redirect. But is there any other ways?
Thanks a lot for your time
Khalid

How about doing it like this instead?
______________ _____________ _____________
| myecom.com | (1) submit | myecom.com | (2) add additional | payment.com |
| product.html |------------>| process.php |-------------------->| pay.php |
| myform | form data | | data & submit | |
-------------- ------------- -------------
| ^ |
| | |
(4) redirect | ---------------------------------
| (3) success or failure info
v
______________
| myecom.com |
| receiver.php |
--------------
Explanation: Your internal processor sends the user data plus the additional data to the external processor. Since it's your server that makes the POST request to the external processor, the user never sees the request and thus cannot see the data that is being sent.
I'm not a PHP programmer, but a quick Google query on "PHP post request" revealed these two links, see if they help you:
How do I send a POST request with PHP?
Three Ways to Make a POST Request from PHP