We Keep Coding

What happens if JMS message is sent without JMSPriority header

I am currently trying to prioritize messages (JMS) going through a complex application.
In the part that concerns me most, users insert the messages into a table in the DB like table INPUT with columns DESTINATION,PRIORITY, MESSAGE. Priority column is not mandatory, both others are mandatory.
The application then takes the information from entries in this table and creates JMS with JMSPriority = PRIORITY header. Body is filled with BODY column and the JMS is then sent to queue specified in DESTINATION.
CODE SNIPPETS:
//pull requests from database and set headers
from(RouteConstants.READ_REQUESTS_FROM_DATABASE) //this is a route formed by SQL
.transacted("PROPAGATION_REQUIRED_JBOSS")
.process(setHeaderProperties)
.to("direct:jms");
//send JMS to destination
from("direct:jms").setBody(simple("${property.MESSAGE}"))
.convertBodyTo(String.class).recipientList(
simple("jms:queue:${property.DESTINATION}?
exchangePattern=InOnly&jmsMessageType=Text&preserveMessageQos=true
&disableReplyTo=true"));
public class SetHeaderProperties implements Processor {
public void process(Exchange exchange) throws Exception {
LinkedCaseInsensitiveMap body = (LinkedCaseInsensitiveMap) exchange.getIn().getBody();
exchange.setProperty("MESSAGE", body.get("MESSAGE").toString());
exchange.setProperty("DESTINATION", body.get("DESTINATION").toString());
Long priority = getPriorityQuery(); //DAO method that returns value of PRIORITY or null if empty
if(priority != null) exchange.setProperty("PRIORITY", priority);
}
//Receive the JMS. Consider this point to be the same that the message was sent to in the second snippet
from("jms:queue:input-msgs").
log(LoggingLevel.DEBUG, CAMEL_LOGGER_NAME, "Received JMSPriority: ${header.JMSPriority}"). //This getter is problematic, see below snippets
process(generalMessageProcessor);
The application behaves as should as long as PRIORITY column is filled. When the value of PRIORITY is null, the getter always returns 4. I understand, that the priority 4 is default, and I am OK with the message being processed as such, but I need to be able to differentiate when priority 4 was set as a fixed value in the database table and therefore requested, or if the priority was not set at all and therefore the program should behave in slightly different route inside the following processor.
Is this at all possible? I would like to avoid changing DDL of the database and also I cannot just fork the program in the SetHeaderProperties processor, as the information would get rewritten in the GeneralMessageProcessor anyway and the setter processor does not have all the necessary classes and fields exposed.
The naive answer I suppose would work is to call the DAO query again whenever I need to check the priority, but that would strain the database and I would like to know if there is more elegant solution to the problem.

Yes, the value 4 is the default JMS priority. Therefore, every message has a priority and there is no such thing like a null priority or no priority at all.
However, one quite simple workaround that does not strain the database would be to set another message header like prioritySetByApplication or whatever name you like. You can then use this header to differ between a default priority and an "explicit" priority of 4.

Hold thread in spring rest request for long-polling

As I wrote in title we need in project notify or execute method of some thread by another. This implementation is part of long polling. In following text describe and show my implementation.
So requirements are that:
UserX send request from client to server (poll action) immediately when he got response from previous. In service is executed spring async method where thread immediately check cache if there are some new data in database. I know that cache is usually used for methods where for specific input is expected specific output. This is not that case, because I use cache to reduce database calls and output of my method is always different. So cache help me store notification if I should check database or not. This checking is running in while loop which end when thread find notification to read database in cache or time expired.
Assume that UserX thread (poll action) is currently in while loop and checking cache.
In that moment UserY (push action) send some data to server, data are stored in database in separated thread, and also in cache is stored userId of recipient.
So when UserX is checking cache he found id of recipient (id of recipient == his id in this case), and then break loop and fetch these data.
So in my implementation I use google guava cache which provide manually write.
private static Cache<Long, Long> cache = CacheBuilder.newBuilder()
.maximumSize(100)
.expireAfterWrite(5, TimeUnit.MINUTES)
.build();
In create method I store id of user which should read these data.
public void create(Data data) {
dataRepository.save(data);
cache.save(data.getRecipient(), null);
System.out.println("SAVED " + userId + " in " + Thread.currentThread().getName());
}
and here is method of polling data:
#Async
public CompletableFuture<List<Data>> pollData(Long previousMessageId, Long userId) throws InterruptedException {
// check db at first, if there are new data no need go to loop and waiting
List<Data> data = findRecent(dataId, userId));
data not found so jump to loop for some time
if (data.size() == 0) {
short c = 0;
while (c < 100) {
// check if some new data added or not, if yes break loop
if (cache.getIfPresent(userId) != null) {
break;
}
c++;
Thread.sleep(1000);
System.out.println("SEQUENCE: " + c + " in " + Thread.currentThread().getName());
}
// check database on the end of loop or after break from loop
data = findRecent(dataId, userId);
}
// clear data for that recipient and return result
cache.clear(userId);
return CompletableFuture.completedFuture(data);
}
After User X got response he send poll request again and whole process is repeated.
Can you tell me if is this application design for long polling in java (spring) is correct or exists some better way? Key point is that when user call poll request, this request should be holded for new data for some time and not response immediately. This solution which I show above works, but question is if it will be works also for many users (1000+). I worry about it because of pausing threads which should make slower another requests when no threads will be available in pool. Thanks in advice for your effort.

Check Web Sockets. Spring supports it from version 4 on wards. It doesn't require client to initiate a polling, instead server pushes the data to client in real time.
Check the below:
https://spring.io/guides/gs/messaging-stomp-websocket/
http://www.baeldung.com/websockets-spring
Note - web sockets open a persistent connection between client and server and thus may result in more resource usage in case of large number of users. So, if you are not looking for real time updates and is fine with some delay then polling might be a better approach. Also, not all browsers support web sockets.
Web Sockets vs Interval Polling
Longpolling vs Websockets
In what situations would AJAX long/short polling be preferred over HTML5 WebSockets?
In your current approach, if you are having a concern with large number of threads running on server for multiple users then you can trigger the polling from front-end every time instead. This way only short lived request threads will be triggered from UI looking for any update in the cache. If there is an update, another call can be made to retrieve the data. However don't hit the server every other second as you are doing otherwise you will have high CPU utilization and user request threads may also suffer. You should do some optimization on your timing.
Instead of hitting the cache after a delay of 1 sec for 100 times, you can apply an intelligent algorithm by analyzing the pattern of cache/DB update over a period of time.
By knowing the pattern, you can trigger the polling in an exponential back off manner to hit the cache when the update is most likely expected. This way you will be hitting the cache less frequently and more accurately.

How to dynamically route message to queues in RabbitMQ

I want to develop solutions that can dynamically route messages to different queues (more than 10000 queues). That's what I have so far:
Exchange with type set to topic. So that I can route messages to different queues based on routing keys.
10000 queues that have routing key as #.%numberOfQueue.#. The %numberOfQueue% is simple numeric value for that queue (but it might be changed for more meaningfull ones).
Producer producing message with routing key like that: 5.10.15.105.10000 which means that message should be routed to queue with keys 5, 10, 15, 105 and 10000 as they comform the patterns of that queues.
That's how it looks like from java client API:
String exchangeName = "rabbit.test.exchange";
String exchangeType = "topic";
boolean exchangeDurable = true;
boolean queueDurable = true;
boolean queueExclusive = false;
boolean queueAutoDelete = false;
Map<String, Object> queueArguments = null;
for (int i = 0; i < numberOfQueues; i++) {
String queueNameIterated = "rabbit.test" + i + ".queue";
channel.exchangeDeclare(exchangeName, exchangeType, exchangeDurable);
channel.queueDeclare(queueNameIterated, queueDurable, queueExclusive, queueAutoDelete, queueArguments);
String routingKey = "#." + i + ".#";
channel.queueBind(queueNameIterated, exchangeName, routingKey);
}
That's how routingKey generated for all messages for queues from 0 to 9998:
private String generateRoutingKey() {
StringBuilder keyBuilder = new StringBuilder();
for (int i = 0; i < numberOfQueues - 2; i++) {
keyBuilder.append(i);
keyBuilder.append('.');
}
String result = keyBuilder.append(numberOfQueues - 2).toString();
LOGGER.info("generated key: {}", result);
return result;
}
Seems good. The problem is that I can't use such long routingKey with channel.basicPublish() method:
Exception in thread "main" java.lang.IllegalArgumentException: Short string too long; utf-8 encoded length = 48884, max = 255.
at com.rabbitmq.client.impl.ValueWriter.writeShortstr(ValueWriter.java:50)
at com.rabbitmq.client.impl.MethodArgumentWriter.writeShortstr(MethodArgumentWriter.java:74)
at com.rabbitmq.client.impl.AMQImpl$Basic$Publish.writeArgumentsTo(AMQImpl.java:2319)
at com.rabbitmq.client.impl.Method.toFrame(Method.java:85)
at com.rabbitmq.client.impl.AMQCommand.transmit(AMQCommand.java:104)
at com.rabbitmq.client.impl.AMQChannel.quiescingTransmit(AMQChannel.java:396)
at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:372)
at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:690)
at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:672)
at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:662)
at com.rabbitmq.client.impl.recovery.AutorecoveringChannel.basicPublish(AutorecoveringChannel.java:192)
I have requirements:
Dynamically choose from producer in which queues produce the messages. It might be just one queue, all queues or 1000 queues.
I have more than 10000 different queues and it might be needed to produce same message to them.
So the questions are:
Can I use such long key? If can - how to do it?
Maybe I can achieve the same goal by different configuration of exchange or queues?
Maybe there are some hash function that can effectivily distinguesh destinations and collapse that in 255 symbols? If so, It should provide way to deal with different publishings (for example how to send to only queues numbered 555 and 8989?)?
Maybe there are some different key strategy that could be used in that way?
How else I can achieve my requirements?

I started using RabbitQM just a short time ago, hope I can help you nonetheless. There can be as many words in the routing key as you like, up to the limit of 255 bytes (as also described in e.g. RabbitMQ Tutorial 5 - Topics). Thus, the topics exchange does not seem to be appropriate for your use case.
Perhaps you can use a headers exchange in this case? According to the concept description:
A headers exchange is designed for routing on multiple attributes that are more easily expressed as message headers than a routing key. Headers exchanges ignore the routing key attribute. Instead, the attributes used for routing are taken from the headers attribute. A message is considered matching if the value of the header equals the value specified upon binding.
See here and here for an example. As I said, I just started with RabbitMQ, therefore, I don't know for sure whether this could be an option for you. If I have time later I try to construct a simple example for you.

How to call Elastic Search for current queue load?

While querying ES extensively, I get
Failed to execute [org.elasticsearch.action.search.SearchRequest#59e634e2] lastShard [true]
org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.search.
action.SearchServiceTransportAction$23#75bd024b
at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:79)
at org.elasticsearch.search.action.SearchServiceTransportAction.execute(SearchServiceTransportAction.java:551)
at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:228)
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:83)
on a quite regular basis.
My plan, now, is to pause the query requests until the queue load is lower than x. You can query the client for its stats
client.admin().cluster().threadPool().stats().iterator();
But since my client is not a data node (I presume that's why), I get queue=0 returned, while the server node throw the above error.
I know why this gets thrown, and I know how to update the setting, but that just postpones this error, and creates others...
How do I ask the cluster nodes what their queue load is?
PS: I'm using the Java Api
What I've tried, without requested result, blank line indicative of another try, unless otherwise specifiied
//Nodes stats
final NodesStatsResponse nodesStatsResponse = client.admin().cluster().prepareNodesStats().execute().actionGet();
final NodeStats nodeStats = nodesStatsResponse.getNodes()[0];
final String nodeId = nodeStats.getNode().getId(); // need this later on
// same as before, but with explicit NodesStatsRequest (with id)
final NodesStatsResponse response = client.admin().cluster().nodesStats(new NodesStatsRequest(nodeId)).actionGet();
final NodeStats[] nodeStats2 = response.getNodes();
for (NodeStats nodeStats3 : nodeStats2) {
Stats stats = nodeStats3.getThreadPool().iterator().next();
}
// Cluster?
final ClusterStatsRequest clusterStatsRequest = new ClusterStatsRequestBuilder(client.admin().cluster()).request();
final ClusterStatsResponse clusterStatsResponse = client.admin().cluster().clusterStats(clusterStatsRequest).actionGet();
final ClusterStatsNodes clusterStatsNodes = clusterStatsResponse.getNodesStats();
// Nodes info?
final NodesInfoResponse infoResponse = client.admin().cluster().nodesInfo(new NodesInfoRequest(nodeId)).actionGet();// here
final NodeInfo[] nodeInfos = infoResponse.getNodes();
for (final NodeInfo nodeInfo : nodeInfos) {
final ThreadPoolInfo info = nodeInfo.getThreadPool();
final Iterator<Info> infoIterator = info.iterator();
while (infoIterator.hasNext()) {
final Info realInfo = infoIterator.next();
SizeValue sizeValue = realInfo.getQueueSize();
// is no == null, then (¿happens?, was expecting a nullpointer, but Thread disappeared)
if (sizeValue == null)
continue;
// normal queue size, no load (oddly found 1000 (expected), and one of 200 in one node?)
final long queueSize = sizeValue.getSingles();
}
}
The issue is that some of the processes need to be called instantly (e.g. user requests), whereas others may wait if the database is too busy (background processes). Preferably, I'd assign a certain amount of the queue to processes that stand on immediate requests, and the other part on background processes (but I haven't seen this option).
Update
It appears, which I didn't expect, that you can get a query overload with a single bulk query, when the total amount of separate searches exceed 1000 (when x shards, or x indices, divide by 1000/x for the number of searches). So bulking,,, not an option, unless you can make a single query. So when you target on 700 search results at once (taking in account the above statement), you'll need to know whether more than 300 items reside in the queue, for then it'll throw stuff.
To sum up:
Assume the load, per call, is the maximum bulkrequest so I cannot combine requests. How, then, can I start pausing requests before elasticsearch starts throwing the above stated exception. So I can pause a part of my application, but not the other? If I know the queue is filled, say, half way, the background process must sleep some time. How do I know the (approximated) queue load?

The way you are trying to look at the queue usage is wrong, as you are not looking at the correct statistics.
Have a look at this piece of code:
final NodesStatsResponse response = client.admin().cluster().prepareNodesStats().setThreadPool(true).execute().actionGet();
final NodeStats[] nodeStats2 = response.getNodes();
for (NodeStats nodeStats3 : nodeStats2) {
ThreadPoolStats stats = nodeStats3.getThreadPool();
if (stats != null)
for (ThreadPoolStats.Stats threadPoolStat : stats) {
System.out.println("node `" + nodeStats3.getNode().getName() + "`" + " has pool `" + threadPoolStat.getName() + "` with current queue size " + threadPoolStat.getQueue());
}
}
First of all you need setThreadPool(true) to be able to get back the thread pool statistics otherwise it will be null.
Secondly, you need ThreadPoolStats not ThreadPoolInfo which is for thread pool settings.
So, it's your second attempt, but incomplete. The 1000 you were seeing was the setting itself (the max queue size), not the actual load.

I'm hoping this is not the answer, source https://www.elastic.co/guide/en/elasticsearch/guide/current/_monitoring_individual_nodes.html#_threadpool_section:
Bulk Rejections
If you are going to encounter queue rejections, it will most likely be
caused by bulk indexing requests. It is easy to send many bulk
requests to Elasticsearch by using concurrent import processes. More
is better, right?
In reality, each cluster has a certain limit at which it can not keep
up with ingestion. Once this threshold is crossed, the queue will
quickly fill up, and new bulks will be rejected.
This is a good thing. Queue rejections are a useful form of back
pressure. They let you know that your cluster is at maximum capacity,
which is much better than sticking data into an in-memory queue.
Increasing the queue size doesn’t increase performance; it just hides
the problem. If your cluster can process only 10,000 docs per second,
it doesn’t matter whether the queue is 100 or 10,000,000—your cluster
can still process only 10,000 docs per second.
The queue simply hides the performance problem and carries a real risk
of data-loss. Anything sitting in a queue is by definition not
processed yet. If the node goes down, all those requests are lost
forever. Furthermore, the queue eats up a lot of memory, which is not
ideal.
It is much better to handle queuing in your application by gracefully
handling the back pressure from a full queue. When you receive bulk
rejections, you should take these steps:
Pause the import thread for 3–5 seconds. Extract the rejected actions
from the bulk response, since it is probable that many of the actions
were successful. The bulk response will tell you which succeeded and
which were rejected. Send a new bulk request with just the rejected
actions. Repeat from step 1 if rejections are encountered again. Using
this procedure, your code naturally adapts to the load of your cluster
and naturally backs off.
Rejections are not errors: they just mean you should try again later.
Particularly this When you receive bulk rejections, you should take these steps I don't like. We should be able to handle oncoming problems on forehand.

java model client server architecture with lock on resources

I have to develop a client server architecture with Java (server side), but I need some advice.
Situation:
a server that exposes an action decrementValue
the server has a variable called Value = integer value
some clients can send a decrementValue request to the server
server, for each requests, do:
-- if value > 0 value = value - 1 and answer the new val to the client
-- if value = 0 answer impossible operation
So, if value = 2 and 3 request arrives at the same time, only 2 request can decrement value.
What is the best solution to do it?
How can I guarantee the exclusive access to value stored in the server, if some request are done in the same time from client to server?
Thank you.

That depends what you mean by best solution. If you just want your program to behave correctly in a concurrent environment, then you should synchronize data that is concurrently modified. That would be a standard (and readable) way to enable exclusive access to shared data.
However, if this is not your question, but rather what would be the most efficient way to do it in Java in theory, (or if this is called in an extremely concurrent context), then my suggestion is as follows:
static final AtomicInteger atomicInteger = new AtomicInteger(initialValue);
...
if (atomicInteger.get() <= 0) {
return "imposibble. value=0";
}
int result = atomicInteger.decrementAndGet();
if (result < 0) {
atomicInteger.incrementAndGet(); // Revert the effect of decrementing
return "imposibble. value=0";
} else {
return "new value=" + result;
}