We Keep Coding

Paginating JMS Queue

In my Java EE application I use JMS to store some messages. I want to display these messages in a JSF paginated table. How can I get the messages from the queue in batches? For the moment I'm using something like this, but it's not very nice because I need to loop through many messages.
Can this be achieved? I'm using JBoss with HornetQ.
browser = session.createBrowser(queue);
List<Message> messagesToReturn = new ArrayList<>();
final Enumeration<ObjectMessage> messages = browser.getEnumeration();
int messagesSoFar = 0;
int count = 0;
while(messages.hasMoreElements()) {
ObjectMessage message = messages.nextElement();
if (count >= offset) {
messagesToReturn.add(new CGSQueueMessage(message));
messagesSoFar += 1;
}
if (messagesSoFar == maxSelect) {
break;
}
count += 1;
}
return messagesToReturn;

There are no methods in the JMS API to get messages from the queue in batches for a paginated use-case like yours.
You could read all the messages from the queue browser into your own data structure and paginate using that.
If there were too many messages to fit them all into an in-memory data structure at once then you could read as many of them as you could reasonably fit into memory (which presumably will be more than what the user sees on any given page) and that would serve as your own kind of application-level page which you could use for serving up the user-level pages. That would reduced the number of times you need to loop through the queue browser.
Aside from that you could dump all the messages from the queue browser into a temporary, random-access file and pull the results from that.
All that said, I think your use-case doesn't ultimately fit with a messaging API like JMS. In my view I think you'd be better suited using something like a database which could easily support this use-case.

How to choose a Key and Offset for a Kafka Producer

I'm following here.While following the code. I came up with two Questions
Is the Key and offset were the same?
According to Google,
Offset: A Kafka topic receives messages across a distributed set of
partitions where they are stored. Each partition maintains the
messages it has received in a sequential order where they are
identified by an offset, also known as a position.
Seems both are very similar for me. Since offset maintain a unique message in the partition: Producers send records to a partition based on the record’s key
What is the best way to choose the Key/Offset for a producer?
For an instance the example which I provided above, they have chosen the timestamp as the Key and offset.
Is this the always the best recommendation?
class IRCMessageListener extends IRCEventAdapter {
#Override
public void onPrivmsg(String channel, IRCUser u, String msg) {
IRCMessage event = new IRCMessage(channel, u, msg);
//FIXME kafka round robin default partitioner seems to always publish to partition 0 only (?)
long ts = event.getInt64("timestamp");
Map<String, ?> srcOffset = Collections.singletonMap(TIMESTAMP_FIELD, ts);
Map<String, ?> srcPartition = Collections.singletonMap(CHANNEL_FIELD, channel);
SourceRecord record = new SourceRecord(srcPartition, srcOffset, topic, KEY_SCHEMA, ts, IRCMessage.SCHEMA, event);
queue.offer(record);
}
Because I'm actually trying to create a custom Kafka connector to get the data from 3rd Party WebSocket API. The API sends real-time data stream messages for a given Key value. So I thought of using that Key for my PartitionKey as well as Offset. But need to make sure I'm right about my thought.

Key is an optional metadata, that can be sent with a Kafka message, and by default, it is used to route message to a specific partition. E.g. if you're sending a message m with key as k, to a topic mytopic that has p partitions, then m goes to the partition Hash(k) % p in mytopic. It has no connection to the offset of a partition whatsoever. Offsets are used by consumers to keep track of the position of last read message in a partition. In your case, if the timestamp is fairly randomly distributed, then it's fine, else you might be causing partition imbalance while using it as key.

These are some basic differences :
Offset : maintained by kafka to keep a track of the records consumed to avoid loss of records and duplicate records while consuming.
Key : it is specific to input events,if it is not available then by default it is mentioned as null,this is useful while writing records to HDFS with default partition-er using kafka connect.every message can have a single key or many messages can have similar key.

GETBULK SNMP4J Request

I'm using snmp4j to try and perform SNMP functions against a remote agent. Due to a number of limitations out of our control I need to perform a GETBULK to obtain a large table in a short space of time.
My current implementation:
public Map<String, String> doGetBulk(#NotNull VariableBinding... vbs)
throws IOException {
Map<String, String> result = new HashMap<>();
Snmp snmp = null;
try {
// Create TransportMapping and Listen
TransportMapping transport = new DefaultUdpTransportMapping();
snmp = new Snmp(transport);
transport.listen();
PDU pdu = new PDU();
pdu.setType(PDU.GETBULK);
pdu.setMaxRepetitions(200);
pdu.setNonRepeaters(0);
pdu.addAll(vbs);
ResponseEvent responseEvent = snmp.send(pdu, this.target);
PDU response = responseEvent.getResponse();
// Process Agent Response
if (response != null) {
for(VariableBinding vb : response.getVariableBindings()) {
result.put("." + vb.getOid().toString(), vb.getVariable().toString());
}
} else {
LOG.error("Error: Agent Timeout... ");
}
} catch (NullPointerException ignore) {
// The variable table is null
} finally {
if (snmp != null) snmp.close();
}
return result;
}
However, this only ever returns 100 results when I know there are 5000+. I know I cant exceed the PDU size so I have so problem with the response being truncated into blocks of 100 but I cant work out how I can get a handle to cascade the request to get the next 100 entries.

It is bad practice to use MaxRepetitions > 100 due to TCP/IP packet fragmentation and the nature of UDP that does not guarantee the order of packets. So most SNMP frameworks and agents have such built-in limit.

All details are already there in the RFC document,
https://www.rfc-editor.org/rfc/rfc1905
4.2.3 tells how the agent side should handle GET BULK requests, and
While the maximum number of variable bindings in the Response-PDU is
bounded by N + (M * R), the response may be generated with a lesser
number of variable bindings (possibly zero) for either of three
reasons.
(1) If the size of the message encapsulating the Response-PDU
containing the requested number of variable bindings would be
greater than either a local constraint or the maximum message size
of the originator, then the response is generated with a lesser
number of variable bindings. This lesser number is the ordered set
of variable bindings with some of the variable bindings at the end
of the set removed, such that the size of the message encapsulating
the Response-PDU is approximately equal to but no greater than
either a local constraint or the maximum message size of the
originator. Note that the number of variable bindings removed has
no relationship to the values of N, M, or R.
(2) The response may also be generated with a lesser number of
variable
bindings if for some value of iteration i, such that i is greater
than zero and less than or equal to M, that all of the generated
variable bindings have the value field set to the `endOfMibView'.
In this case, the variable bindings may be truncated after the (N +
(i * R))-th variable binding.
(3) In the event that the processing of a request with many
repetitions
requires a significantly greater amount of processing time than a
normal request, then an agent may terminate the request with less
than the full number of repetitions, providing at least one
repetition is completed.
About how to do a series of proper GET BULK operations to query all data you want, you can refer to section 4.2.3.1 for an example.

You've set maximum repetition count to 200, that is the server may send you at most 200 rows. So on the one hand, you'll never get more than 200 rows (and least of all 5000 or more). On the other hand, the server may decide to send you less rows, it's practically the server's choice; you tell him what you're able to process.
Usually you request 10-50 rows at max. (BTW: There are many servers out there with buggy SNMP implementations and the higher you set max-repetitions, the higher is the chance you get nothing at all.)
So you have to request row set by row set. Since you probably don't want to implement that yourself I'd recommend to use the TableUtils class. Just start with getTable().

Receiving message synchronically when I use multiple consumer

I am new in Active MQ. There was single consumer for my queue. I was reading message synchronically and unique string is in last of every message. Some messages was large so consumer receive those message in two part and through unique string, consumer make complete message.
StringBuilder snapshotUpdateString = new StringBuilder("");
while(true) {
Message messageData = consumer.receive();
TextMessage textMessage = (TextMessage) messageData;
String receivedMessage=textMessage.getText();
if (receivedMessage.contains("" + (char) PushPortContants.EOF)) {
snapshotUpdateString.append(receivedMessage);
String snapshot = snapshotUpdateString.substring(1, snapshotUpdateString.length() - 1);
parseSaveSsUpdateData.parseAndSave(snapshot);
snapshotUpdateString = new StringBuilder("");
}
else snapshotUpdateString.append(receivedMessage);
}
In queue, messages was increasing high so I add one more consumer. Now consumer is two.
If message is large then single message is received in two part() to different consumer. How can make message to complete message.
In multiple consumer, I used listener. Can I receive message synchronically when i use multiple consumer?

You want to use Message Groups for this. This can be a bit tricky with multiple consumers, and requires a bit more coupling of the producer as it has to be able to set the appropriate headers.
In any case, the producer would need to mark the JMSXGroupId and JMSXGroupSeq to indicate messages and order for the particular group.
It would also be worth reading the link at the bottom of the page I linked above regarding selectors to help you choose the best method.

Netty + ProtoBuffer: A few communication messages for one connection

While reading the Netty tutorial, I've found a simple description of how to integrate Netty and Google Protocol Buffers. I've started to investigate its example (because there is no more information in the documentation) and written a simple application like the example local time application. But this example is using static initialization in PipeFactory Class, e.g.:
import org.jboss.netty.channel.ChannelPipeline;
import org.jboss.netty.channel.ChannelPipelineFactory;
import org.jboss.netty.handler.codec.protobuf.ProtobufDecoder;
import org.jboss.netty.handler.codec.protobuf.ProtobufEncoder;
import org.jboss.netty.handler.codec.protobuf.ProtobufVarint32FrameDecoder;
import org.jboss.netty.handler.codec.protobuf.ProtobufVarint32LengthFieldPrepender;
import static org.jboss.netty.channel.Channels.pipeline;
/**
* #author sergiizagriichuk
*/
class ProtoCommunicationClientPipeFactory implements ChannelPipelineFactory {
public ChannelPipeline getPipeline() throws Exception {
ChannelPipeline p = pipeline();
p.addLast("frameDecoder", new ProtobufVarint32FrameDecoder());
p.addLast("protobufDecoder", new ProtobufDecoder(Communication.DataMessage.getDefaultInstance()));
p.addLast("frameEncoder", new ProtobufVarint32LengthFieldPrepender());
p.addLast("protobufEncoder", new ProtobufEncoder());
p.addLast("handler", new ProtoCommunicationClientHandler());
return p;
}
}
(Please take a look at line p.addLast("protobufDecoder", new ProtobufDecoder(Communication.DataMessage.getDefaultInstance()));)
and just one factory can be created (as I understand) for ClientBootstrap class, I mean bootstrap.setPipelineFactory() method. So, in this situation I can use ONE message to send to server and ONE message to receive from server and it is bad for me, and I think not just for me :( How can I use different messages to and from for just one connection?
Perhaps I can create a few protobufDecoder like this
p.addLast("protobufDecoder", new ProtobufDecoder(Communication.DataMessage.getDefaultInstance()));
p.addLast("protobufDecoder", new ProtobufDecoder(Communication.TestMessage.getDefaultInstance()));
p.addLast("protobufDecoder", new ProtobufDecoder(Communication.SrcMessage.getDefaultInstance()));
or other techniques?
Thanks a lot.

I've found thread of author of netty in google groups and understood that I have to change my architecture or write my own decoder as I wrote above, So, Start to think what way will be easy and better.

If you are going to write your own codecs anyway, you might want to look at implementing the Externalizable interface for custom data objects.
Serializable is low-effort, but worst performance (serializes everything).
Protobuf is a good trade-off between effort and performance (requires .proto maintenance)
Externalizable is high effort, but best performance (custom minimal codecs)
If you already know your project will have to scale like a mountain goat, you may have to go the hard road. Protobuf is not a silver bullet.

Theoretically this can be done by modifying the pipeline for each incoming message to suit the incoming message. Take a look at the port unification example in Netty.
Sequence would be:
1) In frame decoder or another "DecoderMappingDecoder" you check the message type of the incoming message
2) Modify the pipeline dynamically as shown in the example
But why not use different connections and follow this sequence:
1) Add other decoders in pipeline based on the incoming message only once.
2) Add the same instance of channel upstream handler as the last handler in the pipeline, this way all messages get routed to the same instance, which is almost like having a single connection.

the problem is that there is no way to distinct two different protobuf messages from each other in binary format. But there is a way to solve it within the protobuf file:
message AnyMessage {
message DataMessage { [...] }
optional DataMessage dataMessage = 1;
message TestMessage { [...] }
optional TestMessage testMessage = 2;
message SrcMessage { [...] }
optional SrcMessage srcMessage = 3;
}
optional fields that are not set produce no overhead. Additionally you can add an Enum, but it is just a bonus.

The issue is not quite a Netty limitation or encoder/decoder limitation. The problem is that Google Protocol Buffers are offering just a way to serialize/deserialize objects, but is not provide a protocol. They have some kind of RPC implementation as part of standard distribution, but if you'll try to implement their RPC protocol then you'll end up with 3 layers of indirection.
What I have done in one of the project, was to define a message that is basically an union of messages. This message contains one field that is Type and another field that is the actual message. You'll still end-up with 2 indirection layers, but not 3. In this way the example from Netty will work for you, but as was mention in a previous post, you have to put more logic in the business logic handler.

You can use message tunneling to send various types of messages as payload in a single message.
Hope that helps

After long research and suffering...
I came up with idea of using composition of messages into one wrapper message. Inside that message I use oneof key to limit the number of allowed objects to the only one. Checkout the example:
message OneMessage {
MessageType messageType = 1;
oneof messageBody {
Event event = 2;
Request request = 3;
Response response = 4;
}
string messageCode = 5; //unique message code
int64 timestamp = 6; //server time
}