I have a java server (jdk 1.6) that pulls in data from a tibco ems topic through one connection. I am starting to see a back log on being able to pull message from tibco. How well does tibco scale if i break the data into multiple topics and each topic gets a separate connection (same JMS server) to my server.
Also I can't break up the server into multiple servers.
Tibco is by principle made to be used in an enterprise with MASSIVE amounts of data transiting the message server. Instead of pulling messages using the server, you can also try to use tibco the other way around: Publishing the messages and using your server to subscribe to the publisher. You can use the observer/observable design pattern as an analogy. I wouldn't worry about the scaling.
Related
I am working on application that writes to Kafka queue which is read by other application. When I am unable to send message on Kafka due to network or other reason, I need to write messages during Kafka down time to other place e.g Oracle or local file system, so that I don't loose messages generated during down time.Problem with oracle or other DB is it too can go down. Is there any recommendations about how could I achieve fail safe during Kafka down time.
Number of messages generated are approx 20-25 million per day. For messages stored during downtime I am planning to have batch job to re send them to destination application once target application is up again.
Thank you
You can push those messages into a cloud based messaging service like SQS. It supports around 3K messages per second.
There is also a connector that allows you to push back the messages into Kafka directly, with no other headaches.
If you can't export the data out of your local network, then maybe a cluster of RabbitMQ instances may help, although it wouldn't be a plug & play solution.
I want to run a Java based message broker that will route messages to web clients. Web client connections are handled on our server using our custom Java websocket code, which authenticates users against the user database.
I think my server side websocket handler code would connect to ActiveMQ and perform subscription management via AQMP.
I have a specific requirement however:
route messages for a topic specifically to one or more web clients
Note that I don't need to retain messages if a client is not connected. Messages are being used to inform the web client applications of actions they need to take.
I'm considering ActiveMQ but I was hoping people with experience of the product could clarify if it supports this requirement?
If ActiveMQ isn't the best option, could you recommend something else?
Thanks
Yes, ActiveMQ is a great choice for this.
As far as specific approach goes, it depends on your data model and message flow.
You have several options, including:
Produce and consume to a topic-per-client
a. Messages for Client ABC go to topic://CLIENTS.ABC, for Client XYZ go to topic://CLIENTS.XYZ, and the subscribers connect accordingly.
Produce a message with a header and use a consumer-side selector (aka 'filters' in AMQP) to filter messages on a per-client basis. (abc client subscribes to-- ClientId = ABC, xyz client subscribe to-- ClientId = XYZ)
When using WebSockets, you might also look to STOMP which is text-based protocol. (Just depends on your programming language and available libraries that you had in mind)
I understand JMS as depicted by the following diagram:
(source: techhive.com)
Is there any way for me to access the underlying database using JMS or some other thing? Further, the JDBC connections that the JMS server maintains, can I add new connections in it so as to access other databases also and do CRUD operations on them? If yes, how?
Where did you get this from?
Normally JMS is used to send messages to queue (or topics). You have message producers that push messages in the queue and message consumers consume them and process it.
In your exemple it seems that you have multiple queues. One for the messages that need to be processed, and one for each client to retrieve the result the processing of its messages.
With JMS Server you don't necessarily have a database behind. Everything can stay in memory, or can be written to files. You will need database server behind only if you configure your JMS server to be persistent (and to assure that even if server/application crash your messages won't be lost). But in that case you will never have to interact with the database. Only the JMS server will and you will interact with the JMS server sending and consuming messages.
How do I use Tibco Certified Messaging mode? Does WebsphereMQ provide the same functionality? Does the JMS specification define this functionality?
I am guessing that you are looking for a message transport that offers guaranteed delivery. E.g. if the recipient of the message is not available, the message will be delivered when the recipient comes back online again?
TIBCO Rendezvous has a mode called Rendezvous Certified Messaging (RVCM) that stores messages on disk until all recipients has acknowledged it. Both TIBCO EMS and Websphere MQ allows configuring persistent destinations with similar functionality. For details on how to configure and use these please refer to the documentation provided by either vendor for your particular language.
JMS, which is a specification and not an implementation as the above, states that when a message is marked as persistent, the JMS provider must "take extra care to insure the message is not lost in transit due to a JMS provider failure". Please note that both TIBCO EMS and Websphere MQ can be accessed using the JMS API.
The equivalent of TIBCO RVCM in the JMS/EMS world is 'PERSISTENT' messaging to a durable subscription, more specifcially: use publish(..,DeliverMode.PERSISTENT,..) on the publisher and Session.createDurableSubscriber(..) on the consuming side.
That way you will ensure that every message published to this topic will end up at the subscriber, even if the subscriber is down for a while and that all messages are stored on disk before delivery, so it will survived tibemsd downtimes.
But be careful: unlike RVCM, where messages were stored on the publisher, with EMS messages are stored on the daemon (tibemsd), so one subscriber that is not picking up messages will let the memory and disk of the tibemsd grow and grow. Make sure you configure max_msg_memory and msg_swapping and TEST this !
with RVCM one rouge subscriber might influence only the publishers that are actually publishing to it, with EMS one rouge subscriber can influence memory and performance of the whole system.
I am considering an architecture where I have clients that are intermittently connected to a network. I would like to store messages created on these clients in a JMS queue when the network is not available and have these forwarded to a central message broker when the clients are on the network. (The user has control over the network, e.g. dialing in, so it's not an intermittent connection like with a mobile phone.)
Are there any JMS implementations that provide this feature?
You can embed an activeMQ broker into your application
http://activemq.apache.org/how-do-i-embed-a-broker-inside-a-connection.html
Then, I suppose (did not test) that you could use ActiveMQ features which allow you to dispatch messages accross a net of brokers, using the discovery of brokers feature,
http://activemq.apache.org/clustering.html
or simply by adding a queue consumer server side, then dispatching through other brokers through this consumer.
Hope it helps.
The Glassfish Open Message Queue can be embedded (or run stand-alone) in version 4.4 (Support the ability for a broker to run "in process" with any client.). It is very light-weight, and will support other client languages over the STOMP protocol in version 4.4 - besides Java and C. - https://mq.dev.java.net/4.4.html