I have web service for reading and updating data and using spring, spring JDBC for DB access. My controller can be accessed by many channels like desktop, mobile etc. If data is updated using desktop, then same should reflect in mobile immediately. Current approach is calling service continuously to get updated data. I feel that it is worst approach and causing DB performance issue as well.
Is there a possible way such that GET service is called only when there is DB update by other channel instead of continuous polling ? What is best approach for this and how to implement it ?
Continuously calling the service seems like a really bad idea. I think you need a database trigger that fires when rows are inserted/updated/deleted. It could POST something to a Web Service or put something on a Message Queue.
Good luck.
I can think of an architectural answer to the problem. Use a messaging solution between the spring controller and the database. Infact you will need two queues
EventSink queue -
Publish all data change requests originating from any of the channels to this queue.The subscriber will be the service managing the database update aka dbservice .
EventBroadcast queue -
Publish the changed data post db update to this queue. Ideally the dbservice should handle this publish within the same transaction as db update. All channels can subscribe to this queue to receive the update.
The merits to consider this approach would involve
Pros - this approach involves no database services so both performance and de-coupling from database changes.
Cons - Increased complexity
Continuously polling is not as bad as you might imagine. Pushing messages to clients without them making a request requires web-sockets or of the like to achieve this. If it is not a large repose from the server, and is not too often, as in many millions and millions of requests then I would leave it for now.
If however this is a large amount of bandwidth we are talking about it then you wouldn't want to be polling. You would probably want to look at a subscriber type pattern whereby clients would subscribe to be notified when a specific event occurs. When this event occurs the server would then send a message to the clients.
Detecting this event shouldn't require polling a database. The modification to the database should trigger the event. You might do this with point-cuts in Spring if you are into that sort of thing.
Related
I'm not very familiar with JMS so, I can't understand whether I should consider it to use in my case.
I have 3 servers (running on tomcat) which are going to send some notifications to another server (call it PrincipalServer) when some event occured on them. The PrincipalServer is running on tomcat too. When the notifications from one of those 3 servers reach the PrincipalServer it need to handle it in some way, depending on the message (For instance, persist some data in a database). Approximately, the rate of the notification would be 500k-1M a day.
So, should I consider some JMS implementation like ActiveMQ?
It depends on a number of factors, but it may provide a benefit in your case. The main benefit provided by JMS is the ability to reliably queue work that can be done later. There are three key reasons in my mind for using JMS over a web service, rest or ejb call. These are:
The client should return prior to this work being processed. If this work has to be done before returning to the client then don't use JMS, trying to build a synchronous invoke model over JMS while possible is choosing a hammer when you have a screw.
The clients may process bursts of work that the back end can't keep up with. In this case JMS will store the messages until the back end can process the work. Note that you still need to average the number of messages on the Queue to be zero, you can't add messages forever.
The back end may go down independently of the front end. In this case the JMS provider will store the messages until the backend comes back up to process the work.
I have read some articles about microservices architecture, but no one takes the topic of transaction. All that they says that this is hard to do it. Maybe someone can describe how to handle this?
But not from domain side, but from technology side. Lets say that we have business case where we need to invoke two different services and both of them make some changes on database. But how to rollback if some error occurs on the second one?
Who knows some libraries or design patter for this problem?
I may not be the ultimate expert in this, but I'm sure you're heading towards the Distributed Transactions. In order to have them running, all the application service components need a common shared transaction id, and you have to make sure that every component is informed about the state of the transaction. It is asynchronous, so you'll require substantial prog skills.
Here are distributed transactions mentioned or discussed:
https://en.wikipedia.org/wiki/Distributed_transaction
http://contino.co.uk/microservices-not-a-free-lunch/
http://martinfowler.com/articles/microservices.html
It would seem people try to avoid it as it is difficult. Maybe that's why you don't find much about.
Hope this helps a step forward :-)
The best design is having isolated services: each service just do its work within its own transaction and your workflow expects failures on the single service.
If you really need to commit only if all the services are called without errors you should create an higher level service that perform those calls inside an external transaction.
The first raw thing which came to my mind after reading this question is to create every add api with a delete api with ,lets say, an extra boolean flag delFlag.
boolean flag delFlag;
For POST, it will be 0. For DELETE, it will be 1.
Now you maintain a Transaction Manager which is a super service to all your micro services.
In this service maintain the calling queue of all the services and the APIs. As and when a service fails, get the calling api and call the delete method of that service and undone whatever u have done.
PS- Just a raw thought. Correct me if you think it is wrong.
Building on top of previous answers, distributed transactions are the solution. In my opinion you don't want to build your own mechanisms for tracking global transactional state, rather you want to use some kind of product - there are several out there. I have written a lengthy blog article about solving this issue with a Java application server:
http://blog.maxant.co.uk/pebble/2015/08/04/1438716480000.html
two-phase commit can be option.Coordinator send commit request message to cohorts.Cohorts send back ok.After then coordinator sends commit message to cohorts.If any failure happpens coordinator sends rollback messages to cohorts.
You can use a workflow engine(like JBPM,Activiti) to orchestrate the logic and handle transaction failures or compensatory transactions in it to achieve data integrity. This is the similar case you use in SOA architecture with ESB,BPMN and Web services
I'm building a web service that executes a database process (SQL code to run several queries , then move data between two really large tables), I'm assuming some processes might take 2 to 10 hours to execute.
What are the best practices for executing a long running database process from within a Java web service (it's actually REST-based using JAX-RS and Spring)? The process would be executed upon 1 web service call. It is expected that this execution would be done once a week.
Thanks in advance!
It's gotta be asynchronous.
Since your web service call is an RPC, best to have the implementation validate the request, put it on a queue for processing, and immediately send back a response that has a token or URL to check on progress.
Set up a JMS queue and register a listener that takes the message off the queue and persists it.
If this is really taking 2-10 hours, I'd recommend looking at your schema and queries to see if you can speed it up. There's an index missing somewhere, I'd bet.
Where I work, I am currently evaluating different strategies for this exact situation, only times are different.
With the times you state, you may be better served by using Publish/Subscribe message queuing (ActiveMQ).
I have an enterprise application with around 2k concurrent users every day. These users handle customer calls so application speed is of vital importance.
When a user is wrapping up a call they commit all the information they captured. This commit can take anywhere from 10-45 seconds.
I am looking into ways to take the delay away from the user.
We have a web front end running in I.E. the backend is heavy java running on a single EJB.
I wanted to make this commit process asynchronous in that once the user submits the request they don't have to wait for the commit to finish before going on to the next customer. This is what is currently implemented.
Originally I was thinking of just spawning another thread to handle the commit but that's a no no with EJB's.
Other options I can think of would be using JMS or SIB,
What would the best solution be? Is there another alternative I am missing?
There are actually two alternatives for cases like that.
The first one will be to use JMS. It has the advantage that the server provides all required infrastructure and you haven't to implement much yourself.
Another way will be to register the request in a database and have a scheduled event to process all of them.
What you select depends on your requirements. If you need to serve the requests as soon as they arrive, then you need to go with JMS. In both cases you need to persist the outcome of the request in a database and design a web service at the top of it. The front end could use this (through pollling) to present the result to the user.
Would have liked to leave a comment, but don't have the ability.
Another possibility:
Wrap the heavy EJB's in a queue mechanism, and expose a different bean with the same API so your client-facing communications talk to the new bean and are quick. They accept the request, add the job to the queue and return to the client immediately. You don't need to change the heavy EJB's or the client communications, just put a mediator in the way, and add a layer of wrapping.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 9 years ago.
Improve this question
I'm looking for (simple) examples of problems for which JMS is a good solution, and also reasons why JMS is a good solution in these cases. In the past I've simply used the database as a means of passing messages from A to B when the message cannot necessarily be processed by B immediately.
A hypothetical example of such a system is where all newly registered users should be sent a welcome e-mail within 24 hours of registration. For the sake of argument, assume the DB does not record the time when each user registered, but instead a reference (foreign key) to each new user is stored in the pending_email table. The e-mail sender job runs once every 24 hours, sends an e-mail to all the users in this table, then deletes all the pending_email records.
This seems like the kind of problem for which JMS should be used, but it's not clear to me what benefit JMS would have over the approach I've described. One advantage of the DB approach is that the messages are persistent. I understand that JMS message queues can also be persisted, but in that case there seems to be little difference between JMS and the "database as message queue" approach I've described?
What am I missing?
- Don
JMS and messaging is really about 2 totally different things.
publish and subscribe (sending a message to as many consumers as are interested - a bit like sending an email to a mailing list, the sender does not need to know who is subscribed
high performance reliable load balancing (message queues)
See more info on how a queue compares to a topic
The case you are talking about is the second case, where yes you can use a database table to kinda simulate a message queue.
The main difference is a JMS message queue is a high performance highly concurrent load balancer designed for huge throughput; you can send usually tens of thousands of messages per second to many concurrent consumers in many processes and threads. The reason for this is that a message queue is basically highly asynchronous - a good JMS provider will stream messages ahead of time to each consumer so that there are thousands of messages available to be processed in RAM as soon as a consumer is available. This leads to massive throughtput and very low latency.
e.g. imagine writing a web load balancer using a database table :)
When using a database table, typically one thread tends to lock the whole table so you tend to get very low throughput when trying to implement a high performance load balancer.
But like most middleware it all depends on what you need; if you've a low throughput system with only a few messages per second - feel free to use a database table as a queue. But if you need low latency and high throughput - then JMS queues are highly recommended.
In my opinion JMS and other message-based systems are intended to solve problems that need:
Asynchronous communications : An application need to notify another that an event has occurred with no need to wait for a response.
Reliability. Ensure once-and-only-once message delivery. With your DB approach you have to "reinvent the wheel", specially if you have several clients reading the messages.
Loose coupling. Not all systems can communicate using a database. So JMS is pretty good to be used in heterogeneous environments with decoupled systems that can communicate over system boundaries.
The JMS implementation is "push", in the sense that you don't have to poll the queue to discover new messages, but you register a callback that gets called as soon as a new message arrives.
to address the original comment. what was originally described is the gist of (point-to-point) JMS. the benefits of JMS are, however:
you don't need to write the code yourself (and possibly screw up the logic so that it's not quite as persistent as you think it is). also, third-party impl might be more scalable than simple database approach.
jms handles publish/subscribe, which is a bit more complicated that the point-to-point example you gave
you are not tied to a specific implementation, and can swap it out if your needs change in the future, w/out messing w/ your java code.
One advantage of JMS is to enable asynchronous processing which can by done by database solution as well. However following are some other benefit of JMS over database solution
a) The consumer of the message can be in a remote location. Exposing database for remote access is dangerous. You can workaround this by providing additional service for reading messages from database, that requires more effort.
b) In the case of database the message consumer has to poll the database for messages where as JMS provides callback when a message is arrived (as sk mentioned)
c) Load balancing - if there are lot of messages coming it is easy to have pool of message processors in JMS.
d) In general implementation via JMS will be simpler and take less effort than database route
JMS is an API used to transfer messages between two or more clients. It's specs are defined under JSR 914.
The major advantage of JMS is the decoupled nature of communicating entities - Sender need not have information about the receivers. Other advantages include the ability to integrate heterogeneous platforms, reduce system bottlenecks, increase scalability, and respond more quickly to change.
JMS are just kind of interfaces/APIs and the concrete classes must be implemented. These are already implemented by various organizations/Providers. they are called JMS providers. Example is WebSphere by IBM or FioranoMQ by Fiorano Softwares or ActiveMQ by Apache, HornetQ, OpenMQ etc. .Other terminologies used are Admin Objects(Topics,Queues,ConnectionFactories),JMS producer/Publisher, JMS client and the message itself.
So coming to your question - what is JMS good for?
I would like to give a practical example to illustrate it's importance.
Day Trading
There is this feature called LVC(Last value cache)
In Trading share prices are published by a publisher at regular intervals. Each share has an associated Topic to which it is published to. Now if you know what a Topic is then you must know messages are not saved like queues. Messages are published to the subscribers alive at the time the message was published(Exception being Durables subscribers which get all the messages published from the time it was created but then again we don't want to get too old stock prices which discard the possibility of using it). So if a client want to know a stock price he create a subscriber and then he has to wait till next stock price is published(which is again not what we want). This is where LVC comes into picture. Each LVC message has an associated key. If a messages is sent with a LVC key(for a particular stock) and then another update message with same key them the later overrides the previous one. When ever a subscriber subscribes to a topic(which has LVC enabled) the subscriber will get all the messages with distinct LVC keys. If we keep a distinct key per listed company then when client subscribes to it it will get the latest stock price and eventually all the updates.
Ofcourse this is one of the factors other that reliability,security etc which makes JMS so powerful.
Guido has the full definition. From my experience all of these are important for a good fit.
One of the uses I've seen is for order distribution in warehouses. Imagine an office supply company that has a fair number of warehouses that supply large offices with office supplies. Those orders would come into a central location and then be batched up for the correct warehouse to distribute. The warehouses don't have or want high speed connections in most cases so the orders are pushed down to them over dialup modems and this is where asynchronous comes in. The phone lines are not really that important either so half the orders may get in and this is where reliability is important.
The key advantage is decoupling unrelated systems rather than have them share comon databases or building custom services to pass data around.
Banks are a keen example, with intraday messaging being used to pass around live data changes as they happen. It's very easy for the source system to throw a message "over the wall"; the downside is there's very little in the way of contract between these systems, and you normally see hospitalisation being implemented on the consumer's side. It's almost too loosly coupled.
Other advantages are down to the support for JMS out of the box for many application servers, etc. and all the tools around that: durability, monitoring, reporting and throttling.
There's a nice write-up with some examples here: http://www.winslam.com/laramee/jms/index.html
The 'database as message queue' solution may be heavy for the task. The JMS solution is less tightly coupled in that the message sender does not need to know anything about the recipient. This could be accomplished with some additional abstraction in the 'database as message queue' as well so it is not a huge win...Also, you can use the queue in a 'publish and subscribe' way which can be handy depending on what you are trying to accomplish. It is also a nice way to further decouple your components. If all of your communication is within one system and/or having a log that is immediately available to an application is very important, your method seems good. If you are communicating between separate systems JMS is a good choice.
JMS in combination with JTA (Java Transaction API) and JPA (Java persistence API) can be very useful. With a simple annotation you can put several database actions + message sending/receiving in the same transaction. So if one of them fails everything gets rolled back using the same transaction mechanism.