I have read some articles about microservices architecture, but no one takes the topic of transaction. All that they says that this is hard to do it. Maybe someone can describe how to handle this?
But not from domain side, but from technology side. Lets say that we have business case where we need to invoke two different services and both of them make some changes on database. But how to rollback if some error occurs on the second one?
Who knows some libraries or design patter for this problem?
I may not be the ultimate expert in this, but I'm sure you're heading towards the Distributed Transactions. In order to have them running, all the application service components need a common shared transaction id, and you have to make sure that every component is informed about the state of the transaction. It is asynchronous, so you'll require substantial prog skills.
Here are distributed transactions mentioned or discussed:
https://en.wikipedia.org/wiki/Distributed_transaction
http://contino.co.uk/microservices-not-a-free-lunch/
http://martinfowler.com/articles/microservices.html
It would seem people try to avoid it as it is difficult. Maybe that's why you don't find much about.
Hope this helps a step forward :-)
The best design is having isolated services: each service just do its work within its own transaction and your workflow expects failures on the single service.
If you really need to commit only if all the services are called without errors you should create an higher level service that perform those calls inside an external transaction.
The first raw thing which came to my mind after reading this question is to create every add api with a delete api with ,lets say, an extra boolean flag delFlag.
boolean flag delFlag;
For POST, it will be 0. For DELETE, it will be 1.
Now you maintain a Transaction Manager which is a super service to all your micro services.
In this service maintain the calling queue of all the services and the APIs. As and when a service fails, get the calling api and call the delete method of that service and undone whatever u have done.
PS- Just a raw thought. Correct me if you think it is wrong.
Building on top of previous answers, distributed transactions are the solution. In my opinion you don't want to build your own mechanisms for tracking global transactional state, rather you want to use some kind of product - there are several out there. I have written a lengthy blog article about solving this issue with a Java application server:
http://blog.maxant.co.uk/pebble/2015/08/04/1438716480000.html
two-phase commit can be option.Coordinator send commit request message to cohorts.Cohorts send back ok.After then coordinator sends commit message to cohorts.If any failure happpens coordinator sends rollback messages to cohorts.
You can use a workflow engine(like JBPM,Activiti) to orchestrate the logic and handle transaction failures or compensatory transactions in it to achieve data integrity. This is the similar case you use in SOA architecture with ESB,BPMN and Web services
Related
I have an API call that will need to work with more than one aggregate. I have the 2 below ideas in my head to how it should be interacting with the aggregates but I'm open to other ideas.
Is it good practice to send commands from one microservice to another one? Or is it better to have an event handler on microservice B that reacts to events from service A and generates the command all within the microservice B?
Is it good practice to send commands from one microservice to another one? Or is it better to have an event handler on microservice B that reacts to events from service A and generates the command all within the microservice B?
An important thing to recognize in a service architecture: we want the services to be autonomous. So A should continue working while B is down for maintenance, and vice versa.
This implies that we need to support asynchronous messaging from A to B.
Current "best practice" is that you are dealing with messages being delivered asynchronously, then the semantics should be past tense: SomethingHappened at A, and B will react to it, or not, in its own time at its own discretion.
Does it matter? Hard to say -- handle(Event) is a command, CommandReceived is an event.
Note: this is really just services and messaging -- Event-Sourcing/CQRS really don't enter into it.
Martin Fowler described Domain Events in 2005.
Each Domain Event captures information from the external stimulus.
If you think of A as being external to B (which makes sense, if there are service boundaries between them), then the semantics of the Domain Event pattern may be a very good fit.
Why not both?
I approach these "micro-services" slightly differently. I usually have a messaging endpoint for each bounded context. I guess this fits in nicely with the micro-service idea and that endpoint only responds to commands sent to it that apply to that BC. It would then also publish the relevant events.
What I then may also have is an orchestration endpoint that responds to process managers that "belong" to the relevant BC. This endpoint only deals with the state of the process managers and issues commands to whichever BC messaging endpoint it needs to talk to. For instance, after an OrderRegisteredEvent has been received a SendEMailCommand may be issued. OK, that is more of a technical endpoint/BC but none-the-less.
On the BC-only messaging endpoint there is absolutely no between the different BCs. It is only there to service its own BC.
I hope that makes sense.
I have web service for reading and updating data and using spring, spring JDBC for DB access. My controller can be accessed by many channels like desktop, mobile etc. If data is updated using desktop, then same should reflect in mobile immediately. Current approach is calling service continuously to get updated data. I feel that it is worst approach and causing DB performance issue as well.
Is there a possible way such that GET service is called only when there is DB update by other channel instead of continuous polling ? What is best approach for this and how to implement it ?
Continuously calling the service seems like a really bad idea. I think you need a database trigger that fires when rows are inserted/updated/deleted. It could POST something to a Web Service or put something on a Message Queue.
Good luck.
I can think of an architectural answer to the problem. Use a messaging solution between the spring controller and the database. Infact you will need two queues
EventSink queue -
Publish all data change requests originating from any of the channels to this queue.The subscriber will be the service managing the database update aka dbservice .
EventBroadcast queue -
Publish the changed data post db update to this queue. Ideally the dbservice should handle this publish within the same transaction as db update. All channels can subscribe to this queue to receive the update.
The merits to consider this approach would involve
Pros - this approach involves no database services so both performance and de-coupling from database changes.
Cons - Increased complexity
Continuously polling is not as bad as you might imagine. Pushing messages to clients without them making a request requires web-sockets or of the like to achieve this. If it is not a large repose from the server, and is not too often, as in many millions and millions of requests then I would leave it for now.
If however this is a large amount of bandwidth we are talking about it then you wouldn't want to be polling. You would probably want to look at a subscriber type pattern whereby clients would subscribe to be notified when a specific event occurs. When this event occurs the server would then send a message to the clients.
Detecting this event shouldn't require polling a database. The modification to the database should trigger the event. You might do this with point-cuts in Spring if you are into that sort of thing.
I read somewhere use of webservcies in apps. After a lot of research I am able to create one Webservice which will accept Json and JsonP both format as request and response accordingly. I developed the webservcies using Java, Apache Axis2, Hibernate and MySQL as database. there are few problems and I dont know how to solve ?
Insert or delete option, sometimes if at a time more than two users call that service that is insert or delete any row the queries go in sleep mode and next time someone tries to fetch that service he couldnt. Accroding to server log it says error SQL Lockout State. If I checks Processlist in MYSQL it is showing that query in Sleep, I have to kill to resume.
The performance of webservice doesnt seems to be upto mark, it takes time some more time as what i experienced it shouldn't. In simple words how to obtain better performance by the services
How to implement security feature such that if a user logins he/she can be provided an id and validation of that id so that unauthorized access can be prevented
Or just guide me what should be the most appropriate and optmized Webservice methodology that can be used using Java
Answer to this question is not specific to Android. Below are my investigations which might be useful for you.
For the point about MySQL connections going to sleep mode, you can do the following.
Debug the datasource used by Hibernate, try to increase the pool size & check for any issues in it.
Define a timeout period for connections. JBoss has several configurations related to this like blocking-timeout-millis, idle-timeout-minutes etc.
Declare a mechanism to validate periodically the connection resources in the pool for activeness. You can explore OracleStaleConnectionChecker for options.
Configure miniumn connections in the pool. This is important because when all the stale connections are discarded, empty pool needs to be pre-filled & ready with active connections.
Coming to performance of Insert/Delete operations & SQL Lockout State, please try to re-order the sequence of the queries which you are firing to DB at every request. This may not be a deadlock situation but sequencing DB queries correctly will definitely lead to less lockout time and better performance.
This answer may be of use for you. Hibernate: Deadlock found when trying to obtain lock
Web-services which you have developed may require some performance optimization to make them upto the mark. Below are first few steps you can take to bring the performance up.
Avoid nested loops. Every extra parameter in the iterated lust increase the order of the lopp exponentially.
Remove early initialization of objects. This may lead to long unwanted GC cycles.
Apart from above optimizations, there are several frameworks & tools at your service to evaluate the code quality & its performance. PMD, FindBugs, JMeter, Java profiler are few of them to name.
Shishir
You are going to have to profile your server and see where the time is spent. I really like YourKit for doing thread profile. visualvm which comes with the JDK can help also.
There are all sorts of reasons your web service can be slow:
Latency from client to server
Handling the HTTP request on the server
Handling the HTTP response on the client
Making the database call (sounds like you already have some kind of locking / blocking going on there)
You are going to have to get markers to tell you how long it took to go from A to B to C to D back to C back to B back to A kind of thing. We would be speculating heavily from here on what is exactly going on in your program, but we can give you the ideas / tools to figure it out.
If you use YourKit, connect it to your server process. Have nothing else connecting to your server (for instance your client is not sending requests). Try it with your client requesting, you should see your accepting threads receive the HTTP request and then delegate to either your processing thread or do the processing itself. You can use YourKit to see how much time is spent in different functions during that call time.
Try it with your client making the call.
Try it using a simple HTTP request tool like wget or maybe your IDE has a webservice test tool (for instance intellij does), or you can download a simple HTTP test tool.
By testing it in a simple tool that just outputs the response, you can eliminate any client processing issues. You can also achieve a similar test in Chrome or Firefox and use the developer tools to see time to fulfill request.
In my experience, the framework for handling the requests and delegating can introduce some performance issues. I ripped Grails out of a production environment because of its performance issues (before any Grails / Groovy flames come my way, we were operating at a much higher rate than typical web applications, and I am sure Grails has made some headway in the last couple years... alas, it was not for my need at that time)
BTW, I doubt you are operating a load where you will be critiquing the web service framework you chose to use. I have been happy with Spring MVC and DropWizard (Jersey JAX-RS), and Grails is easy to use too.
You should make a simple static content response in your webservice and see how quickly that returns vs a request that makes a database call.
Also, what kind of table are you using in MySQL? InnoDB? MyISAM? They have different locking schemes. That could be causing your MySQL issue.
The key to all of it, break the problem up into parts, and measure each and eliminate parts one by one till you go, everytime I do X it is slower (like everytime I make a database call its slower)
In Java the the way you will be able to find more support online via documentation/forums is to develop the web service as a REST web service using Spring MVC.
You can base yourself on this resource and take it from there:
Spring MVC REST Hello World Web Service
Using Spring you can create a RestFul webservice easily and spring does all the ground work you needed. As others had mentioned you can consume the webservice in any type of client - including Android.
A detailed guide available here:
https://spring.io/guides/gs/rest-service/
Here are my suggestions:
Make APIs only read or write database. If an API combines reading and writing, it is possible to cause deadlock;
Use a light-weight HTTP server. Powerful HTTP server is possibly consuming more.
Make use of thread. Have more threads could be helpful when you are facing a ton of users.
Make more things static. You could avoid unnecessary queries.
I think mhoglan's answer is detailed enough.
We have a system (Java web application) that's been in active development / maintenance for a long time now (something like ten years).
What we're looking at doing is implementing a RESTful API to the web app. This web application, using Jersey, will be a separate project with the intent that it should be able to run alongside the main application or deployed in the cloud.
Because of the nature and age of our application, we've had to implement a (somewhat) comprehensive caching layer on top of the database (postgres) to help keep load down. Anyway, for the RESTful API, the idea is that GET requests will go to the cache first instead of the database to keep load of the database.
The cache will be populated in a way to help ensure that most things registered API users will need should be in there.
If there is a cache miss, the needed data should be retrieved from the database (also being entered into the cache in the process).
Obviously, this should remain transparent from the RESTful endpoint methods in my code. We've come up with the idea of creating a 'Broker' to handle communications with the DB and the cache. The REST layer will simply pass across ids (if looking to retrieve) or populated Java objects (if looking to insert / update) and the broker will take care of retrieving / updating / invalidating, etc.
There is also the issue of extensibility. To begin with, the API will be living alongside the rest of servers so access to the database won't be an issue however if we deploy to the cloud, we're going to need a different Broker implementation that will communicate with the system (namely the database) in a different manner (potentially through the use of an internal API).
I already have a rough idea on how I can implement this but it struck me that is probably a problem for which a suitable pattern could exist. If I could follow an established pattern as opposed to coming up with my own solution, that'll probably be a better choice. Any ideas?
Ehcache has an implementation of just such a cache that it calls a SelfPopulatingCache.
Requests are made to the cache, not to the database. Then if there is a cache miss Ehcache will call the database (or whatever external data source you have) on your behalf.
You just need to implement a CacheEntryFactory which has a single method:
Object createEntry(Object key) throws Exception;
So as the name suggests, Ehcache implements this concept with a pretty standard factory pattern...
There's no pattern. Just hide the initial DB services behind interfaces, build tests around their intended behavior, then switch in an implementation that uses the caching layer. I guess dependency injection would be the best thing to help you do that?
Sounds like decorator pattern will suit your need: http://en.wikipedia.org/wiki/Decorator_pattern.
You can create an DAO interface for data access, something like:
Value get(long id);
And firstly create a direct DB implementation, then create a Cache implementation which calls underlying DAO instance, in this case it should be the DB implementation.
The Cache implementation will try to get value from its own managed Cache, and from underlaying DAO if it fails.
So both of your old application or the REST will only see DAO interface, without knowing any implemntation details, and in future you can change the implementation freely.
The best design pattern for transparently caching HTTP requests is to use an HTTP cache.
I have an enterprise application with around 2k concurrent users every day. These users handle customer calls so application speed is of vital importance.
When a user is wrapping up a call they commit all the information they captured. This commit can take anywhere from 10-45 seconds.
I am looking into ways to take the delay away from the user.
We have a web front end running in I.E. the backend is heavy java running on a single EJB.
I wanted to make this commit process asynchronous in that once the user submits the request they don't have to wait for the commit to finish before going on to the next customer. This is what is currently implemented.
Originally I was thinking of just spawning another thread to handle the commit but that's a no no with EJB's.
Other options I can think of would be using JMS or SIB,
What would the best solution be? Is there another alternative I am missing?
There are actually two alternatives for cases like that.
The first one will be to use JMS. It has the advantage that the server provides all required infrastructure and you haven't to implement much yourself.
Another way will be to register the request in a database and have a scheduled event to process all of them.
What you select depends on your requirements. If you need to serve the requests as soon as they arrive, then you need to go with JMS. In both cases you need to persist the outcome of the request in a database and design a web service at the top of it. The front end could use this (through pollling) to present the result to the user.
Would have liked to leave a comment, but don't have the ability.
Another possibility:
Wrap the heavy EJB's in a queue mechanism, and expose a different bean with the same API so your client-facing communications talk to the new bean and are quick. They accept the request, add the job to the queue and return to the client immediately. You don't need to change the heavy EJB's or the client communications, just put a mediator in the way, and add a layer of wrapping.