One spring service is implemented in one java deployment unit(JVM). Another spring service is implemented in another JVM. Making service call from 1st jvm to 2nd jvm. Service interface could be either rest or soap over http. Need to keep single transaction over multiple jvms, meaning if any service fails every thing must be rolled back. How to do this. Any code examples.
Use global transactions (i.e., JTA),
Use XA resources (RDBMS and JMS connections), do "Full XA with 2PC".
For further reference on the Spring transaction management, including the JTA/XA scenario, read: http://docs.spring.io/spring/docs/current/spring-framework-reference/htmlsingle/#transaction
REST faces the exact same problem as SOAP-based web services with regards to atomic transactions. There is no stateful connection, and every operation is immediately committed; performing a series of operations means other clients can see interim states.
Unless, of course, you take care of this by design. First, ask yourself: do I have a standard set of atomic operations? This is commonly the case. For example, for a banking operation, removing a sum from one account and adding the same sum to a different account is often a required atomic operation. But rather than exporting just the primitive building blocks, the REST API should provide a single "transfer" operation, which encapsulates the entire process. This provides the desired atomicity, while also making client code much simpler. This appracoh is known as low granularity services, or high-level batch operations.
If there is no simple, pre-defined set of desired atomic operation sequences, the problem is more severe. A common solution is the batch command pattern. Define one REST method to demarcate the beginning of a transaction, and another to demarcate its end (a 'commit' request). Anything sent between these sets of operations is queued by the server but not committed, until the commit request is sent.
This pattern complicates the server significantly -- it must maintain a state per client. Normally, the first operation ('begin transaction') returns a transaction ID (TID), and all subsequent operations, up to and including the commit, must include this TID as a parameter.
It is a good idea to enforce a timeout on transactions: if too much time has passed since the initial 'begin transaction' request, or since the last step, the server has the right to abort the transaction. This prevents a potential DoS attack that causes the server to waste resources by keeping too many transactions open. The client design must keep in mind that each operation must be checked for a timeout response.
It is also a good idea to allow the client to abort a transaction, by providing a 'rollback' API.
The usual care required in designing code that uses multiple concurrent transactions applies as usual in this complex design scenario. If at all possible, try to limit the use of transactions, and support high-level batch operations instead.
I take no credit of this information, i'm just a director, credit goes to This article
Also please read Transactions in REST?
You can get some handy code samples here http://www.it-soa.eu/en/resp/atomicrest/userguide/index.html
Related
I have a method
#Transactional
public void updateSharedStateByCommunity(List[]idList)
This method is called from the following REST API:
#RequestMapping(method = RequestMethod.POST)
public ret_type updateUser(param) {
// call updateSharedStateByCommunity
}
Now the ID lists are very large, like 200000, When I try to process it, then it takes lots of time and on client side timeout error occurred.
So, I want to split it to two calls with list size of 100000 each.
But, the problem is, it is considered as 2 independent transactions.
NB: The 2 calls is an example, it can be divided to many times, if number ids are more larger.
I need to ensure two separate call to a single transaction. If any one of the 2 calls fails, then it should rollback to all operation.
Also, in the client side, we need to show progress dialog, so I can't use only timeout.
The most obvious direct answer to your question IMO is to slightly change the code:
#RequestMapping(method = RequestMethod.POST)
public ret_type updateUser(param) {
updateSharedStateByCommunityBlocks(resolveIds);
}
...
And in Service introduce a new method (if you can't change the code of the service provide an intermediate class that you'll call from controller with the following functionality):
#Transactional
public updateSharedStatedByCommunityBlocks(resolveIds) {
List<String> [] blocks = split(resolveIds, 100000); // 100000 - bulk size
for(List<String> block :blocks) {
updateSharedStateByCommunity(block);
}
}
If this method is in the same service, the #Transactional in the original updateSharedStateByCommunity won't do anything so it will work. If you'll put this code into some other class, then it will work since the default propagation level of spring transaction is "Required"
So it addresses harsh requirements: you wanted to have a single transaction - you've got it. Now all the code runs in the same transaction. Each method now runs with 100000 and not with all the ids, everything is synchronous :)
However, this design is problematic for many different reasons.
It doesn't allow to track the progress (show it to the user) as you've stated by yourself in the last sentence of the question. REST is synchronous.
It assumes that network is reliable and waiting for 30 minutes is technically not a problem (leaving alone the UX and 'nervous' user that will have to wait :) )
In addition to that, the network equipment can force closing the connection (like load balancers with pre-configured request timeout).
That's why people suggest some kind of asyncrhonous flow.
I can say that you still can use the async flow, spawn the task, and after each bulk update some shared state (in-memory in the case of a single instance) and persistent (like database in the case of cluster).
So that the interaction with the client will change:
Client calls "updateUser" with 200000 ids
Service responds "immediately" with something like "I've got your request, here is a request Id, ping me once in a while to see what happens.
Service starts an async task and process the data chunk by chunk in a single transaction
Client calls "get" method with that id and server reads the progress from the shared state.
Once ready, the "Get" methods will respond "done".
If something fails during the transaction execution, the rollback is done, and the process updates the database status with "failure".
You can also use more modern technologies to notify the server (web sockets for example), but it's kind of out of scope for this question.
Another thing to consider here: from what I know, processing 200000 objects should be done in much less then 30 minutes, its not that much for modern RDBMSs.
Of course, without knowing your use case its hard to tell what happens there, but maybe you can optimize the flow itself (using bulk operations, reducing the number of requests to db, caching and so forth).
My preferred approach in those scenarios is make the call asynchronous (Spring Boot allow this using the #Async annotation), hence the client won't expect for any HTTP response. The notification could be done via a WebSocket that will push a message to the client with the progress each X items processed.
Surely it will add more complexity to your application, but if you design the mechanism properly, you'll be able to reuse it for any other similar operation you may face in the future.
The #Transactional annotation accepts a timeout (although not all underlying implementations will support it). I would argue against trying to split the IDs into two calls, and instead try to fix the timeout (after all, what you really want is a single, all-or-nothing transaction). You can set timeouts for the whole application instead of on a per-method basis.
From technical point, it can be done with the org.springframework.transaction.annotation.Propagation#NESTED Propagation, The NESTED behavior makes nested Spring transactions to use the same physical transaction but sets savepoints between nested invocations so inner transactions may also rollback independently of outer transactions, or let them propagate. But the limitation is only works with org.springframework.jdbc.datasource.DataSourceTransactionManager datasource.
But for really large dataset, it still need more time to processing and make the client waiting, so from solution point of view, maybe using async approach will be more better but it depends on your requirement.
I am working on building a microservice which is using transaction manager implemented based on Java Transaction API(JTA).
My question is does Trasaction maanger have ability to handle concurrency issue in distributed database scenario's .
Scenario:
Assume there are multiple instance of a service running and we get two requests to update balance amount by 10 in an account. Initially an account can have $100 and the first instance gets that and increments it to $10 but has not been commited yet.
At the same time the second instance also retreive's account which is still 100 and increments it by $10 and then commits it updating balance to $110 and then service one updates account again to $110.
By this time you must have figured that balance was supposed to be incremented by $20 and not 10. Do I have to write some kind of Optimistic lock exception mechanism to prevent the above scenario or will Transaction Manager based on JTA specification already ensure such a thing will not happen ?
does Trasaction maanger have ability to handle concurrency issue in distributed database scenario's .
Transactions and concurrency are two independent concepts and though Transactions become most siginificant in context where we also see concurrency , transactions can be important without concurrency.
To answer your question : No , Transaction Manager generally does not concern itself with handling issues that arise with concurrent updates. It takes a very naive and simple ( and often most meaningful ) approach : if after the start of a transaction , it detects that the state has become inconsistent ( because of concurrent updates ) it would simply raise it as an exception and Rollback the transaction. If only it can establish that all the conditions of the ACID properties of the transaction are still valid will it commit the transaction.
For such type of requests, you can handle through Optimistic Concurrency where you would have a column on the database (Timestamp) as a reference to the version number.
Each time when a change is commited it would modify the timestamp value.
If two requests try to commit the change at the same time, only one of them will succeed as the version (Timestamp) column will change by then negating other request from comitting its changes.
The transaction manager (as implementation of the JTA specification) makes transparent a work above multiple resources. It ensures all the operations happens as a single unit of work. The "work above multiple resources" mean that that the application can insert data to database and meanwhile it sends a message to a JMS broker. Transaction manager guarantees ACID properties to be hold for this two operations. In simplistic form when the transaction finishes successfully the application developer can be sure both operation was processed. When some trouble happens is on the transaction manager to handle it - possibly throw an exception and rollback the data changes. Thus neither operation was processed.
It makes this transparent for the application developer who does not need to care to update first database and then JMS and checks if all data changes were really processed or a failure happens.
In general the JTA specification was not written with microservice architecture in mind. Now it really depends on your system design(!) But if I consider you have two microservices where each one has attached its own transaction manager then the transaction manager can't help you to sort out your concurrency issue. Transaction managers does not work (usually) in some synchronization. You don't work with multiple resources from one microservice (what is the usecase for the transaction manager) but with one resource from multiple microservices.
As there is the one resource it's the synchronization point for all you updates. It depends on it how it manages concurrency. Considering it's a SQL database then it depends on the level of the isolation it uses (ACID - I = isolation, see https://en.wikipedia.org/wiki/ACID_(computer_science)). Your particular example talks about lost update phenomena (https://vladmihalcea.com/a-beginners-guide-to-database-locking-and-the-lost-update-phenomena/). As both microservices tries to update one record. One solution for the avoiding the issue is using optimistic/pesimistic locking (you can implement it on your own by e.g. timestamps as stated above), the other is to use serializable isolation level in your database, or you can design your application for not reading and updating data based on what is read first time but change the sql query having the update atomic (or there are possibly other strategies how to work with your data model to achieve the desired outcome).
In summary - it depends on how your transaction manager is implemented, it can help you in a way but it's not its purpose. Your goal should be to check how the isolation level is set up at the shared storage and consider if your application needs to handle lost update phenomena at application level or your storage cang manage it for you.
Let's imagine a problem:
I have a REST service, which is implemented using Java/MySQL/Spring and HTTP/JSON technologies.
The clients of the REST service are mobile applications.
So it's possible that someone will decompile the code and will get the API of the REST service.
( Yes, the code is obfuscated etc, but anyway ).
Problem: there is a POST method to send money to other user of the application.
I am worried, that someone can get the API, write a bot and make this POST request 500 or 5,000 or even 50,000 times per second.
As a result, he might send more money than he actually has, because if 1000 requests are processed simultaneously then the balance checking might be
successful for all 1000 request, however the real amount of money on an account might be enough only for, lets say, 50 requests.
So, basically, it's more like the standard "race" condition with multiple threads.
The problem is, that I have multiple servers and they are not related with each other anyhow.
So, 300 request can come to server A, 300 requests can come to server B and rest requests can come to server C.
The best idea what I have is to use something like "SELECT ... FOR UPDATE" and synchronize on database level.
However, I would like to consider another solutions.
Any ideas or suggestions?
You have a few options:
Rely on ACID implementation of the database (MySQL in your case). Assuming you are using InnoDB engine, you need to choose the right transaction isolation level (SET TRANSACTION syntax) in combination with the right locking reads mechanism (SELECT ... FOR UPDATE and SELECT ... LOCK IN SHARE MODE Locking Reads). You need to understand these concepts well in order to do the right choice. It might be possible that simply using the right isolation level will already prevent the race condition even without the locking reads. The cons are you are trading off consistency for scalability and tying your application to RDBMS database thus it will be more difficult for you to move to NoSQL for example.
Decompose your back end into web tier and service tier (option suggested by atk in the comments). This will allow you to scale the web tier instances independently while keeping a single service tier instance. Having a single service tier instance makes it possible to use Java synchronization mechanisms such as synchronised blocks or ReadWriteLock. Although this solution will work I wouldn't recommend it since it reduces the scalability of your service tier.
This is an enhancement of the previous option. You can use a Distributed lock manager instead of built-in java synchronization mechanisms. It will allow you to scale your web tier and service tier independently.
For mission critical applications, it's best to have multiple levels of locking mechanisms.
"SELECT ... FOR UPDATE" is a good way to do so, but they're pretty expensive, and when you try to bombard this with Charles, you'll see that your upper API stack will suffer, and that simple mechanism will cripple your infrastructure pretty easily, much like a DDoS event.
Implement it at the Load Balancer/Proxy layer first, to throttle N-number of requests per specified intervals from a single IP address.
Then apply a shared cache layer lock, where all your boxes go to synchronize on certain keys depending on which critical transaction you want to lockdown. For example, you can use Redis GETSET or INCR functionality to atomically set a flag, before entering the critical code path. Reject anything else quickly to avoid those bad actors from holding on to CPU/memory.
You can also implement things like an APC cache (before hitting your Redis / Memcache cluster) to do similar locking on a per box basis. This is quicker as there's no network latency involved.
The above are necessary on top of using "SELECT ... FOR UPDATE"
MongoDB introduced Bulk() since version 2.6, I checked the APIs, it's seems great to me.
Before this API, if I need to do a bulk insert, I have to store documents in a List, them use insert() to insert the whole List. In a multi-thread environment, concurrency should also be considered.
Is there a queue/buffer implemented inside the bulk API? each time I
put something into the bulk before execute(), the data is stored int
he queue/buffer, is that right?
Thus, I don't need to write my own queue/buffer, just use Bulk.insert() or Bulk.find().update(), is that right?
Could someone tell me more about the queue. Do I still need to concern the concurrency issues?
Since a Bulk is created like db.collection.initializeUnorderedBulkOp(), so if a bulk instance is not released, it will stay connected to the MongoDB server, is that right?
From the basic idea of "do you need to store your own list?", then not really, but I suppose it all really depends on what you are doing.
For a basic idea of the internals of what is happening under the Bulk Operations API the best place to look is at the individual command forms for each type of operation. So the relevant manual section is here.
So you can think of the "Bulk" interface as being a list or collection of all of the operations that you add to it. And you can pretty much add to that as much as you wish to ( within certain memory and practical constraints ) and consider that the "drain" method for this "queue" is the .execute() method.
As noted in the documentation there, regardless of how many operations you "queue" this will only actually send to the server in groups of 1000 operations at a time at maximum. The other thing to keep in mind is that there is no governance that makes sure that these 1000 operations requests actually fit under the 16MB BSON limit. So that is still a hard limit with MongoDB and you can only effectively form one "request" at a time that totals in less than that data limit in size when sending to the server.
So generally speaking, it is often more practical to make your own "execute/drain" requests to the sever once per every 1000 or less entries or so. Mileage may vary on this but there are some considerations to make here.
With respect to either "Ordered" or "UnOrdered" operations requests, in the former case all queued operations will be aborted in the event of an error being generated in the batch sent. Meaning of course all operations occuring after the error is encountered.
In the later case for "UnOrdered" operations, there is not fatal error reported, but rather in the WriteResult that is returned you get a "list" of any errors that are encountered, in addition to the "UnOrdered" meaning that the operations are not necessarily "applied" in any particular order, which means you cannot "queue" operations that rely on something else in the "queue" being processed before that operation is applied.
So there is the concern of how large a WriteResult you are going to get and indeed how you handle that response in your application. As stated earlier, mileage may vary to the importance of this being a very large response to a smaller and manageable response.
As far and concurrency is concerned there is really one thing to consider here. Even though you are sending many instructions to the sever in a single call and not waiting for individual transfers and acknowledgements, it is still only really processing one instruction at a time. These are either ordered as implied by the initialize method, or "un-ordered" where that is chosen and of course the operations can then run in "parallel" as it were on the server until the batch is drained.
But there is no "lock" until the "batch" completes, so it is not a substitute for a "transaction", so don't make that mistake as a design point. The same MongoDB rules apply, but the benefit here is "one write to server" and "one response back", rather that one for each operation.
Finally, as to whether there is some "server connection" held here by the API, then the answer is not there is not. As pointed to by the initial points of looking at the command internals, this "queue" building is purely "client side only". There is no communication with the server in any way until the .execute() method is called. This is "by design" and actually half the point, as mainly we don't want to be sending data to the server each time you add an operation. It is done all at once.
So "Bulk Operations" are a "client side queue". Everything is stored within the client side until the .execute() "drains" the queue and sends the operations to the server all at once. A response is then given from the server containing all of the results from the operations sent that you can handle however you wish.
Also, once .execute() is called, no more operations can be "queued" to the bulk object, and neither can .execute() be called again. Depending on implementation, you can have some further examination of the "Bulk" object and results. But the general case is where you need to send more "bulk" operations, you re-initialize and start again, just as you would with most queue systems.
Summing up:
Yes. The object effectively "queues" operations.
You don't need your own lists. The methods are "list builders" in themselves
Operations are either "Ordered" or "Un-Ordered" as far as sequence, but all operations are individually processed by the server as per normal MongoDB rules. No transactions.
The "initialize" commands do not talk to the server directly and do not "hold connections" in themselves. The only method that actually "talks" to the server is .execute()
So it is a really good tool. You get much better write operations that you do from legacy command implementations. But do not expect that this offers functionality outside of what MongoDB basically does.
Is there an open-source Java library that adds XA support to databases that don't support it natively? That is, it wraps a non-XA JDBC datasource and takes care of the necessary commits/rollbacks behind the scenes for 2-phase commits?
No, because it's impossible.
Let's review what XA is designed to achieve. It's a consensus protocol for guaranteeing ACID properties on transactions that span multiple resource managers. To do that it utilises a two phase commit protocol: the transaction manager prepares each resource manager, then commits each of them.
For the protocol to function correctly, the resource manager e.g. database, must make certain guarantees at the prepare stage. These include a) not making any changes visible to other processes until the commit phase ('Isolation'), b) ensuring it can perform the update at commit time if required, even if it crashes between prepare and commit ('Durability') and c) ensuring that data manipulated in different transactions exhibits the promised consistency properties. Realistically the only way to implement that is exclusive locking. Even resource managers e.g. pgsql and oracle, that use MVCC or other techniques during most operations will take exclusive locks at prepare.
Without access to the db internals, you can't acquire locks and hold them across connections. Hence you can't write code that can meet the transactional requirements. So, no layering of XA on top of a database engine - it has to be baked in.
However...
You can fake some aspects of the XA behaviour. Depending on your exact application requirements this may allow a useful solution to be crafted.
First up, you can use Last Resource Optimization (aka Last Resource Commit Optimization or Last Resource Gambit) to enlist a single non-XA i.e. one phase resource into a XA transaction with one or more real XA resources. By ordering the one phase resource last in the processing order you can achieve something that behaves like XA for most scenarios. It breaks horribly if a crash occurs at certain points in the execution, so you have to custom write data reconciliation code or rely on a human to handle that contingency. Depending on the semantics of your data that may or may not be an attractive option.
Next up, you can implement a custom driver that operates much like semantic replication. It records the sequence of SQL operations to a log at prepare time, but does not actually apply them to the db until the commit phase. This works for transactional updates that are isolated at the application level, but won't work if you're relying on the db to do concurrency control for you. For example, you may find the commit fails because something else snuck in a conflicting update between the prepare and commit phases. You could use an external lock manager, but only if your custom driver is the only thing talking to the db. As soon as a client that is not aware of that lock manager comes along all bets are off.
Finally, you can invert that model and use compensation based transactions under XA. In this model you apply the updates at prepare time and apply additional operations to reverse their effect in the rollback phase if needed. This has two drawbacks: concurrent operations may read and operate on the prematurely committed values of a tx that later rolls back, as there is no isolation between the prepare and commit; also depending on the business logic it's not easy to generate suitable compensation statements. Even if you can, you need quite a lot of complex plumbing to ensure they are run properly even in crash scenarios.
Realistically you're probably limited to LRCO, which is supported out of the box by most transaction managers. The other options require substantial transactions expertise to get right and the dev/test overhead usually isn't justified. If LRCO won't work for you then frankly it's going to be easier to redesign your app to avoid the need for XA.