We have an application running in 2 containers/pods. This applications reads request from ActiveMQ and for processing this request this applications needs to update 10 tables. We have these hundreds of ASYNC request in ActiveMQ that needs to be processed in hundreds per second and each request tries to update the 10 tables in one oracle database. Because of these updates to 10 tables at the same fraction of a second some of the request fails frequently with this error "Error updating database. Cause: java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource"
Is there a better way to handle these kind of scenarios like a better architecture ?
Is there a better way to handle these kind of scenarios using spring framework?
Your premise is wrong; deadlocks are not caused simply by a large amount of activity on many tables. They're a special kind of lock that occur when different sessions try to lock the same row in a different order. When that happens, the only solution is to kill one of the sessions to release the lock.
The first step in investigating deadlocks is to look in the alert log and find the trace file generated by a deadlock. It will show the statements and objects involved. If you're lucky, the mistake is caused by a missing foreign key index or a bitmap index on a transactional table, which can be resolved with a simple DDL change.
If you're unlucky, the application is changing tables in different orders. In that case, you'll need to change the application to always process changes the same way. I don't think I can offer any advice for that, it depends entirely on your application.
Related
Context
We have a Spring boot application (an API used by an angular frontend).
It is running on a docker container.
It is using a single instance of a PostgreSQL database.
Our application had some load problems so we asked us to scale it.
We told us to run our API on several docker containers for that.
We have several questions / problems dealing with code synchronization over multiple docker instances executing our code.
Problem 1
We have some #Scheduled jobs integrated and deployed with our API code.
We don't want these scheduled jobs to be executed by all container instances, but only one.
I think we can simply handle this by disabling jobs on other containers through environment variables with the "-" value to disable the Spring scheduled cron.
https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/scheduling/annotation/Scheduled.html#CRON_DISABLED
Does it sounds right?
Problem 2
The other problem is that we use Spring's #Lock annotation on some repository methods.
public interface IncrementRepository extends JpaRepository<IncrementEntity, UUID> {
#Lock(LockModeType.PESSIMISTIC_FORCE_INCREMENT)
Optional<IncrementEntity> findByAnnee(String pAnneeAA);
#Lock(LockModeType.PESSIMISTIC_WRITE)
IncrementEntity save(IncrementEntity pIncrementEntity);
}
This is critical for us to have a lock on that as we get / compute an increment used to act as a unique identifier of some of our data.
If I correctly understood this locking mechanism :
if a process execute this code, the Spring JPA #Transaction will acquire a lock on the IncrementEntity (lock the database table).
when another process tries do do the same thing before the first lock has been released by the first transaction, it should have a PessimisticLockException and the second transaction will rollback
this is managed by Spring at application level, NOT directly at database level (right??)
So what will happen if we're running our code on several containers ?
app running in container 1 sets a lock
app running in container 2 execute the same code and tries to set the same lock while the first one has not been released yet
each Spring application running in different containers will probably acquire the lock without problems as they don't share the same information?
Please tell me if I correctly understood how it works, and if we will effectively have a problem running such code on several docker containers.
I guess that solution would be to set a lock directly on the database table, as we have only one instance of it?
Is there a way to easily set / release the lock at database level using Spring JPA code ?
Or perhaps I misunderstood and setting a lock using Spring's #Lock annotation sets a real database lock ?
In that case, perhaps we don't have any problem at all, as the lock is correctly set on the database itself, shared by all containers instances??
Problem 3
To avoid having too much exceptions and reject some requests trying to acquire a lock at the same time, we also added a synchronized block around the above code.
String numIncrement;
synchronized (this.mutex) {
try {
numIncrement = this.incrementService.getIncrement(var);
} catch (Exception e) {
// rethrow custom technical exception
}
}
This way concurrent requests should be delayed and queued, which is better for our users experience.
I guess that we will also have problems here as docker instances doesn't share the same JVM, so synchronization can work only in the scope of the container itself... right?
Conclusion
For all these problems, please tell me if you have some solutions to workaround / adapt our code so it can be compatible with app scaling.
Following a set of tests I can confirm these points about my original question
Problem 1
We can disable a Spring CRON with the - value
#Scheduled(cron = "-")
Problem 2
The Spring's JPa #Lock annotation sets a lock on the database itself. It is not managed by Spring software.
So when duplicating containers, if the Spring app in the first container sets a lock, the database is locked and when the second app in another container tries to get data it has the PessimisticLockException.
Problem 3
Synchronized code using the synchronized JAVA keyword is obviously managed by JVM, so there is no code mutual exclusion between containers.
Is it possible to set a DefaultUncaughtExceptionHandler for every worker and thread in Dataflow ?
Something like Thread.setDefaultUncaughtExceptionHandler.
No, I don't believe so. The Dataflow model for expressing user code operates at a higher level of abstraction than individual workers and threads. Can you expand a bit more on why you'd like to do this globally?
The Dataflow Service already retries all uncaught exceptions in your user code. Elements are processed in groups called bundles -- if any element in a bundle causes an exception to be thrown, the entire bundle is retried. In batch mode, failing bundles are retried until a single bundle has failed 4 times, at which point the job is failed. In streaming mode, failing bundles are retried indefinitely, although you can use the update feature to update your code to better handle the issue.
All of these exceptions will appear in both the job logs in Dataflow UI and in your Cloud Logging worker logs. See more info here.
I am trying to implement row level security so our application can enforce more stringent access control.
One of the technologies we are looking into is Oracle's Virtual Private Database, which allows row level security by basically augmenting all queries against specific tables with a where clause predicate. Since we are in a web environment, we need to set up a special context within Oracle, inside a single request's thread. We use connection pooling with a service account.
I started to look into Eclipse Link and Hibernate. Eclipse Link seems to have events that fit perfectly into this model.
This would involve us migrating from hibernate, which is not a problem, but we would then be bound to EL for these events.
Oracle seems to imply that they implement at the data source level in Web Logic product.
The context is set and cleared by the WebLogic data source code.
Question: Is it more appropriate to do this at the DataSource level with some series of events. What are the events or methods that I should pay the most attention too?
Added Question: How would I extend a connection pool to safely initialize an oracle context with some custom data? I am digging around in Apache, and it seems like extending BasicDataSource doesn't give me access to anything that would allow me to clean up the connection when Spring is done with it.
I need to set up a connection, and clean up a connection as the exit / enter the connection pool. I am hoping for an implementation that is so simple, no one can mess it up by breaking some delicate balance of products.
- Specifically we are currently using Apache Commons DBCP Basic Data Source
This would allow us to use various ways to connect to the database and still have our security enforced. But I don't see a great example or set of events to work with, and rolling my own security life cycle is never a good idea.
I eventually solved my problem by extending some of the Apache components.
First I extended org.apache.commons.pool.impl.GenericObjectPool and overrode both borrowObject() and returnObject(). I knew the type of the objects in the pool (java.sql.Connection) so I could safely cast and work with them.
Since for my case I was using Oracle VPD, I was able to set information in the Application context. I recommend you read about that in more depth. It is a little complicated and there are a lot of different options to hide or share data at various contexts level, and across RAC nodes. Start
In essence what I did was generate a nonce and use it to instantiate a session within oracle, and then set the access level of the user to a variable in that session, that the Oracle VPD policy would then read and use to do the row level filtering.
I instantiated and destroyed that information in my overridden borrowObject() and returnObject() The SQL I ran was something like this:
CallableStatement callStat =
conn.prepareCall("{call namespace.cust_ctx_pkg.set_session_id(" + Math.random() + ")}");
callStat.execute();
Note math.random() isn't a good nonce.
Next was to simply extend org.apache.commons.dbcp.BasicDataSource and set my object pool by overriding createConnectionPool(). Note that the way I did this disabled some functionality I did not need, so you may need to rewrite more or less than I did.
You can try any object level security mechanism for simplicity, like Spring Security ACL.
You will want to do this at the application layer. You will want a pre-commit hook and a post read hook.
The pre-commit hook is used to ensure that data from the client is being presented by a user authorized to modify that data. This prevents an unauthorized user from overwriting data that they shouldn't be able to access.
It's not intuitive, but the post read hook is used to keep the client from accessing data the user shouldn't be allowed to view. This happens post-view because this is being enforced at the application layer, not at the data layer. The application has no way to know if the caller is allowed to access the data until it's been retrieved from the data layer. In the post read hook you evaluate the credential on each row returned against the credential of the logged in user in order to determine whether or not access is allowed. If access is denied on any row then an exception would be raised and the data would not be returned to the client.
Application level security done in this way requires that you have a way to connect each row in a table to a permission/role required to access it and a way to evaluate a user's permissions on the server at runtime.
Hope that helps.
You will get better control by using one of the other Commons DBCP Datasources.
The Basic one is just that: basic :)
The ones in org.apache.commons.dbcp.datasources package gives you more fine-grained control.
We run several spring batch jobs within tomcat in the same web application that serves up our UI. Lately we have been adding many more jobs and we are noticing that when we patch our app, several jobs may get stuck in a STARTING or STARTED status. Many of those jobs ensure that another job is not running before they start up, so this means after we patch the server, some of our jobs are broken until we manually run SQL to update the statuses of the jobs to ABANDONED or STOPPED.
I have read here that JobScope and StepScope jobs don't play nicely with shutting down.
That article suggests not using JobScope or StepScope but I can't help but think that this is a solved problem where people must be doing something when the application exits to prevent this problem.
Are there some best practices for handling this scenario? What are you doing in your applications?
We are using spring-batch version 3.0.3.RELEASE
I will provide you an idea on how to solve this scenario. Not necessarily a spring-batch solution.
Everytime I need to add jobs in an application I do as this:
Create a table to control the jobs (queue, priority, status, etc.)
Create a JobController class to manage all jobs
All jobs are defined by the status R-running, F-Finished, Q-Queue (you can add more as you need like aborted, cancelled, etc) (the jobs control these statuses)
The jobController must be loaded only once, you can define it as a spring bean for this
Add a boolean attribute to JobController to inform if you already checked the jobs when you instantiate it. Set it to false
Check if there are jobs with the R status which means that in the last stop of the server they were running so you update every job with this R status to Q and increase their priority so it will get executed first after a restart of the server. This check is inside the if for that boolean attribute, after the check set it to true.
That way every time you call the JobController for the first time and there are unfinished jobs from a server crash you will be able to set then all to a status where it can be executed again. And this check will happens only once since you will be checking that boolean attribute.
A thing that you should be aware of is caution with your jobs priority, if you manage it wrong you may run into a starvation problem.
You can easily adapt this solution to spring-batch.
Hope it helps.
I have a scenario where the unit of work is defined as:
Update table T1 in database server S1
Update table T2 in database server S2
And I want the above unit of work to happen either completely or none at all (as the case with any database transaction). How can I do this? I searched extensively and found this post close to what I am expecting but this seems to be very specific to Hibernate.
I am using Spring, iBatis and Tomcat (6.x) as the container.
It really depends on how robust a solution you need. The minimal level of reliability on such a thing is XA transactions. To use that, you need a database and JDBC driver that supports it for starters, then you could configure Spring to use it (here is an outline).
If XA isn't robust enough for you (XA has failure scenarios, such as if something goes wrong in the second phase of commits, such as a hardware failure) then what you really need to do is put all the data in one database and then have a separate process propagate it. So the data may be inconsistent, but it is recoverable.
Edit: What I mean is that put the whole of the data into one database. Either the first database, or a different database for this purpose. This database would essentially become a queue from which the final data view is fed. The write to that database (assuming a decent database product) will be complete, or fail completely. Then, a separate thread would poll that database and distribute any missing data to the other databases. So if the process should fail, when that thread starts up again it will continue the distribution process. The data may not exist in every place you want it to right away, but nothing would get lost.
You want a distributed transaction manager. I like using Atomikos which can be run within a JVM.