This example is modified and simplified, but resembles a real problem I have with legacy code that I am trying to fix.
I have a MSSQL CAR table (where id is the primary key)
id
make
status
1
BMW
WASHED
2
BMW
DIRTY
3
BMW
DIRTY
4
Ford
DIRTY
...
...
...
and a /washNextCars endpoint.
#PatchMapping("/washNextCars")
public ResponseEntity<List<Car>> washNextCars(
#RequestBody WashCarBody body,
#RequestParam String status,
#RequestParam String make,
#RequestParam Integer limit) {
...
}
The following request changes the state of the next 10 cars from DIRTY to WASHED
PATCH /washNextCars?status=DIRTY&make=BMW&limit=10
{
"status": "WASHED"
}
This is the rough business logic:
#Transactional
public List<Car> washNextCars(WashCarDTO dto) {
// SELECT TOP ? * FROM CAR WHERE make=? AND status='DIRTY' ORDER BY id ASC
List<Car> nextCars = carRepository.findNextCars(limit, make, status);
// do some complex validation on every car if it can be washed, therefore can't do everything in one single UPDATE query
cars.forEach(car -> {
car.setStatus("WASHED");
});
carRepository.saveAll(nextCars);
}
This endpoint works fine when called in sequence, but when called in parallel, it tries to perform the update to the same set of cars at the same time, not the next batch of cars.
Questions:
How can I synchronize the calls until the first one is done?
Note this has to be achieved on the database level, because the API is deployed as multiple replicas
I found the following options:
#Transactional(isolation = READ_UNCOMMITED)
This would require to update the cars to WASHED before any other statement in the business logic is executed, is that correct?
#Lock(PESSIMISTIC_WRITE_LOCK) - Did not provide the desired effect
What is the right way to achieve this?
Try with UPDLOCK:
SELECT ... FROM tab1 WITH (UPDLOCK) WHERE ...
Something similar to
SELECT FOR UPDATE
in Oracle.
Related
Sample Scenario
I have a limit that controls the total value of a column. If I make a save that exceeds this limit, I want it to throw an exception. For example;
Suppose I have already added the following data: LIMIT = 20
id
code
value
1
A
15
2
A
5
3
B
12
4
B
3
If I insert (A,2) it exceeds the limit and I want to get exception
If I insert (B,4) the transaction should be successful since it didn't exceed the limit
code and value are interrelated
What can I do
I can check this scenario with required queries. For example, I write a method for it and I can check it in the save method. That's it.
However, I'm looking for a more useful solution than this
For example, is there any annotation when designing Entity ?
Can I do this without calling the method that provides this control every time ?
What examples can I give ?
#UniqueConstraint checking if it adds the same values
Using transaction
The most common and long-accepted way is to simply abstract in a suitable form (in a class, a library, a service, ...) the business rules that govern the behavior you describe, within a transaction:
#Transactional(propagation = Propagation.REQUIRED)
public RetType operation(ReqType args) {
...
perform operations;
...
if(fail post conditions)
throw ...;
...
}
In this case, if when calling a method there is already an open transaction, that transaction will be used (and there will be no interlocks), if there is no transaction created, it will create a new one so that both the operations and the postconditions check are performed within the same transaction.
Note that with this strategy both operation and invariant check transactions can combine multiple transactional states managed by the TransactionManager (e.g. Redis, MySQL, MQS, ... simultaneously and in a coordinated manner).
Using only the database
It has not been used for a long time (in favor of the first way) but using TRIGGERS was the canonical option used some decades ago to check postconditions, but this solution is usually coupled to the specific database engine (e.g. in PostgreSQL or MySQL).
It could be useful in the case where the client making the modifications is unable or unwilling (not safe) to check postconditions (e.g. bash processes) within a transaction. But nowadays it is infrequent.
The use of TRIGGERS may also be preferable in certain scenarios where efficiency is required, as there are certain optimization options within the database scripts.
Neither Hibernate nor Spring Data JPA have anything built-in for this scenario. You have to program the transaction logic in your repository yourself:
#PersistenceContext
EntityManager em;
public addValue(String code, int value) {
var checkQuery = em.createQuery("SELECT SUM(value) FROM Entity WHERE code = :code", Integer.class);
checkQuery.setParameter("code", code);
if (checkQuery.getSingleResult() + value > 20) {
throw new LimitExceededException("attempted to exceed limit for " + code);
}
var newEntity = new Entity();
newEntity.setCode(code);
newEntity.setValue(value);
em.persist(newEntity);
}
Then (it's important!) you have to define SERIALIZABLE isolation level on the #Transactional annotations for the methods that work with this table.
Read more about serializable isolation level here, they have an oddly similar example.
Note that you have to consider retrying the failed transaction. No idea how to do this with Spring though.
You should use a singleton (javax/ejb/Singleton)
#Singleton
public class Register {
#Lock(LockType.WRITE)
public register(String code, int value) {
if(i_can_insert_modify(code, value)) {
//use entityManager or some dao
} else {
//do something
}
}
}
I'm learning Spring MVC and I want find a car via an id but get in return the name.
In my service class I call a generic method getXXXById. This is something JPA gives me by nature.
I know that I get the whole entity but how can I just receive the corresponding name to the id.
Example: I call getCarById(2) and it gives me back Tesla.
My Table:
id | Name
----------
1 | Ford
2 | Tesla
My Service:
class CarService {
// code ...
public Optional<CarEntity> getCarById(int id) {
return carRepository.findById(id);
}
There are two options to do that.
Making your own query
You could write your own query in JQPL to retrive only names.
For example you could create method like that in your repository.
#Query("select t.name from CarEntity where id = ?1")
public String findNameById(Integer id);
more information on this feature of Spring Data Jpa HERE
Projections
Second option is to make projection. As it is written in documentation
Spring Data query methods usually return one or multiple instances of the aggregate root managed by the repository. However, it might sometimes be desirable to rather project on certain attributes of those types. Spring Data allows to model dedicated return types to more selectively retrieve partial views onto the managed aggregates.
In simple words, it allows you to aggregate your results form queries in some limited set of attributes rather then whole entity.
Specifically for your needs I'd suggest to use first approch, but it is worth to know both.
I have a RESTful service that exposes resources like /user/{id}
Now, the user can provide the credentials, get the token and access the resource. However, once authenticated, the user can access the resources for any id.
Meaning, user1 can access the URIs like /user/1 as well as user/2 and so on. I ended up using a Principal in the controller methods and started checking the id of the Principal with the id the user is trying to access.
Further, the user has multiple resources associated with it. Say, user1 owns res1 and res2, user2 owns res3 and res4. These can be accessed via /user/1/res/2. I need a way where I can prevent /user/1/res/3 as res3 is owned by user1 and not user2.
But I believe that this problem is very common and I am not really convinced with my solution.
Is there a better way to deal with this problem?
Thanks
You should not be exposing resourse /user/{id} at all if you all user can do is access only their own ID.
If I understand correctly, just exposing /user is enough, find ID of user from Principal or session etc and return result.
If you really want to do it, you can have custom implementation of #PreAuthorize. Got this code from a blog.
#PreAuthorize("isUsersRes(#id)")
#RequestMapping(method = RequestMethod.GET, value = "/users/{id}")
public UsersfindById(#PathVariable long id) {
return Users.findOne(id);
}
public class CustomMethodSecurityExpressionRoot
extends SecurityExpressionRoot implements MethodSecurityExpressionOperations {
public CustomMethodSecurityExpressionRoot(Authentication authentication) {
super(authentication);
}
And implemenation of isUsersRes
public class CustomMethodSecurityExpressionRoot
extends SecurityExpressionRoot implements MethodSecurityExpressionOperations {
public boolean isMember(Long OrganizationId) {
//user logic
}
Check complete blog here
This is a common problem with varied solutions. Also its not a problem related to REST alone. We have had this ever since apps exist. Employee can see his salary slip, leave records, etc but not another employee's.
One solution I like the most is 'security in depth'. This idea comes from how I have seen this work in banking systems for decades. This needs to get supported in the DB layer first.
You would need a table design like this example (or whatever your app's entity hierarchical structure is):
Organisation
-Dept
--user
And all non-master tables need to have a relation to one of these entities. Example:
Payslip -> user
Leave record -> user
Manager -> dept
HR Manager -> org
etc...
You would need another table to map out the basic access levels (This can get complex if we need to implement different sub access levels)
user1:dept2:org1
user2:dept2:org1
(I have seen some implementations that send this table's info as part of an encrypted access token that is used on every access request if the access has to be sessionless.)
You have not mentioned a framework/language but most languages have a database layer. For example if the DB layer is hibernate-java. There are interceptors (https://docs.jboss.org/hibernate/core/3.6/javadocs/org/hibernate/Interceptor.html#onPrepareStatement(java.lang.String)) that can be used to modify the query thats being executed.
Every query to DB will go with additional where-clause for these relationship keys.
We can get clever with Spring AOP, REST interceptors and a lot of other techniques on top of this basic architecture to enforce this security.
Idea will be that DB layer does not return data thats not accessible to the logged in user principal irrespective of what queries higher layer code uses.
if this is in place, a REST GET call for
/payslip/user1/Jan-2017
will end up with a 404 and not a 403.
Expecting this to be solved by a framework or a superficial set of interceptors is both risky and not future proof. We end up continuously tweaking the interceptors as the url patterns evolve.
Addition to show table examples:
ACL table
user, uid, dept, org
--------------------
jhon, 1 , 1 , 1
mary, 2 , 2 , 1
will, 3 , 2 , 1
Payslip table
--------------
month, net, deductions,..., uid
-------------------------------------
Jan , 200, 15.5 ,..., 3
Feb , 200, 15.5 ,..., 3
Project table
-------------
pname, pstart, pbudget, dept
------------------------------------
mark1, 12/21 , 20000 , 2
markx, 12/31 , 40000 , 2
What you want is user roles and permissions + cross user control. To find out user roles and permissions refer this
Also additionally you may want to cross check their user ID to the resource ID. Since you cannot let user1's resource ID 1 to view by user2, you will need to add userID as part of the resource id ex:- /user/user_id_1.
Otherwise we don't have a logical way to separate which resources are applicable to which users.
I'm facing a weird issue in Grails. When I make a findBy call after changing and saving values of a domain, I am still getting the old values even after the values get persisted to the database. I can see the values are changed in my table.
My code is something like this:
Car car = Car.findByCarId(carId)
car.modelName = "some_model_name"
car.save() // Not flushing here
Tire tire = Tire.findByIdAndCarId(tireId,carId)
tire.manufacturer = "some_manufacturer"
tire.save()
Light light = Light.findByIdAndCarId(lightId,carId)
light.manufacturer = "some_manufacturer"
light.save()
Mirror mirror = Mirror.findByIdAndCarId(mirrorId,carId)
mirror.manufacturer = "some_manufacturer"
mirror.save()
My domain also has a few one-to-many associations. Let's say something like this:
class Car {
String modelName
static hasMany = [tires : Tire, mirrors : Mirror, lights : Light]
}
After these changes, when I make a DB call for the Car domain, I still get the older values:
Car car = Car.findById(carId)
println car.modelName // This gives older value
I know this is because the values are yet to be persisted to the database. I want to know if I use car.save(flush: true) in the above code, will it cause collection was not processed by flush() error? However, I am also getting the older values even after the values are persisted to the database. (e.g. when I make the above query after a long time) I can see the values are changed in my tables, but when I do the above query, it gives me the old values. Does Hibernate cache this query automatically? I use the above query quite a lot of times.
When I use withNewSession, it retrieves the new values:
Car car
Car.withNewSession {
car = Car.findById(carId,[readOnly : true])
}
println car.modelName // Gives new value
I want to know how this is giving the new values everytime, since I'm not flushing the current session. Instead, I'm only using a new Hibernate session. Does readOnly command flush the current session? The above code works fine for me. Should I use withNewSession instead of flushing while saving?
Thanks.
All your business logic has to be inside a transactional service. All services in grails are transactional by default. Take care about notations as #Transactional or #NotTransactional
If you want to manage data in a controller (not recommended) you should surround your code into a Transaction. An allow hibernate to manage the transaction, not breaking it with flush.
All changes are commited after a transaction finishes.
Remember that you could also use the refresh method which re-reads the state of the given instance from the underlying database.
domainInstance.refresh()
I am bit confused about what data should a DTO contain.
For example let's assume that we have two tables: User, and Orders.
Orders table contains id_users, which is foreign key to user table.
Obviously I have two DAOs, MysqlUserDao and MysqlOrdersDao, with crud operations, and two transfer objects User, and Order, in which I store jdbc rowset.
If I want to get the list of users and for each user all his orders how should I do:
1) In my MysqlUserDao create a function: getUsersAndOrders(select users.,orders. from users join orders)
And my User DTO should have a OrderList property in where i put orders ?
2) In my MysqlUserDao i create a function getAllUsers(select * from users),
and foreach user I use MysqlOrdersDao function getOrder(id_user);
And some clarifications:
1) For each table in database I need to create a DAO object? or just for complex ones?
For example products and images, should be 2 dao or just one?
2) a DTO object should have only properties and setter getter, or it is possible to have other methods like convertEuroToUsd etc.
thanks
In your scenario #1 is the best option because #2 generates too much overhead.
1) In my MysqlUserDao create a function: getUsersAndOrders(select users.,orders. from users join orders) And my User DTO should have a OrderList property in where i put orders ?
Clarifications:
1: If your database has a good Design, then a DAO for each table is a good approach. There some cases where you can merge DAOs together (e.g: inheritance).
2: Yes. It should be a plain bean (or POJO if you want). I suggest creating another layer where you can define your workflow. I've seem people calling this extra layer as model, sometimes DataManager, sometimes just Manager.
For instance: When creating a order you should insert a record in Order table and also insert a record in the Notification table (because end users will be notified via email every time a order is created)
class OrderManager {
private OrderDAO oDao;
private NotificationDao nDao;
public saveOrder(OrderDTO o) {
Long orderId = oDao.save(o);
NotificationDTO n = new NotificationDTO();
n.setType(NotificationType.ORDER_CREATED);
n.setEntityId(orderId);
nDao.save(n);
}
}
UPDATE:
In most cases we can say that:
"Managers" may handle many DAOs;
DAOs should not contain other DAOs and are tied to a DTO;
DTOs can contain other DTOs
There is an important idea of LAZY or EAGER load when it comes to handling collections. But this is another subject :D
Disclaimer:
+ The following assumes that these DTOs are used mainly for persistence, i.e., for use with DAOs.
+ this approach is very oriented towards a relational database persistence
+ it is assumed a user can have placed orders, but that an order can have at most one user
+ also, that you want to query/process separatedly orders and users
I would have done the following:
a DTO for User (UserDTO + UserDAO)
a DTO for Orders (OrderDTO + OrderDAO)
a DTO to connect both (UserOrderDTO + UserOrderDAO)
I would not have references in the UserDTO to any OrderDTO
I may have a reference in the OrderDTO to the UserDTO as an attribute having a string id (being the string id the user id), but also I may not. I assume the later.
a Service Application to manage the different DAOs associated to the Order (OrderSA)
The resulting code would be as follows:
class OrderManagerServiceApplication {
private OrderDAO oDao;
private UserDao uDao;
private UserOrderDao uoDao;
public saveOrder(OrderDTO o, String userId) {
// Save the order
Long orderId = oDao.save(o);
// Save the association to the user who ordered
UserOrderDTO uodto=new UserOrderDTO(orderId,userId);
uoDao.save(uodto);
}
public List<OrderDTO> getOrdersForUser(String userId) {
// get the orders associated to the user
List<String> orderIds=uoDao.getAllForUser(userId);
// retrieve the order DTOs
ArrayList<OrderDTO> result=new ArrayList<OrderDTO>();
for (String orderId:orderIds){
result.add(oDAO.getOrder(orderId));
}
return result;
}
public UserDTO getUserForOrder(Stirng orderId) {
// get the user associated with the order
String userId=uoao.getUserForOrder(orderId);
// retrieve the user DTO
return uDAO.getUser(userId);
}
}