Small question regarding SpringBoot and SpringData, and how to save a pojo into many databases concurrently, in parallel please.
I have a very simple SpringBoot application which does nothing but expose a rest endpoint to save a pojo:
#RestController
public class SaveController {
#Autowired
MyElasticRepository myElasticRepository;
#Autowired
MyMongoRepository myMongoRepository;
#Autowired
MyAARepository myAARepository;
//#Autowired MyBBRepository, MyCCRepository, ... MyYYRepository
#Autowired
MyZZRepository myZZRepository;
#GetMapping("/saveSequential")
public String saveSequential(#RequestBody MyPojo myPojo) {
MyPojo myPojoFromElastic = myElasticRepository.save(myPojo);
MyPojo myPojoFromMongo = myMongoRepository.save(myPojo);
MyPojo myPojoFromAA = myAARepository.save(myPojo);
// myBBRepository.save(myPojo) myCCRepository.save(myPojo) ... myYYRepository.save(myPojo)
MyPojo myPojoFromZZ = myZZRepository.save(myPojo);
return ...;
}
}
However, the pojo needs to be saved in many databases, by many, imagine a good dozens of different databases.
As of now, as you can see from the code, the pojo is saved in each of the databases sequentially. I timed the application, as well as monitoring the DBs, the inserts come one after another.
Hypothetically, if one save takes one second, and I have 20 DB, the rest endpoints takes 20ish seconds to complete.
Since the operation is not dependent of any others, i.e. saving the pojo in Mongo, has no dependency on the data saved in Oracle, etc... I would like to optimize the performance by doing the operation in parallel.
I.e, if each save takes one second, and I have 20 DBs, to parallel the save, which should still take something like oneish second. (I am exaggerating)
For the sake of the question, let us imagine the machine doing the save has many cores, is a very good machine, etc.
What I tried:
I tried using the #Async annotation on the repository, such as:
#Async
#Repository
public interface MyElasticRepository extends ElasticsearchRepository<MyPojo, String> {
}
But unfortunately, timing the endpoint, it still takes a sequential time.
May I ask how to achieve this parallel, concurrent save please?
If possible, I would like to leverage existing features of Spring Framework, and not having to rewrite boiler plate concurrency code.
Thank you!
I think you are best off creating a service layer with an async method.
import org.springframework.data.repository.Repository;
import java.util.concurrent.CompletableFuture;
import org.springframework.stereotype.Service;
import org.springframework.scheduling.annotation.AsyncResult;
#Service
public class MyPojoPersister {
#Async
#CompletableFuture<MyPojo> savePojo(Repository repo, MyPojo pojo) {
return CompletableFuture.completedFuture(repo.save(pojo));
}
}
Then your controller would look something like this:
#RestController
public class SaveController {
#Autowired
MyElasticRepository myElasticRepository;
#Autowired
MyMongoRepository myMongoRepository;
#Autowired
MyAARepository myAARepository;
//#Autowired MyBBRepository, MyCCRepository, ... MyYYRepository
#Autowired
MyZZRepository myZZRepository;
#Autowired MyPojoPersister myPojoPersister;
#GetMapping("/saveSequential")
public String saveSequential(#RequestBody MyPojo myPojo) {
var futureList = Stream.of(myElasticRepository, myMongoRepository, myAARepository, myZZRepository)
.map(repo -> myPojoPersister.savePojo(repo, myPojo))
.collect(Collectors.toList());
CompletableFuture.allOf(futureList.toArray(new CompletableFuture[list.size()])).join();
var someString = futureList.stream()
.map(CompletableFuture::get())
.map(MyPojo::getId())
.collect(Collectors.joining(","));
return someString;
}
}
I added some assumptions that you want to return a comma separated list of the ids of the pojos since they would presumably be different for each repo. But do whatever you need to with the values of the futures.
Don't forget to enable asynchronicity!
#SpringBootApplication
#EnableAsync
public class MyPojoApplication {
public static void main(String[] args) {
SpringApplication.run(AsyncMethodApplication.class, args).close();
}
}
Related
Let's imagine a scenario where I have a Spring Boot app (in a controller/service/repository pattern) which contains controller for cars. Now I would like to create paths that look for example like this:
"/api/cars/bmw"
"/api/cars/mercedes"
"/api/cars/audi"
And so on. And for each of these car producers I would like to have multiple endpoints, some of them common for all car producers (not sure if it really matters but just for the sake of it lets say for example "/order" and "/cancelOrder"), but some of them different.
Now what would be the proper way to implement this scenario? Is there a way to make a CarController for the /car/{producer} path which would be a proxy for other controllers like AudiController handling /car/audi requests? Having 3 car producers in one controller and a service for each car producer is ok, but having 30 would make a lot of injected dependencies (30 services injected into the controller) if I would have only one CarController.
I believe node.js Express framework would allow us to do this in the car "controller" script (not sure if they are called controllers in express or what):
var app = express();
app.use('/audi', '{path to audi controller script}');
Is there a similar possibility in Spring? Or is this maybe a bad idea, an antypattern?
Another idea that is quite simple but seems not that elegant is to skip the CarController and implement a:
AudiController with the #RequestMapping("/api/cars/audi")
BmwController with the #RequestMapping("/api/cars/bmw")
MercedesController with the #RequestMapping("/api/cars/mercedes")
etc.
So, what would be the best idea here?
Spring has #PathVariable for this, which can be used in the following way:
#RestController
#RequestMapping("/api/cars/")
public class MyController {
#GetMapping
#RequestMapping("{producer}/cancelOrder")
public String cancelOrder(#PathVariable String producer) {
return "Cancel order for " + producer;
}
}
We can call this endpoint like this: http://localhost:8080/api/cars/Mercedes/cancelOrder
There is no such thing as child controller in Spring. If you want to create a separate controller for every producer having some common functionality, you can use inheritance:
Parent controller class (please note that this does not have any Controller annotation):
#RequestMapping("/api/cars/")
public class BaseCarController {
#GetMapping("/common")
public String common() {
return "Common stuff";
}
}
Child controller classes:
#RestController
public class MercedesController extends BaseCarController{
private MercedesService mercedesService;
// Spring will autowire this by default, no need to add #Autowired
public MercedesController(MercedesService mercedesService) {
this.mercedesService = mercedesService;
}
#GetMapping
#RequestMapping("Mercedes/cancelOrder")
public String cancelOrder() {
return "Cancel order for Mercedes.";
}
}
#RestController
public class AudiController extends BaseCarController{
private AudiService audiService;
// Spring will autowire this by default, no need to add #Autowired
public AudiController(AudiService audiService) {
this.audiService = audiService;
}
#GetMapping
#RequestMapping("Audi/cancelOrder")
public String cancelOrder() {
return "Cancel order for Audi.";
}
}
We can call the produces specific endpoints like this:
http://localhost:8080/api/cars/Mercedes/cancelOrder or http://localhost:8080/api/cars/Audi/cancelOrder. Moreover we can call the common endpoint in the following way: http://localhost:8080/api/cars/common
While working with a project that involves requesting multiple data types from a database I came to a following question:
Lets say I have 2 java classes that correspond to database entities:
Routes
public class Route {
public Route(int n, int region, Date fdate, boolean changed, int points,
int length) {
super();
this.n = n;
this.region = region;
this.fdate = fdate;
this.changed = changed;
this.points = points;
this.length = length;
}
}
Carrier
public class Carrier {
public Carrier(...) {
this.id = src.getId();
this.name = src.getName();
this.instId = src.getInstId();
this.depotId = src.getDepotId();
}
If so, what's the correct approach of creating Dao interfaces and classes? I'm doing it like this -
#Repository
public class CarrierDaoImpl implements CarrierDao{
#Autowired
DataSource dataSource;
public List<Carrier> getAllOrgs() { ... }
}
#Repository
public class RoutesDaoImpl implements RoutesDao {
#Autowired
DataSource dataSource;
public ArrayList<AtmRouteItem> getRoutes(AtmRouteFilter filter) { ... }
}
I'm creating a #Repository DAO for every java class item\db entity and then 2 separate controllers for requests about carriers and routes. Like this:
#RestController
#RequestMapping(path = "/routes")
public class RoutesController {
#Autowired
RoutesDao routesDao;
#GetMapping(value = {"/getRoutes/", "/getRoutes"})
public ArrayList<Route> getRoutes() { ... } }
And same for controller Carriers. Is it correct and if not what's the correct approach?
Sorry for styling issues, that's my first question on stackoverflow :)
I would suggest creating services marked with #Service annotation (i.e. CarrierService interface and CarrierServiceImpl implementation). Than inject them into controllers. Use repositories within services because some database operations will require transactions and a better place for managing transactions are services. Also services can do more specialized job which will require access to multiple repositories so you can inject them. And don’t forget to mark your services with #Transactional annotation.
It's correct to have a DAO for each entity.
When working with JPA repositories you have no choice but to provide the entity. For instance:
public interface FooRepository extends JpaRepository<Foo,Long>{}
Same for the REST controllers, you have to bring together functionalities by object as you do.
You can improve your mapping to be more RESTful. To retrieve all routes, don't specify a path:
#GetMapping
public ArrayList<RouteResource> getRoutes() { ... }
(I never use #GetMapping yet but it should work like that)
And if you want specific route:
#GetMapping("/get/{id}")
public RouteResource getRoute() {...}
You should return resources instead of entities to client.
I have implemented a CodeService which will retrieve a list of countries from the code table.
In my ShippingService, I would like to check if the order is shipped to a certain country.
In this case, should I be using CodeService or CodeDAO to retrieve the list of countries.
public interface CodeService {
public List<String> getCountryList();
}
#Service
public class CodeServiceImpl implements CodeService {
#Autowired
CodeDAO codeDao
public List<String> getCountryList() {
return codeDao.getCountryList();
}
}
#Service
public class ShippingServiceImpl implements ShippingService {
#Autowired
CodeDAO codeDao;
#Autowired
CodeSevice codeService;
public void addOrder(Order order) {
List<String> countries = codeService.getCountryList();
//List<String> countries = codeDao.getCountryList();
}
}
If you dont have any additional logic such as in this case, I think its best to call directly to the DAO.
Pros: Calling always the service, would allow you to seperate completly the DAO layer from you app code.
Cons: you will create 2 redundant classes for each dao, that will only delegate the call to the DAO.
The services layer should be used for business logic over the fetched data. For example: if you want to add permissions to the fetched countries.
If additional logic would be added to the countries in the future, its best to do a refactoring and create the service.
I'm not sure where to open my Transaction object. Inside the service layer? Or the controller layer?
My Controller basically has two services, let's call them AService and BService. Then my code goes something like:
public class Controller {
public AService aService = new AService();
public BService bService = new BService();
public void doSomething(SomeData data) {
//Transaction transaction = HibernateUtil.getSession().openTransaction();
if (data.getSomeCondition()) {
aService.save(data.getSomeVar1());
bService.save(data.getSomeVar2());
}
else {
bService.save(data.getSomeVar2());
}
//transaction.commit(); or optional try-catch with rollback
}
}
The behavior I want is that if bService#save fails, then I could invoke a transaction#rollback so that whatever was saved in aService would be rolled back as well. This only seems possible if I create one single transaction for both saves.
But looking at it in a different perspective, it looks really ugly that my Controller is dependent on the Transaction. It would be better if I create the Transaction inside the respective services, (something like how Spring #Transactional works), but if I do it that way, then I don't know how to achieve what I want to happen...
EDIT: Fixed code, added another condition. I am not using any Spring dependencies so the usage of #Transactional is out of the question.
You can accomplish what you're asking with another layer of abstraction and using composition.
public class CompositeABService {
#Autowired
private AService aservice;
#Autowired
private BService bservice;
#Transactional
public void save(Object value1, Object value2) {
aservice.save( value1 );
bservice.save( value2 );
}
}
public class AService {
#Transactional
public void save(Object value) {
// joins an existing transaction if one exists, creates a new one otherwise.
}
}
public class BService {
#Transactional
public void save(Object value) {
// joins an existing transaction if one exists, creates a new one otherwise.
}
}
This same pattern is typically used when you need to interact with multiple repositories as a part of a single unit of work (e.g. transaction).
Now all your controller needs to depend upon is CompositeABService or whatever you wish to name it.
What would be a somewhat equivalent class for TransactionScope(.Net) in Spring.
I know absolutely nothing about .Net, so I'm not certain if this is what you're looking for. You can use SimpleTransactionScope in order to maintain objects across the lifecycle of a transaction. It is not registered by default, so you will have to register it with spring core like any custom scope, and give it a stringy name. Then if you want you can also create an annotation specifically to register one.
It was my issue that suggested this a few years ago, after this question was created for certain. We requested it specifically for timestamps across multiple methods for injection in a service. You can do something like this.
public class SimpleTransactionFactoryPostProcessor implements BeanFactoryPostProcessor {
#Override
public void postProcessBeanFactory(ConfigurableListableBeanFactory factory) throws BeansException {
factory.registerScope("transaction", new SimpleTransactionScope());
}
}
#Configuration
class MyConfiguration {
#Scope(scopeName = "transaction")
#Bean
Instant nowInstant() {
return Instant.now();
}
}
#Service
class MyService {
private final ObjectFactory<Instant> nowFactory;
MyService( #Qualifier("nowInstant") ObjectFactory<Instant> nowFactory ) {
this.nowFactory = nowfactory
}
#Transactional
public boolean nowisEqualAlways() {
var now = nowFactory.getObject();
var sameNow = nowFactory.getObject();
return Objects.equals( now, sameNow );
}
}
If you don't do this, your now could actually change during your transaction by a small amount of time. You can test that simply by spamming now calls in a test.
It may not be required for your needs, so it's hard for me to tell (obviously your needs are probably long past, hopeful.y this helps someone in the future though)
The #Transactional annotation looks equivalent.
This can be placed on classes and methods and can be defined with propagation, isolation, rollback etc.