I am working with a spring boot. I am trying to send data from one database to the other.
First, I did this by making a get request to get the data from the first database and applied post through Web Client to send the data to the other database. It worked!
But when I tried to do it with cron scheduler with #Scheduled annotation it's not posting the data to the database. Even though the function is working fine, as i tried printing stuff through that function, but the WebClient is not posting the data (also checked the data, it was fine).
The Cron class is:
#Component
public class NodeCronScheduler {
#Autowired
GraphService graphService;
#Scheduled(cron = "*/10 * * * * *")
public void createAllNodesFiveSeconds()
{
graphService.saveAlltoGraph("Product");
}
}
saveAlltoGraph function takes all the tuples from a Product table and send post request to the api of graph database, which makes node from the tuples.
Here is the function:
public Mono<Statements> saveAlltoGraph(String label) {
JpaRepository currentRepository = repositoryService.getRepository(label);
List<Model> allModels = currentRepository.findAll();
Statements statements = statementService.createAllNodes(allModels, label);
//System.out.println(statements);
return webClientService.sendStatement(statements);
}
First, the label "Product" is used to get the JpaRepository related to that table. Then we fetch all the tuples of that table in the list, and we create objects according to that, (We can use a serializer to get the JSON).
Here is the sendStatement function:
public Mono<Statements> sendStatement(Statements statements){
System.out.println(statements);
return webClient.post().uri("http://localhost:7474/db/data/transaction/commit")
.body(Mono.just(statements), Statements.class).retrieve().bodyToMono(Statements.class);
}
Everything is working when we call this saveAlltoGraph using a get request mapping, but not working with the scheduler.
I tried with adding .block() and .subscribe() to that. And things started working with the cron scheduler.
Related
I have the following scenario. Below is a minimalist version of what I am trying to do
in a simple spring boot REST API Controller and Service.
func()
{
String lv=vService.getlv("1.2.1");
String mv=vService.getmv("1.3.1");
}
#Service
public class VService{
public String getlv(String version){
JsonArray lvVersions=makeHTTPGETLv();
String result=getVersion(lvVersions,version);
return result;
}
public String getMv(String version){
JsonArray mvVersions=makeHTTPGETMv();
String result=getVersion(mvVersions,version);
return result;
}
private String getVersion(JsonArray versions, String version){
Map<String,String> versionMap=new HashMap<>();
for(JsonElement je: versions){
JsonObject jo=je.getAsJsonObject();
//some logic
versionMap.put(jo.get("ver").getAsString(),jo.get("rver").getAsString);
}
return versionMap.get(version);
}
}
The getVersion method internally builds a map every time it is called.
The getlv method and getMv method both invokes the getVersion method
In the call getVersion(lvVersions,version); the lvVersions value will always be same,
It is a JSON array containing the mapping between versions.
And so each time getlv is called we are making a GET request makeHTTPGETLv and then
searching for the corresponding mapping of the given version.
Due to the static nature of this, it could be cached.
The multiple GET calls to makeHTTPGETLv could be avoided as it will always return the same JSON array (the changes could occur in some days though) and also the map that is built inside the getVersion method is unchanging.
Since the JsonArray could change, which is the response of a GET request,
So there could be a logic to update the cache every x minute. which could be 5 minutes or 60 mins.
How do I achieve this in spring boot?
Also to avoid taking the time for the first call, I could use Eager loading.
Steps Taken:
I tried using the #Cacheable annotation on top of the functions. It didn't have any effect. It seems I will have to put some more logic there. I could put out a separate function that builds the map that is being built inside the getVersion function. And so use this map for every call. But this map needs to be updated periodically (could be 5 mins, 60 mins, or even 1 day) which will require a GET call and building up the map. This reduces the problem of updating the Map every x minute. Because then I could directly use the map to fetch the corresponding version and avoid the GET call and parsing the response every time.
Similar Reducible/Smaller/alternate problem:
class Test {
private Map<String,String> map;
private void buildMap(){
}
}
The buildMap function updates the map attribute. Is there a way that buildMap function gets called every 30 minutes so that the map remains updated? That sounds like a cron job. Not sure if some caching is helpful here or if cron job and how to achieve that in spring boot.
I am trying to call a lot of APIs that needs a lot of time to finish. so I've tried to use threading to make it faster. I'm using Spring Boot, and the internet suggest me to use #Async but there's a problem. after using it for testing, I've realized that #Async only works for every async method calls. I don't know if I was mistaken about this, but overall I've created a controller that adds a lot of #Async method calls (up to 50 #Async method calls) and then run the CompletableFuture.allOf(apiCallsList.toArray(new CompletableFuture[0])).join()
Is there a way to make it simpler? because I don't know how to set the taskExecutor to limit the threadingPool if it was written like this.
Example code:
//get 50 person objects to the personPool
if (personPool.size() == 50) {
List<CompletableFuture<Map<String, String>>> results = new ArrayList<>();
for (Person person: personPool) {
//calls async api method call to a list
results.add(service.asyncApiProcess(service.callApi(person.getName())));
}
//run all async methods in paralel
CompletableFuture.allOf(results.toArray(new CompletableFuture[0])).join();
for (CompletableFuture<Map<String, String>> result : results) {
//write to xlsx
}
personPool.clear(); //clears the personPool for the next 50 data
}
I want to merge 2 responses and return a Flux.
private Flux<Response<List<Company>, Error>> loopGet(List<Entity> registries, Boolean status) {
return Flux.fromIterable(registries)
.flatMap(this::sendGetRequest)
.mergeWith(Mono.just(fetch(status)));
}
This is what I am doing, is working but I would like the merge to wait before calling the Mono.just (fetch (status)).
I'll explain, sendGetRequest returns a Mono that makes an API call and from the result saves things to db. Subsequently the merge goes to call the db with the fetch method, but that data is not updated yet. If I then make the call again, I get the updated data.
You need concatWith and fromCallable to ensure that fetch is called lazily after the get requests are finished.
private Flux<Response<List<Company>, Error>> loopGet(List<Entity> registries, Boolean status) {
return Flux.fromIterable(registries)
.flatMap(this::sendGetRequest)
.concatWith(Mono.fromCallable(() -> fetch(status)));
}
I would like to know a simple solution for receiving images and simple data in a single post using Spring. I am a beginner in Java so I would like to know the easy way. I've used several backend frameworks and I've encountered this problem in all of them.
I have the following problem:
I was receiving a multipart/form-data like this
public CasaVenda storeCasaVendaOld(#RequestParam("dormitorios") Integer dormitorios, #RequestParam("preco") Double preco, #RequestParam("foto_1") MultipartFile foto_1){
I receive some numbers along with an image. This is a typical first attempt of beginner's implementation.Validate will require code to be writeen in the controller and I have to receive far more parameters than described here, so it's a bad implementation.
I thought about receiving a model
public CasaVenda storeCasaVenda(#Valid #RequestBody CasaVenda casa)
Now I can validate using annotations and so. The problem is with the file. Is there a simple solution to receive the file in one post request or should I split the process of seding the overall data and the files spareted? I mean I can make the process of the resource creation two steps, first it enters the overall data and afterwards it includes the photos.
Its pretty easy to define an object:
public class MyObject {
private Integer dormitorios;
private Double preco;
...
getters/setters/constructors/etc.
...
// I'm not sure whether you can place a MultipartFile here as well to process image,
// however it doesn't make sense to validate it anyway
}
Then you can use this object in the controller, it will map all the query params to the fields of the object automatically by spring:
public CasaVenda storeCasaVendaOld(MyObject myObject) {
}
Now, you can place Validation annotations inside MyObject and it will be validated, just do not use #RequestParam annotation before the object...
I'm creating a Rest Api with Spring Boot. I'm using the default configuration of Hikari, so it has a pool size of 10 connections.
I encountered an error when i try to post 10 parallel requests to a specific route. The error says Connection is not available, request timed out after 30001ms.
This route saves data into a MySQL database, it usually takes some milliseconds to complete one save operation.
Why is this problem happening? Should it complete the save operation and then free the database connection for the next operation?
This error seems to appear only with saving operations where the save function creates a new entity.
Properties
spring.jpa.hibernate.ddl-auto=create-drop
spring.datasource.url=jdbc:mysql://192.168.1.88:3306/question?serverTimezone=UTC
spring.datasource.username=luigi
spring.datasource.password=root
Repository
public interface RandomRepository extends CrudRepository<Random, Integer> {
}
Controller
#RestController
public class RandomController {
private RandomRepository randomRepository;
public RandomController(RandomRepository randomRepository) {
this.randomRepository = randomRepository;
}
#GetMapping("/")
public String createRandom() {
return String.valueOf(Math.random());
}
#PostMapping("save")
public Random save(){
Random random = new Random();
random.setNumber(Math.random());
randomRepository.save(random);
return random;
}
}
I've found a solution by making the save method synchronized. Is this the right way? Did someone encountered the same problem?
Possible solution
#PostMapping("save")
public synchronized Random save(){
Random random = new Random();
random.setNumber(Math.random());
randomRepository.save(random);
return random;
}
I expected that the save operation would be completed easly, but it stucks until it crashes after 30 seconds
We faced a very similar issue when working with Spring Boot & Couchbase.
When the number of requests was high, the connection to the DB got stuck.
The solution that we used was to move to Async method calls for all levels - from the controller down to the DB operations.
See this post & answer