I noticed a significant performance degradation when upgrading Spring-Boot from 1.4.2 to 1.4.3 while using spring-integration.
I could reproduce the problem with the following program:
pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<!--<version>1.4.2.RELEASE</version>-->
<version>1.4.3.RELEASE</version>
</parent>
<artifactId>performance-test</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-integration</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
</project>
Application.java:
package performance.test;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
#SpringBootApplication
public class Application {
public static void main(final String[] args) {
SpringApplication.run(Application.class, args);
}
}
UsersManager.java:
package performance.test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.integration.annotation.MessageEndpoint;
import org.springframework.integration.annotation.ServiceActivator;
import org.springframework.messaging.handler.annotation.Payload;
#MessageEndpoint
public class UsersManager {
private static final Logger LOGGER = LoggerFactory.getLogger(UsersManager.class);
#ServiceActivator(inputChannel = "testChannel")
public void process(#Payload String payload) {
LOGGER.debug("payload: {}", payload);
}
}
IntegrationTest.java:
package performance.test;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.SpringApplicationConfiguration;
import org.springframework.messaging.MessageChannel;
import org.springframework.messaging.support.MessageBuilder;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
#RunWith(SpringJUnit4ClassRunner.class)
#SpringApplicationConfiguration(classes = Application.class)
public class IntegrationTest {
private static final Logger LOGGER = LoggerFactory.getLogger(IntegrationTest.class);
#Autowired
private MessageChannel testChannel;
#Test
public void basic_test() {
for (int i = 1; i <= 10; i++) {
final long start = System.currentTimeMillis();
test(1000);
final long end = System.currentTimeMillis();
LOGGER.info("test run: {} took: {} msec", i, end - start);
}
}
private void test(int count) {
for (int i = 0; i < count; i++) {
testChannel.send(MessageBuilder.withPayload("foo").setHeader("monkey", "do").build());
}
}
}
When I use Spring-Boot 1.4.2, I get the following results:
2016-12-27 14:55:39.331 INFO 4939 --- [ main] performance.test.IntegrationTest : Started IntegrationTest in 0.975 seconds (JVM running for 1.52)
2016-12-27 14:55:39.468 INFO 4939 --- [ main] performance.test.IntegrationTest : test run: 1 took: 123 msec
2016-12-27 14:55:39.552 INFO 4939 --- [ main] performance.test.IntegrationTest : test run: 2 took: 83 msec
2016-12-27 14:55:39.632 INFO 4939 --- [ main] performance.test.IntegrationTest : test run: 3 took: 80 msec
2016-12-27 14:55:39.696 INFO 4939 --- [ main] performance.test.IntegrationTest : test run: 4 took: 63 msec
2016-12-27 14:55:39.763 INFO 4939 --- [ main] performance.test.IntegrationTest : test run: 5 took: 67 msec
2016-12-27 14:55:39.842 INFO 4939 --- [ main] performance.test.IntegrationTest : test run: 6 took: 78 msec
2016-12-27 14:55:39.895 INFO 4939 --- [ main] performance.test.IntegrationTest : test run: 7 took: 53 msec
2016-12-27 14:55:39.950 INFO 4939 --- [ main] performance.test.IntegrationTest : test run: 8 took: 55 msec
2016-12-27 14:55:40.004 INFO 4939 --- [ main] performance.test.IntegrationTest : test run: 9 took: 54 msec
2016-12-27 14:55:40.057 INFO 4939 --- [ main] performance.test.IntegrationTest : test run: 10 took: 53 msec
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.776 sec - in performance.test.IntegrationTest
However, when I use Spring-Boot 1.4.3, there is significant performance degradation:
2016-12-27 14:57:41.705 INFO 5122 --- [ main] performance.test.IntegrationTest : Started IntegrationTest in 1.002 seconds (JVM running for 1.539)
2016-12-27 14:57:41.876 INFO 5122 --- [ main] performance.test.IntegrationTest : test run: 1 took: 156 msec
2016-12-27 14:57:42.031 INFO 5122 --- [ main] performance.test.IntegrationTest : test run: 2 took: 153 msec
2016-12-27 14:57:42.251 INFO 5122 --- [ main] performance.test.IntegrationTest : test run: 3 took: 220 msec
2016-12-27 14:57:42.544 INFO 5122 --- [ main] performance.test.IntegrationTest : test run: 4 took: 293 msec
2016-12-27 14:57:42.798 INFO 5122 --- [ main] performance.test.IntegrationTest : test run: 5 took: 254 msec
2016-12-27 14:57:43.111 INFO 5122 --- [ main] performance.test.IntegrationTest : test run: 6 took: 312 msec
2016-12-27 14:57:43.544 INFO 5122 --- [ main] performance.test.IntegrationTest : test run: 7 took: 432 msec
2016-12-27 14:57:44.112 INFO 5122 --- [ main] performance.test.IntegrationTest : test run: 8 took: 567 msec
2016-12-27 14:57:44.807 INFO 5122 --- [ main] performance.test.IntegrationTest : test run: 9 took: 695 msec
2016-12-27 14:57:45.620 INFO 5122 --- [ main] performance.test.IntegrationTest : test run: 10 took: 813 msec
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.989 sec - in performance.test.IntegrationTest
I have no idea what could be the cause for this.
Are the other persons having the same issue?
This might be related to SPR-14929.
In this case, removing #Payload from the message parameter solves the issue (it is not needed in this case - you only need a payload annotation if there are multiple parameters).
Related
I'm using Optaplanner for a hospital bed allocation project. I'm working with spring boot, spring JPA and postgres as a DB. I've set all classes with #PlanningEntity and #PlanningSolution annotations, set
#PlanningVariable, write the constraints with drool and configure it with '.xml' file .After solving the value of planning variable, which in my cas"bed" , doesn't change!!
this is the main class:
#SpringBootApplication(scanBasePackages = { "com.asma.optaplanner.demo" })
public class OptaPlannerDemoApplication {
public static void main(String[] args) {
SpringApplication.run(OptaPlannerDemoApplication.class, args);
}
#Bean
public CommandLineRunner demoData(PatientAdmissionScheduleHelper helper) {
return args -> {
SolverFactory<PatientAdmissionSchedule> solverfactory = SolverFactory.createFromXmlResource("Solver_Config.xml");
helper.initilizeDataBase();
System.out.println("before solving");
Solver<PatientAdmissionSchedule> solver = solverfactory.buildSolver();
PatientAdmissionSchedule unsolvedSchedule = helper.getSchedule();
unsolvedSchedule.getAdmissionList().forEach(admission -> {
System.out.println(admission.toString());
});
solver.solve(unsolvedSchedule);
PatientAdmissionSchedule solvedSchedule = solver.getBestSolution();
System.out.println("after solving");
solvedSchedule.getAdmissionList().forEach(admission -> {
System.out.println(admission.toString());
});
};
}
the result:
before solving
2020-06-03 21:33:05.256 WARN 9228 --- [ main] o.d.c.kie.builder.impl.KieBuilderImpl : File 'Constraint.drl' is in folder '' but declares package 'com.asma.optaplanner.demo'. It is advised to have a correspondance between package and folder names.
PatientName patient1bed=Bed [externalCode= bed11, room= Room [name=room1, capacity=2], indexInRoom=1], From 1, To5
PatientName patient2bed=Bed [externalCode= bed12, room= Room [name=room1, capacity=2], indexInRoom=2], From 2, To4
2020-06-03 21:33:06.208 INFO 9228 --- [ main] o.o.core.impl.solver.DefaultSolver : Solving started: time spent (77), best score (-80hard/0soft), environment mode (REPRODUCIBLE), random (JDK with seed 0).
2020-06-03 21:37:06.210 INFO 9228 --- [ main] o.o.c.i.l.DefaultLocalSearchPhase : Local Search phase (0) ended: time spent (240079), best score (0hard/0soft), score calculation speed (38961/sec), step total (418).
2020-06-03 21:37:06.222 INFO 9228 --- [ main] .c.i.c.DefaultConstructionHeuristicPhase : Construction Heuristic phase (1) ended: time spent (240091), best score (0hard/0soft), score calculation speed (1222/sec), step total (2).
2020-06-03 21:37:06.222 INFO 9228 --- [ main] o.o.core.impl.solver.DefaultSolver : Solving ended: time spent (240091), best score (0hard/0soft), score calculation speed (38946/sec), phase total (2), environment mode (REPRODUCIBLE).
after solving
2020-06-03 21:37:06.226 INFO 9228 --- [ main] ConditionEvaluationReportLoggingListener :
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2020-06-03 21:37:06.238 ERROR 9228 --- [ main] o.s.boot.SpringApplication : Application run failed
java.lang.IllegalStateException: Failed to execute CommandLineRunner
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:798) [spring-boot-2.3.1.BUILD-SNAPSHOT.jar:2.3.1.BUILD-SNAPSHOT]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:779) [spring-boot-2.3.1.BUILD-SNAPSHOT.jar:2.3.1.BUILD-SNAPSHOT]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:322) [spring-boot-2.3.1.BUILD-SNAPSHOT.jar:2.3.1.BUILD-SNAPSHOT]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237) [spring-boot-2.3.1.BUILD-SNAPSHOT.jar:2.3.1.BUILD-SNAPSHOT]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) [spring-boot-2.3.1.BUILD-SNAPSHOT.jar:2.3.1.BUILD-SNAPSHOT]
at com.asma.optaplanner.demo.OptaPlannerDemoApplication.main(OptaPlannerDemoApplication.java:18) [classes/:na]
Caused by: java.lang.NullPointerException: null
at com.asma.optaplanner.demo.model.Admission.toString(Admission.java:120) ~[classes/:na]
at com.asma.optaplanner.demo.OptaPlannerDemoApplication.lambda$2(OptaPlannerDemoApplication.java:40) [classes/:na]
at java.util.ArrayList.forEach(Unknown Source) ~[na:1.8.0_171]
at com.asma.optaplanner.demo.OptaPlannerDemoApplication.lambda$0(OptaPlannerDemoApplication.java:39) [classes/:na]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:795) [spring-boot-2.3.1.BUILD-SNAPSHOT.jar:2.3.1.BUILD-SNAPSHOT]
... 5 common frames omitted
Constraint.drl :
package com.asma.optaplanner.demo;
//list any import classes here.
dialect "java"
import com.asma.optaplanner.demo.model.Admission ;
import com.asma.optaplanner.demo.model.AdmissionDemand ;
import com.asma.optaplanner.demo.model.Room ;
import com.asma.optaplanner.demo.model.Bed ;
import org.optaplanner.core.api.score.buildin.hardsoft.HardSoftScoreHolder;
//declare any global variables here
global HardSoftScoreHolder scoreHolder;
//Hard Constraints
//same gender at the same room in the same night
rule "SameRoomGenderconstraint"
when
$leftAdmission: Admission(
bed != null,
$room : Room,
$leftFrom : DateFromIndex,
$leftTo : DateToIndex,
$leftGender : gender)
$rightAdmission : Admission(
room == $room,
DateToIndex >= $leftFrom ,
DateFromIndex <= $leftTo ,
$rightFrom : DateFromIndex,
$rightTo : dateToIndex,
gender == $leftGender)
then
scoreHolder.addHardConstraintMatch(kcontext,
-10 * (1 + Math.max($leftTo, $rightTo) - Math.max($leftFrom, $rightFrom)));
end
rule "2PatientInTheSameBed"
//include attributes such as "salience" here...
when
$leftAdmission: Admission(
bed != null,
$bed : bed,
$leftFrom : dateFromIndex,
$leftTo : dateToIndex,
$leftId : id)
$rightAdmission: Admission(
bed == $bed,
dateToIndex >= $leftFrom ,
dateFromIndex <= $leftTo ,
$rightFrom : dateFromIndex,
$rightTo : dateToIndex,
id != $leftId)
then
scoreHolder.addHardConstraintMatch(kcontext,
-5 * (1 + Math.max($leftTo, $rightTo) - Math.max($leftFrom, $rightFrom)));
end
In case your #PlanningSolution.PlanningEntity :(#PlanningEntityCollectionProperty / #PlanningEntityProperty) don't change, you should review your drool file, it might be a bit hard to debug .drl files, u might try ConstraintProvider intreface via java, it will be easier to understand solving routine/rule.
Plus, the change is related to "ranged properties" annotated "#ValueRangeProvider", from which you can plan/optimize your solution.
As you're creating a SolverFactory in Spring Boot manually (instead of autowiring it with the optaplanner-spring-boot-starter), do pass the ClassLoader parameter to avoid common issues.
If you copied optaplanner-example's PatientAdmissionSchedule domain classes, note that it has nullable=true on the #PlanningVariable, so it can return unassigned entities. In fact, if you don't have a constraint (typically a medium constraint) to minimize that, all entities are likely to be assigned to null.
I want to create a service that combines results from two reactive sources.
One is producing Mono and another one is producing Flux. For merging I need the same value of mono for every flux emitted.
For now I have something like this
Flux.zip(
service1.getConfig(), //produces flux
service2.getContext() //produces mono
.cache().repeat()
)
This gives me what I need,
service2 is called only once
context is provided for every configuration
resulting flux has as many elements as configurations
But I have noticed that repeat() is emitting a massive amount of elements after context is cached. Is this a problem?
Is there something I can do to limit number of repeats to the number of received configurations, yet still do both request simultaneously?
Or this is not an issue and I Can safely ignore those extra emitted elements?
I tried to use combineLatest but depending on timing I some elements fo configuration can get lost and not processed.
EDIT
Looking at the suggestions from #Ricard Kollcaku I have created sample test that shows why this is not what I'm looking for.
import java.util.concurrent.atomic.AtomicLong;
import java.util.stream.Stream;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
import reactor.core.scheduler.Schedulers;
import reactor.test.StepVerifier;
public class SampleTest
{
Logger LOG = LoggerFactory.getLogger(SampleTest.class);
AtomicLong counter = new AtomicLong(0);
Flux<String> getFlux()
{
return Flux.fromStream(() -> {
LOG.info("flux started");
sleep(1000);
return Stream.of("a", "b", "c");
}).subscribeOn(Schedulers.parallel());
}
Mono<String> getMono()
{
return Mono.defer(() -> {
counter.incrementAndGet();
LOG.info("mono started");
sleep(1000);
return Mono.just("mono");
}).subscribeOn(Schedulers.parallel());
}
private void sleep(final long milis)
{
try
{
Thread.sleep(milis);
}
catch (final InterruptedException e)
{
e.printStackTrace();
}
}
#Test
void test0()
{
final Flux<String> result = Flux.zip(
getFlux(),
getMono().cache().repeat()
.doOnNext(n -> LOG.warn("signal on mono", n)),
(s1, s2) -> s1 + " " + s2
);
assertResults(result);
}
#Test
void test1()
{
final Flux<String> result =
getFlux().flatMap(s -> Mono.zip(Mono.just(s), getMono(),
(s1, s2) -> s1 + " " + s2));
assertResults(result);
}
#Test
void test2()
{
final Flux<String> result = getFlux().flatMap(s -> getMono().map((s1 -> s + " " + s1)));
assertResults(result);
}
void assertResults(final Flux<String> result)
{
final Flux<String> flux = result;
StepVerifier.create(flux)
.expectNext("a mono")
.expectNext("b mono")
.expectNext("c mono")
.verifyComplete();
Assertions.assertEquals(1L, counter.get());
}
Looking at the test results for test1 and test2
2020-01-20 12:55:22.542 INFO [] [] [ parallel-3] SampleTest : flux started
2020-01-20 12:55:24.547 INFO [] [] [ parallel-4] SampleTest : mono started
2020-01-20 12:55:24.547 INFO [] [] [ parallel-5] SampleTest : mono started
2020-01-20 12:55:24.548 INFO [] [] [ parallel-6] SampleTest : mono started
expected: <1> but was: <3>
I need to reject your proposal. In both cases getMono is
- invoked as many times as items in flux
- invoked after first element of flux arrives
And those are interactions that I want to avoid. My services are making http requests under the hood and they may be time consuming.
My current solution does not have this problem, but if I add logger to my zip I will get this
2020-01-20 12:55:20.505 INFO [] [] [ parallel-1] SampleTest : flux started
2020-01-20 12:55:20.508 INFO [] [] [ parallel-2] SampleTest : mono started
2020-01-20 12:55:21.523 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.528 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.529 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.529 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.529 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.529 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.530 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.530 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.530 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.530 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.531 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.531 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.531 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.531 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.531 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.532 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.532 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.532 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.532 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.533 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.533 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.533 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.533 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.533 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.533 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.533 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.534 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.534 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.534 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.534 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.534 WARN [] [] [ parallel-2] SampleTest : signal on mono
2020-01-20 12:55:21.535 WARN [] [] [ parallel-2] SampleTest : signal on mono
As you can see there is a lot of elements emitted by combining cache().repeat() together and I want to know if this is an issue and if yes then how to avoid it (but keep single invocation of mono and parallel invocation).
I think what you are trying to achieve could be done with Flux.join
Here is some example code:
Flux<Integer> flux = Flux.concat(Mono.just(1).delayElement(Duration.ofMillis(100)),
Mono.just(2).delayElement(Duration.ofMillis(500))).log();
Mono<String> mono = Mono.just("a").delayElement(Duration.ofMillis(50)).log();
List<String> list = flux.join(mono, (v1) -> Flux.never(), (v2) -> Flux.never(), (x, y) -> {
return x + y;
}).collectList().block();
System.out.println(list);
Libraries like Project Reactor and RxJava try to provide as much combinations of their capabilities as possible, but do not provide access to the instruments of combining capabilities. And as a result, there are always corner cases which are not covered.
My own DF4J, as far as I know, is the only asynchronous library which provides the means to combine capabilities. For example, this is how user can zip Flux and Mono: (of course, this class is not part of DF4J itself):
import org.df4j.core.dataflow.Actor;
import org.df4j.core.port.InpFlow;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
abstract class ZipActor<T1, T2> extends Actor {
InpFlow<T1> inpFlow = new InpFlow<>(this);
InpFlow<T2> inpScalar = new InpFlow<>(this);
ZipActor(Flux<T1> flux, Mono<T2> mono) {
flux.subscribe(inpFlow);
mono.subscribe(inpScalar);
}
#Override
protected void runAction() throws Throwable {
if (inpFlow.isCompleted()) {
stop();
return;
}
T1 element1 = inpFlow.removeAndRequest();
T2 element2 = inpScalar.current();
runAction(element1, element2);
}
protected abstract void runAction(T1 element1, T2 element2);
}
and this is how it can be used:
#Test
public void ZipActorTest() {
Flux<Integer> flux = Flux.just(1,2,3);
Mono<Integer> mono = Mono.just(5);
ZipActor<Integer, Integer> actor = new ZipActor<Integer, Integer>(flux, mono){
#Override
protected void runAction(Integer element1, Integer element2) {
System.out.println("got:"+element1+" and:"+element2);
}
};
actor.start();
actor.join();
}
The console output is as follows:
got:1 and:5
got:2 and:5
got:3 and:5
You can do it with just a simple change
getFlux()
.flatMap(s -> Mono.zip(Mono.just(s),getMono(), (s1, s2) -> s1+" "+s2))
.subscribe(System.out::println);
Flux<String> getFlux(){
return Flux.just("a","b","c");
}
Mono<String> getMono(){
return Mono.just("mono");
}
if you want to use zip but you can achieve same results using flatmap
getFlux()
.flatMap(s -> getMono()
.map((s1 -> s + " " + s1)))
.subscribe(System.out::println);
}
Flux<String> getFlux() {
return Flux.just("a", "b", "c");
}
Mono<String> getMono() {
return Mono.just("mono");
}
in both result is:
a mono
b mono
c mono
EDIT
Ok now i understand it better. Can you try this solution.
getMono().
flatMapMany(s -> getFlux().map(s1 -> s1 + " " + s))
.subscribe(System.out::println);
Flux<String> getFlux() {
return Flux.defer(() -> {
System.out.println("init flux");
return Flux.just("a", "b", "c");
});
}
Mono<String> getMono() {
return Mono.defer(() -> {
System.out.println("init Mono");
return Mono.just("sss");
});
}
I se spring boot 2.
I have a method who use a scheduler.
I try to log some info
#Scheduled(cron = "0 52 8 * * *")
private void sendEmailOrders() throws MessagingException {
List<FactoryEmailNCDto> factoryEmails = prepareDataNoncompliantSampling();
log.info("start sendEmailOrders: " + factoryEmails != null ? String.valueOf(factoryEmails.size()) : "0");
for (FactoryEmailNCDto factoryEmail : factoryEmails) {
String message = mailContentBuilder.build(factoryEmail);
if (factoryEmail.getEmails() != null && !factoryEmail.getEmails().isEmpty()) {
log.info("prepare to sent email to : " + factoryEmail.getFactoryName());
mailService.sendHtmlMail(factoryEmail.getEmails(), "Order", message);
}
}
log.info("end sendEmailOrders");
}
I get
2019-03-19 08:52:00.379 INFO 17831 --- [ scheduling-1]
com.mermacon.facade.OrdersFacade : 2 2019-03-19 08:52:00.760 INFO
17831 --- [ scheduling-1] com.mermacon.facade.OrdersFacade :
prepare to sent email to : NY 2019-03-19 08:52:03.898 ERROR 17831 ---
[ scheduling-1] o.s.s.s.TaskUtils$LoggingErrorHandler :
Unexpected e
Why I don't get this string: start sendEmailOrders in the log?
I have controler:
#GetMapping("/old")
public Product getOld() {
Product omeOld = productService.getOneOld();
log.info(String.valueOf(omeOld.getId()));
return omeOld;
}
Service:
#Override
#Transactional
public Product getOneOld() {
Product aNew = productsRepository.findTop1ByStatusOrderByCountAsc("NEW");
try {
Thread.sleep(5000L);
} catch (InterruptedException e) {
e.printStackTrace();
}
return aNew;
}
And repository:
#Repository
public interface ProductsRepository extends JpaRepository<Product, Long> {
Product findTop1ByStatusOrderByCountAsc(String status);
}
I start JMeter and send 5 request in 5 threads. In result I get 5 response after 5 seconds. Each request was processed by seconds. But in log I see next:
2018-09-14 14:04:35.524 INFO 9048 --- [nio-8080-exec-1] c.e.l.demo.controller.ProductController : 1
2018-09-14 14:04:35.525 INFO 9048 --- [nio-8080-exec-2] c.e.l.demo.controller.ProductController : 1
2018-09-14 14:04:35.532 INFO 9048 --- [nio-8080-exec-3] c.e.l.demo.controller.ProductController : 1
2018-09-14 14:04:35.534 INFO 9048 --- [nio-8080-exec-4] c.e.l.demo.controller.ProductController : 1
2018-09-14 14:04:35.534 INFO 9048 --- [nio-8080-exec-6] c.e.l.demo.controller.ProductController : 1
Each thread select the same row and process it. I need that first thread select first row, second thread select second row and etc. I try use #Lock(LockModeType.PESSIMISTIC_WRITE) :
#Lock(LockModeType.PESSIMISTIC_WRITE)
Product findTop1ByStatusOrderByCountAsc(String status);
Now when I start JMeter I have next behavior:
first thread worck 5 sec, after that second thread work 5 sec and etc. 25 secons all 5 threads. And in log:
2018-09-14 14:11:40.564 INFO 13724 --- [nio-8080-exec-5] c.e.l.demo.controller.ProductController : 1
2018-09-14 14:11:45.566 INFO 13724 --- [nio-8080-exec-4] c.e.l.demo.controller.ProductController : 1
2018-09-14 14:11:50.567 INFO 13724 --- [nio-8080-exec-2] c.e.l.demo.controller.ProductController : 1
2018-09-14 14:11:55.568 INFO 13724 --- [nio-8080-exec-1] c.e.l.demo.controller.ProductController : 1
2018-09-14 14:12:00.570 INFO 13724 --- [nio-8080-exec-3] c.e.l.demo.controller.ProductController : 1
All threads select the same row(if I change this roe in first thread - it will not select in second thread if the conditions do not match).
I try this:
#Query(value = "Select * from products where status = ?1 order by count asc LIMIT 1 for update", nativeQuery = true)
Product findTop1ByStatusOrderByCountAsc(String status);
the result is the same.
But I need - first thread select first row and block it/ Second thread select next not blocked row and process. I try next:
#Query(value = "Select * from products where status = ?1 order by count asc LIMIT 1 for update of products skip locked", nativeQuery = true)
Product findTop1ByStatusOrderByCountAsc(String status);
And it work fine! :
2018-09-14 14:25:00.355 INFO 7904 --- [io-8080-exec-10] c.e.l.demo.controller.ProductController : 4
2018-09-14 14:25:00.355 INFO 7904 --- [nio-8080-exec-4] c.e.l.demo.controller.ProductController : 3
2018-09-14 14:25:00.355 INFO 7904 --- [nio-8080-exec-9] c.e.l.demo.controller.ProductController : 1
2018-09-14 14:25:00.358 INFO 7904 --- [nio-8080-exec-5] c.e.l.demo.controller.ProductController : 5
2018-09-14 14:25:00.359 INFO 7904 --- [nio-8080-exec-2] c.e.l.demo.controller.ProductController : 6
Each select in each thread select one row from non blocked rows!
But how can I repeat this with Oracle? In oracle I can not write LIMIT 1 and if I use ROWNUM = 1 each thread select same row always.
I have a simple class:
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class DummyService {
private final Logger logger = LoggerFactory.getLogger(getClass());
private boolean dataIndexing = false;
public boolean isDataIndexing() {
logger.info("isDataIndexing: {}", dataIndexing);
return dataIndexing;
}
public void cancelIndexing() {
logger.info("cancelIndexing: {}", dataIndexing);
dataIndexing = false;
}
public void createIndexCorp() {
logger.info("createIndexCorp: {}", dataIndexing);
createIndex();
}
public void createIndexEntr() {
logger.info("createIndexEntr: {}", dataIndexing);
createIndex();
}
private void createIndex() {
logger.info("createIndex: {}", dataIndexing);
if(dataIndexing)
throw new IllegalStateException("Service is busy!");
dataIndexing = true;
try {
while(dataIndexing) {
Thread.sleep(100);
logger.debug("I am busy...");
}
logger.info("Indexing canceled");
} catch (InterruptedException e) {
logger.error("Error during sleeping", e);
} finally {
dataIndexing = false;
}
}
}
and a unit test, with which i want to test object behavior:
public class CommonUnitTest
{
#Test
public void testCreateIndexWithoutAsync() throws InterruptedException {
final long sleepMillis = 500;
final DummyService indexService = new DummyService();
assertFalse(indexService.isDataIndexing());
new Thread(() -> {
indexService.createIndexCorp();
}
).start();
Thread.sleep(sleepMillis);
assertTrue(indexService.isDataIndexing());
// TaskExecutor should fails here
new Thread(() -> {
indexService.createIndexEntr();
logger.error("Exception expected but not occurred");
}
).start();
assertTrue(indexService.isDataIndexing());
indexService.cancelIndexing();
Thread.sleep(sleepMillis);
assertFalse(indexService.isDataIndexing());
}
}
The behaviour of object must be: If the method createIndexCorp or createIndexEntr is called by one thread, then another thread must get exception by trying to call one of this methods. But this does not happens! Here is the log:
2015-10-15 17:15:06.277 INFO --- [ main] c.c.o.test.DummyService : isDataIndexing: false
2015-10-15 17:15:06.318 INFO --- [ Thread-0] c.c.o.test.DummyService : createIndexCorp: false
2015-10-15 17:15:06.319 INFO --- [ Thread-0] c.c.o.test.DummyService : createIndex: false
2015-10-15 17:15:06.419 DEBUG --- [ Thread-0] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:06.524 DEBUG --- [ Thread-0] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:06.624 DEBUG --- [ Thread-0] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:06.724 DEBUG --- [ Thread-0] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:06.818 INFO --- [ main] c.c.o.test.DummyService : isDataIndexing: true
2015-10-15 17:15:06.820 INFO --- [ main] c.c.o.test.DummyService : isDataIndexing: true
2015-10-15 17:15:06.820 INFO --- [ Thread-1] c.c.o.test.DummyService : createIndexEntr: true
2015-10-15 17:15:06.820 INFO --- [ main] c.c.o.test.DummyService : cancelIndexing: true
2015-10-15 17:15:06.820 INFO --- [ Thread-1] c.c.o.test.DummyService : createIndex: true
2015-10-15 17:15:06.824 DEBUG --- [ Thread-0] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:06.921 DEBUG --- [ Thread-1] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:06.924 DEBUG --- [ Thread-0] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:07.021 DEBUG --- [ Thread-1] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:07.024 DEBUG --- [ Thread-0] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:07.121 DEBUG --- [ Thread-1] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:07.124 DEBUG --- [ Thread-0] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:07.221 DEBUG --- [ Thread-1] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:07.224 DEBUG --- [ Thread-0] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:07.321 DEBUG --- [ Thread-1] c.c.o.test.DummyService : I am busy...
2015-10-15 17:15:07.321 INFO --- [ main] c.c.o.test.DummyService : isDataIndexing: true
You can see that second thread can start process, but it should get the exception. Also the last assertion in the test code fails. How can that happen ? I dont understand this behavior. I tried to use volatile and synchronized keyword, but nothing helps. What is wrong with DummyService ?
You have 3 threads, t0, t1 and tm (main).
The order of operations is like this:
tm starts t0
t0 checks dataIndexing flag - false, goes into the loop, sets flag to true
tm sleeps
tm starts t1
tm sets indexing flag to false
t1 checks dataIndexing flag - false, goes into the loop, sets flag to true
t0 continues the loop because it missed that brief period when indexing was cancelled
If you sleep in the main tm before setting indexing flag to false, then t1 will get the exception.
You need to synchronize access to variables shared between multiple threads. I.e. checking the state of the flag and changing it needs to be done while holding a mutex.
It seems you're hitting the difference between logging and the actual execution. The threads can conceivably run cancel and create index in the space between logging and and the exception, thus the second thread slipping by and preventing cancelling of the first and the second.
It is not advisable to allow simultaneous changes to shared resource, namely private boolean dataIndexing. There are two solutions (at least):
1.A syncronized method to allow for change of the shared resource (thus limiting access to only one thread at a time)
private synchronized void setDataIndexing(boolean value) {
dataIndexing = value;
}
2.Guarding each change of this value in a syncronized section (in both the = true and the = false places):
syncronized (this) {
dataIndexing = /* the relevant value */;
}
I would advise a separate method, but good to know the alternatives.
Not an answer to your question, but this is completely unsynchronized:
if (dataIndexing)
throw new IllegalStateException("Service is busy!");
dataIndexing = true;
Is the service busy if your execution reaches the throw statement? Not necessarily! Another thread could have changed the value of dataIndexing from true to false in between the test and the throw.
What's worse, maybe much worse, is that two threads might both reach the statement after the throw at the same time:
Thread A Thread B
tests dataIndexing, finds it to
be false.
Tests dataIndexing, finds it to be false.
sets dataIndexing = true; sets dataIndexing = true;
... ...
Also, this is unreliable, and it takes time.
Thread.sleep(sleepMillis);
assertTrue(indexService.isDataIndexing());
Better to design your classes for testability. If your test needs to wait until isDataIndexing(), then your class should provide a means for the test to wait()...
Also, don't underestimate the importance of making tests that complete in the least amount of time possible. When you have a system that has thousands or tens of thousands of test cases, the seconds really start to add up.