I'm trying to use Citrus to build a configurable mockup that I'd like to run from the command line passing different parameters to the test each time I run it.
I tried to use this as a reference to get started https://github.com/citrusframework/citrus/issues/325 but could not get it to work in my case.
I've got a running test running with one test case inside and I can run it like that
mvn clean verify -Dit.test=myTest#myTestCase
But when trying to run the following app:
package com.grge.citrus;
import com.consol.citrus.dsl.design.DefaultTestDesigner;
import com.consol.citrus.Citrus;
import com.consol.citrus.context.TestContext;
import org.springframework.context.ConfigurableApplicationContext;
import com.grge.citrus.*;
import com.consol.citrus.dsl.testng.TestNGCitrusTestDesigner;
public class App {
public static void main(String[] args) {
String suiteName = "mySuite";
Citrus citrus = Citrus.newInstance();
// my test with one testcase inside
TestNGCitrusTestDesigner myTest = new MyTest();
try {
citrus.beforeSuite(suiteName);
// This works it come from the sample
citrus.run(new SampleJavaDslTest().getTestCase());
// fails null pointer exception
citrus.run(myTest.getTestCase());
citrus.afterSuite(suiteName);
} finally {
((ConfigurableApplicationContext) citrus.getApplicationContext()).close();
}
}
private static class SampleJavaDslTest extends DefaultTestDesigner {
public SampleJavaDslTest() {
super();
echo("Hello from Java DSL!");
}
}
}
It fails with the following error
22:08:21,691 DEBUG citrus.Citrus| Loading Citrus application properties
22:08:21,696 DEBUG citrus.Citrus| Setting application property citrus.spring.java.config=com.grge.citrus.VCenterActorConfigSSL
22:08:22,025 DEBUG BeanDefinitionReader| Loaded 0 bean definitions from location pattern [classpath*:citrus-context.xml]
22:08:23,059 DEBUG server.HttpServer| Starting server: vCenterServer ...
22:08:23,217 DEBUG CachingServletFilter| Initializing filter 'request-caching-filter'
22:08:23,220 DEBUG CachingServletFilter| Filter 'request-caching-filter' configured successfully
22:08:23,220 DEBUG et.GzipServletFilter| Initializing filter 'gzip-filter'
22:08:23,220 DEBUG et.GzipServletFilter| Filter 'gzip-filter' configured successfully
22:08:23,222 DEBUG rusDispatcherServlet| Initializing servlet 'vCenterServer-servlet'
22:08:23,241 INFO rusDispatcherServlet| FrameworkServlet 'vCenterServer-servlet': initialization started
22:08:23,250 DEBUG rusDispatcherServlet| Servlet with name 'vCenterServer-servlet' will try to create custom WebApplicationContext context of class 'org.springframework.web.context.support.XmlWebApplicationContext', using parent context [null]
22:08:23,865 DEBUG rusDispatcherServlet| Unable to locate MultipartResolver with name 'multipartResolver': no multipart request handling provided
22:08:23,869 DEBUG rusDispatcherServlet| Unable to locate LocaleResolver with name 'localeResolver': using default [org.springframework.web.servlet.i18n.AcceptHeaderLocaleResolver#16fb356]
22:08:23,873 DEBUG rusDispatcherServlet| Unable to locate ThemeResolver with name 'themeResolver': using default [org.springframework.web.servlet.theme.FixedThemeResolver#1095f122]
22:08:23,894 DEBUG rusDispatcherServlet| No HandlerExceptionResolvers found in servlet 'vCenterServer-servlet': using default
22:08:23,896 DEBUG rusDispatcherServlet| Unable to locate RequestToViewNameTranslator with name 'viewNameTranslator': using default [org.springframework.web.servlet.view.DefaultRequestToViewNameTranslator#733c423e]
22:08:23,910 DEBUG rusDispatcherServlet| No ViewResolvers found in servlet 'vCenterServer-servlet': using default
22:08:23,915 DEBUG rusDispatcherServlet| Unable to locate FlashMapManager with name 'flashMapManager': using default [org.springframework.web.servlet.support.SessionFlashMapManager#681aad3b]
22:08:23,922 DEBUG rusDispatcherServlet| Published WebApplicationContext of servlet 'vCenterServer-servlet' as ServletContext attribute with name [org.springframework.web.servlet.FrameworkServlet.CONTEXT.vCenterServer-servlet]
22:08:23,922 INFO rusDispatcherServlet| FrameworkServlet 'vCenterServer-servlet': initialization completed in 678 ms
22:08:23,923 DEBUG rusDispatcherServlet| Servlet 'vCenterServer-servlet' configured successfully
22:08:54,266 INFO server.HttpServer| Started server: vCenterServer
22:08:54,362 INFO port.LoggingReporter|
22:08:54,362 INFO port.LoggingReporter| ------------------------------------------------------------------------
22:08:54,362 INFO port.LoggingReporter| .__ __
22:08:54,362 INFO port.LoggingReporter| ____ |__|/ |________ __ __ ______
22:08:54,362 INFO port.LoggingReporter| _/ ___\| \ __\_ __ \ | \/ ___/
22:08:54,362 INFO port.LoggingReporter| \ \___| || | | | \/ | /\___ \
22:08:54,362 INFO port.LoggingReporter| \___ >__||__| |__| |____//____ >
22:08:54,362 INFO port.LoggingReporter| \/ \/
22:08:54,362 INFO port.LoggingReporter|
22:08:54,363 INFO port.LoggingReporter| C I T R U S T E S T S 2.7.8
22:08:54,363 INFO port.LoggingReporter|
22:08:54,363 INFO port.LoggingReporter| ------------------------------------------------------------------------
22:08:54,363 DEBUG port.LoggingReporter| BEFORE TEST SUITE
22:08:54,363 INFO port.LoggingReporter|
22:08:54,363 INFO port.LoggingReporter|
22:08:54,363 INFO port.LoggingReporter| BEFORE TEST SUITE: SUCCESS
22:08:54,363 INFO port.LoggingReporter| ------------------------------------------------------------------------
22:08:54,363 INFO port.LoggingReporter|
22:08:54,370 DEBUG t.TestContextFactory| Created new test context - using global variables: '{}'
22:08:54,370 INFO port.LoggingReporter|
22:08:54,370 INFO port.LoggingReporter| ------------------------------------------------------------------------
22:08:54,370 DEBUG port.LoggingReporter| STARTING TEST SampleJavaDslTest <com.grge.citrus>
22:08:54,370 INFO port.LoggingReporter|
22:08:54,370 DEBUG citrus.TestCase| Initializing test case
22:08:54,372 DEBUG context.TestContext| Setting variable: citrus.test.name with value: 'SampleJavaDslTest'
22:08:54,372 DEBUG context.TestContext| Setting variable: citrus.test.package with value: 'com.grge.citrus'
22:08:54,372 DEBUG citrus.TestCase| Test variables:
22:08:54,372 DEBUG citrus.TestCase| citrus.test.name = SampleJavaDslTest
22:08:54,372 DEBUG citrus.TestCase| citrus.test.package = com.grge.citrus
22:08:54,373 INFO actions.EchoAction| Hello from Java DSL!
22:08:54,493 INFO port.LoggingReporter|
22:08:54,493 INFO port.LoggingReporter| TEST SUCCESS SampleJavaDslTest (com.grge.citrus)
22:08:54,493 INFO port.LoggingReporter| ------------------------------------------------------------------------
22:08:54,493 INFO port.LoggingReporter|
Exception in thread "main" java.lang.NullPointerException
at com.consol.citrus.dsl.testng.TestNGCitrusTestDesigner.getTestCase(TestNGCitrusTestDesigner.java:113)
at com.grge.citrus.App.main(App.java:25)
Effectively using the debugger I can see that when entering the try section, myTest has:
applicationContext
testDesigner
and many other attributes set to null and so the testDesigner.getTestCase() raise an Exception
Thx
Since Citrus 2.7 the framework provides a main CLI class that does exactly what you want:
https://github.com/citrusframework/citrus/blob/10d3954fd6c051d32e9cd974f685322c6454c779/modules/citrus-core/src/main/java/com/consol/citrus/main/CitrusApp.java#L38
Citrus application option usage:
-h or --help = Displays cli option usage
-d or --duration = Maximum time in milliseconds the server should be up and running - server will terminate automatically when time exceeds
-c or --config = Custom Spring configuration class
-s or --skipTests = Skip test execution
-p or --package = Test package to execute
-D or --properties = Default system properties to set
-e or --exit = Force system exit when finished
-t or --test = Test class/method to execute
-j or --jar = External test jar to load tests from
Related
I have a single step springbatch application. The job is as follows:
#Bean
public Job databaseCursorJob(#Qualifier("databaseCursorStep") Step exampleJobStep,
JobBuilderFactory jobBuilderFactory) {
return jobBuilderFactory.get("databaseCursorJob")
.incrementer(new RunIdIncrementer())
.flow(exampleJobStep)
.end()
.build();
}
I start the job from a springboot application. This afternoon, I attempted to add a second step to the job. Essentially as follows:
#Bean
public Job databaseCursorJob(#Qualifier("databaseCursorStep") Step exampleJobStep,
JobBuilderFactory jobBuilderFactory) {
return jobBuilderFactory.get("databaseCursorJob")
.incrementer(new RunIdIncrementer())
.flow(exampleJobStep).next(partitionStep())
.end()
.build();
}
In other words, just adding the "next(partitionStep()). However, ever since I did this, the job finishes without executing any step (see shell output below). In fact, even after removing the second step and going back to the original job, it refuses to execute the step. Before attempting to add the second step, I never once encountered this problem. I have gone so far as restarting my VM and it still skips the step. I am rather dead in the water until I resolved this. Grateful for any insights. thanks.
2020-09-01 14:49:00.260 INFO 6913 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8087 (http) with context path ''
2020-09-01 14:49:00.263 INFO 6913 --- [ main] f.p.r.Application : Started Application in 7.752 seconds (JVM running for 9.092)
2020-09-01 14:49:00.268 INFO 6913 --- [ main] o.s.b.a.b.JobLauncherCommandLineRunner : Running default command line with: []
2020-09-01 14:49:00.579 INFO 6913 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=databaseCursorJob]] launched with the following parameters: [{}]
2020-09-01 14:49:00.698 INFO 6913 --- [ main] o.s.batch.core.job.SimpleStepHandler : Step already complete or not restartable, so no action to execute: StepExecution: id=120, version=4, name=databaseCursorStep, status=COMPLETED, exitStatus=COMPLETED, readCount=1, filterCount=0, writeCount=1 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=2, rollbackCount=0, exitDescription=
2020-09-01 14:49:00.730 INFO 6913 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=databaseCursorJob]] completed with the following parameters: [{}] and the following status: [COMPLETED]
My issue was that my job had no way recover if there was an error or stuck in an unknown state. The step was not "already complete", it never completed. Its status was still "STARTED", and exit code "UNKNOWN" because it never exited. Anyway, my job repository is not in memory, but captured to a local DB, which is why it never resolved itself even after restarting VM (shame on me for not remembering this). So, I was able to fix by wiping out the job instance history, however that was a band-aid. I still have to fix my code to prevent it from happening again.
I also learned I could diagnose by examining the job repository in the database (its all there).
I really resolved this thanks Mr Hassine who responded above several times and pointed me in the right direction. The solution to prevent in the future is indeed addressed in the link he provided in his first response: Spring Batch error (A Job Instance Already Exists) and RunIdIncrementer generates only once
I configured a Spring Boot starter P6Spy decorator as per the instructions on their site:
## p6spy ###
# Register P6LogFactory to log JDBC events
decorator.datasource.p6spy.enable-logging=true
decorator.datasource.datasource-proxy.query.log-level=debug
decorator.datasource.datasource-proxy.slow-query.enable-logging=true
decorator.datasource.datasource-proxy.slow-query.log-level=warn
decorator.datasource.datasource-proxy.slow-query.logger-name=
# Use com.p6spy.engine.spy.appender.MultiLineFormat instead of com.p6spy.engine.spy.appender.SingleLineFormat
decorator.datasource.p6spy.multiline=true
# Use logging for default listeners [slf4j, sysout, file]
decorator.datasource.p6spy.logging=slf4j
# Log file to use (only with logging=file)
decorator.datasource.p6spy.log-file=spy.log
# Custom log format, if specified com.p6spy.engine.spy.appender.CustomLineFormat will be used with this log format
decorator.datasource.p6spy.log-format=
<dependency>
<groupId>com.github.gavlyukovskiy</groupId>
<artifactId>p6spy-spring-boot-starter</artifactId>
<version>1.5.8</version>
</dependency>
but getting only empty output from p6spy:
2019-12-24 16:22:13.103 DEBUG 11672 --- [ntainer#0-0-C-1] o.s.jdbc.core.JdbcTemplate : Executing prepared SQL query
2019-12-24 16:22:13.103 DEBUG 11672 --- [ntainer#0-0-C-1] o.s.jdbc.core.JdbcTemplate : Executing prepared SQL statement [SELECT COMIT_ID FROM DPL_PARTY.USR_DTL WHERE INDVDL_ID = (SELECT INDVDL_ID FROM DPL_PARTY.INDVDL_TLCMMNCTN WHERE EML_VAL = ?)]
2019-12-24 16:22:13.181 INFO 11672 --- [ntainer#0-0-C-1] p6spy :
2019-12-24 16:22:13.182 DEBUG 11672 --- [ntainer#0-0-C-1] c.s.a.repository.UserRepository : Obtained comitId=XBBKRHL for email of niren.sinha#bnymellon.com from the DB.
2019-12-24 16:22:13.286 INFO 11672 --- [ntainer#0-0-C-1] p6spy :
The query themselves execute fine. What am I missing here? Thanks.
Try to leave out the empty configuration for the log format:
# Custom log format, if specified com.p6spy.engine.spy.appender.CustomLineFormat will be used with this log format
# decorator.datasource.p6spy.log-format=
I think if you have the property without a value, it will be passed as an empty string.
spring butch configuration file:
#Configuration
public class TransitionConfiguration {
#Autowired
public JobBuilderFactory jobBuilderFactory;
#Autowired
public StepBuilderFactory stepBuilderFactory;
#Bean
public Step step1() {
return stepBuilderFactory.get("step1")
.tasklet((contribution, chunkContext) -> {
System.out.println(">> This is step 1");
return RepeatStatus.FINISHED;
}).build();
}
#Bean
public Step step2() {
return stepBuilderFactory.get("step2")
.tasklet((contribution, chunkContext) -> {
System.out.println(">> This is step 2");
return RepeatStatus.FINISHED;
}).build();
}
#Bean
public Job jobSimpleNext() {
System.out.println("starting job");
return jobBuilderFactory.get("jobNext")
.start(step1())
.next(step2())
.build();
}
}
in case i've started it:
2019-06-09 23:32:02.052 INFO 19976 --- [ restartedMain] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
2019-06-09 23:32:02.190 INFO 19976 --- [ restartedMain] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
starting job
2019-06-09 23:32:02.495 INFO 19976 --- [ restartedMain] o.s.b.c.r.s.JobRepositoryFactoryBean : No database type set, using meta data indicating: MYSQL
2019-06-09 23:32:02.601 INFO 19976 --- [ restartedMain] o.s.b.c.l.support.SimpleJobLauncher : No TaskExecutor has been set, defaulting to synchronous executor.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils (file:/home/sergii/.gradle/caches/modules-2/files-2.1/org.springframework/spring-core/5.2.0.M2/9a84d456ad8d5151da06fc8a85540da9cc95d734/spring-core-5.2.0.M2.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2019-06-09 23:32:02.705 INFO 19976 --- [ restartedMain] o.s.b.d.a.OptionalLiveReloadServer : LiveReload server is running on port 35729
2019-06-09 23:32:02.738 INFO 19976 --- [ restartedMain] o.v.s.b.l.s.t.TransitionsApplication : Started TransitionsApplication in 2.12 seconds (JVM running for 2.588)
2019-06-09 23:32:02.740 INFO 19976 --- [ restartedMain] o.s.b.a.b.JobLauncherCommandLineRunner : Running default command line with: []
2019-06-09 23:32:02.891 INFO 19976 --- [ restartedMain] o.s.b.c.l.support.SimpleJobLauncher : Job: [SimpleJob: [name=transitionJobNext]] launched with the following parameters: [{}]
2019-06-09 23:32:02.931 INFO 19976 --- [ restartedMain] o.s.batch.core.job.SimpleStepHandler : Step already complete or not restartable, so no action to execute: StepExecution: id=19, version=3, name=step1, status=COMPLETED, exitStatus=COMPLETED, readCount=0, filterCount=0, writeCount=0 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=1, rollbackCount=0, exitDescription=
2019-06-09 23:32:02.945 INFO 19976 --- [ restartedMain] o.s.batch.core.job.SimpleStepHandler : Step already complete or not restartable, so no action to execute: StepExecution: id=20, version=3, name=step2, status=COMPLETED, exitStatus=COMPLETED, readCount=0, filterCount=0, writeCount=0 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=1, rollbackCount=0, exitDescription=
2019-06-09 23:32:02.954 INFO 19976 --- [ restartedMain] o.s.b.c.l.support.SimpleJobLauncher : Job: [SimpleJob: [name=transitionJobNext]] completed with the following parameters: [{}] and the following status: [COMPLETED]
2019-06-09 23:32:02.963 INFO 19976 --- [ Thread-7] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...
2019-06-09 23:32:02.974 INFO 19976 --- [ Thread-7] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.
Checking logs we can detect that everything is completed.
BUT
Tasklets was not executed. I really confused because of this.
Checking DB I've detected that job status is COMPLETED but exit_code is NOOP.
I've detected also, that using .allowStartIfComplete(true) for step triggers tasklet execution. But that like a hack which can be used for the step only single time in the same job scope (it doesn't work for case double execution step in the same job). For example:
#Bean
public Step step2() {
return stepBuilderFactory.get("step2")
.allowStartIfComplete(true)
.tasklet((contribution, chunkContext) -> {
System.out.println(">> This is step 2");
return RepeatStatus.FINISHED;
}).build();
}
In case I start project in terminal - it works correctly.
My env:
OpenJDK Runtime Environment Zulu11.29+3-CA (build 11.0.2+7-LTS)
Gradle 5.4.1
Idea community (just installed latest version)
OS ubuntu 18.04
command to start application from idea is:
/home/sergii/.sdkman/candidates/java/current/bin/java -javaagent:/home/sergii/IDE/idea-IC-191.7479.19/lib/idea_rt.jar=33229:/home/sergii/IDE/idea-IC-191.7479.19/bin -Dfile.encoding=UTF-8 -classpath /home/sergii/Development/projects/my/spring/spring-batch-learning/out/production/classes:/home/sergii/Development/projects/my/spring/spring-batch-learning/out/production/resources:/home/sergii/.gradle/caches/modules-2/files-2.1/org.springframework.boot/spring-boot-starter-actuator/2.2.0.M3/f7d3810d75b6fb01b7aa1d8df9e21b3bea8c2dbc/spring-boot-starter-actuator-2.2.0.M3.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.springframework.boot/spring-boot-devtools/2.2.0.M3/853cb206490a2946646d4bac02446b8dc564be30/spring-boot-devtools-2.2.0.M3.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.springframework.boot/spring-boot-starter-batch/2.2.0.M3/fe0bb4352d0c49d9b088db13c6a89fd9619bf2dc/spring-boot-starter-batch-2.2.0.M3.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.springframework.boot/spring-boot-starter-mail/2.2.0.M3/ffc975519f3c14e7577d0407845240caa3fc10f9/spring-boot-starter-mail-2.2.0.M3.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.springframework.boot/spring-boot-starter-jdbc/2.2.0.M3/fbd92e6461e5e37186c360f0080af27719780811/spring-boot-starter-jdbc-2.2.0.M3.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.springframework.boot/spring-boot-starter/2.2.0.M3/d858f3131933381d6661c0f08b6bd9669f581123/spring-boot-starter-2.2.0.M3.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.springframework.boot/spring-boot-actuator-autoconfigure/2.2.0.M3/3a6536cc550e7e555be62d16720c285b40c3c95a/spring-boot-actuator-autoconfigure-2.2.0.M3.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/mysql/mysql-connector-java/8.0.16/6088b7a25188ab4b3ab865422a8ec77ade29236/mysql-connector-java-8.0.16.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.springframework.batch/spring-batch-core/4.2.0.M2/60b52cb2d85ead44ecf7b0bf47dfeb6e672316b6/spring-batch-core-4.2.0.M2.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/io.micrometer/micrometer-core/1.1.4/96eabfe2343a4a4676d215b2122cbbc4d4b6af9b/micrometer-core-1.1.4.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.springframework/spring-context-support/5.2.0.M2/fa3009cdce6b4155da8833bb003577991cd71908/spring-context-support-5.2.0.M2.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/com.sun.mail/jakarta.mail/1.6.3/787e007e377223bba85a33599d3da416c135f99b/jakarta.mail-1.6.3.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.springframework.boot/spring-boot-autoconfigure/2.2.0.M3/3d990f3a7716875013570a1ddd9c79a5dc556390/spring-boot-autoconfigure-2.2.0.M3.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.springframework.boot/spring-boot-actuator/2.2.0.M3/e08a8914653e13395ddaf23fde41570d919b8109/spring-boot-actuator-2.2.0.M3.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.springframework.boot/spring-boot/2.2.0.M3/fc0b424da418b242c3d953bd6bbf06f03bfb1925/spring-boot-2.2.0.M3.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.springframework.boot/spring-boot-starter-logging/2.2.0.M3/ae4dc76f8f14327ca3a792584f666d484f21b5/spring-boot-starter-logging-2.2.0.M3.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/jakarta.annotation/jakarta.annotation-api/1.3.4/a858ec3f0ebd2b8d855c1ddded2cde9b381b0517/jakarta.annotation-api-1.3.4.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.springframework/spring-context/5.2.0.M2/c85d8095c7765d8d38b9b6a357aa347b617bab79/spring-context-5.2.0.M2.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.springframework/spring-jdbc/5.2.0.M2/ea5a41f2b01a2a5f88426e3a509f53479e2b6c41/spring-jdbc-5.2.0.M2.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.springframework.batch/spring-batch-infrastructure/4.2.0.M2/b522e7c7f1c5ebb575136f149b0408462d391e22/spring-batch-infrastructure-4.2.0.M2.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.springframework/spring-aop/5.2.0.M2/8deb00a5c5e18b6c594877afe4e82b8b8d64ccb/spring-aop-5.2.0.M2.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.springframework/spring-tx/5.2.0.M2/7042f477e471796be677f71ec63575ac7c47a749/spring-tx-5.2.0.M2.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.springframework/spring-beans/5.2.0.M2/c4aa2bb803602ebc26a7ee47628f6af106e1bf55/spring-beans-5.2.0.M2.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.springframework.retry/spring-retry/1.2.4.RELEASE/e5a1e629b2743dc7bbe4a8d07ebe9ff6c3b816ce/spring-retry-1.2.4.RELEASE.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.springframework/spring-expression/5.2.0.M2/cdf6909ed2decf704486ca85395e97177fd7535b/spring-expression-5.2.0.M2.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.springframework/spring-core/5.2.0.M2/9a84d456ad8d5151da06fc8a85540da9cc95d734/spring-core-5.2.0.M2.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-databind/2.9.8/11283f21cc480aa86c4df7a0a3243ec508372ed2/jackson-databind-2.9.8.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.hdrhistogram/HdrHistogram/2.1.9/e4631ce165eb400edecfa32e03d3f1be53dee754/HdrHistogram-2.1.9.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.latencyutils/LatencyUtils/2.0.3/769c0b82cb2421c8256300e907298a9410a2a3d3/LatencyUtils-2.0.3.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/com.zaxxer/HikariCP/3.3.1/bb447db60818ecfdbb1b99e7bd096ba7a252d91a/HikariCP-3.3.1.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.yaml/snakeyaml/1.24/13a9c0d6776483c3876e3ff9384f9bb55b17001b/snakeyaml-1.24.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.datatype/jackson-datatype-jsr310/2.9.8/28ad1bced632ba338e51c825a652f6e11a8e6eac/jackson-datatype-jsr310-2.9.8.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/javax.batch/javax.batch-api/1.0/65392d027a6eb369fd9fcd1b75cae150e25ac03c/javax.batch-api-1.0.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.codehaus.jettison/jettison/1.2/765a6181653f4b05c18c7a9e8f5c1f8269bf9b2/jettison-1.2.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/com.sun.activation/jakarta.activation/1.2.1/8013606426a73d8ba6b568370877251e91a38b89/jakarta.activation-1.2.1.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/ch.qos.logback/logback-classic/1.2.3/7c4f3c474fb2c041d8028740440937705ebb473a/logback-classic-1.2.3.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-to-slf4j/2.11.2/6d37bf7b046c0ce2669f26b99365a2cfa45c4c18/log4j-to-slf4j-2.11.2.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.slf4j/jul-to-slf4j/1.7.26/8031352b2bb0a49e67818bf04c027aa92e645d5c/jul-to-slf4j-1.7.26.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-annotations/2.9.0/7c10d545325e3a6e72e06381afe469fd40eb701/jackson-annotations-2.9.0.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.springframework/spring-jcl/5.2.0.M2/988d5bac4a51ed2675626378ee79f8447eda2002/spring-jcl-5.2.0.M2.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/com.fasterxml.jackson.core/jackson-core/2.9.8/f5a654e4675769c716e5b387830d19b501ca191/jackson-core-2.9.8.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.26/77100a62c2e6f04b53977b9f541044d7d722693d/slf4j-api-1.7.26.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/ch.qos.logback/logback-core/1.2.3/864344400c3d4d92dfeb0a305dc87d953677c03c/logback-core-1.2.3.jar:/home/sergii/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.2/f5e9a2ffca496057d6891a3de65128efc636e26e/log4j-api-2.11.2.jar org.vl.spring.batch.learning.springbatchlearning.transition.TransitionsApplication
Question
I guess the issues with idea, because of it works in terminal. Is any ways to analyze and detect a reason for my case and fix it, how?
P.S.
I've done next unsuccessful tries to fix it:
reinstal idea;
recreate project from prepared sources and from spring start;
restart idea invalidating caches
-
It seem was no intellij idea issues...
I've tried to remove my configured datasource trying experiment with default one using embedded db. It works correctly. Next thing that I've done, I used my first datasource configuration but with different step names. That works. Checking DB I've detected that every executed step has COMPLETED status.
Continue investigation...
Every my step has COMPLETED status in the DB. By default spring batch doesn't allow execute completed steps.
To make COMPLETED steps handled one more time we can:
update COMPLETED status in DB with STOPPED or FAILED - that is really hack
use in step builder function .allowStartIfComplete(true) - the feature of spring batch.
I'm trying to send and receive a TCP message via Citrus-Framework, where I used this thread as a reference:
Receiving messages from tcp server with Citrus framework and Spring Integration times out
I'm using a python message repeater, which returns the received message. I receive a payload with python and returns it but citrus times out. I tried all serializers (except SingleTerminatior, causes a context error).
I tried different serializers but none seem to solve my problem, Citrus always times out.
15:03:58,013 DEBUG t.TestContextFactory| Created new test context - using global variables: '{}'
15:03:58,024 INFO port.LoggingReporter|
15:03:58,024 INFO port.LoggingReporter|------------------------------------------------------------------------
15:03:58,024 DEBUG port.LoggingReporter| STARTING TEST sendSpringIntegrationMessageTest <com.consol.citrus.samples>
15:03:58,025 INFO port.LoggingReporter|
15:03:58,025 DEBUG citrus.TestCase| Initializing test case
15:03:58,026 DEBUG context.TestContext| Setting variable: citrus.test.name with value: 'sendSpringIntegrationMessageTest'
15:03:58,027 DEBUG context.TestContext| Setting variable: citrus.test.package with value: 'com.consol.citrus.samples'
15:03:58,028 DEBUG citrus.TestCase| Test variables:
15:03:58,028 DEBUG citrus.TestCase| citrus.test.name = sendSpringIntegrationMessageTest
15:03:58,028 DEBUG citrus.TestCase| citrus.test.package = com.consol.citrus.samples
15:03:58,029 INFO port.LoggingReporter|
15:03:58,030 DEBUG port.LoggingReporter| TEST STEP 1/2: send
15:03:58,049 DEBUG nnel.ChannelProducer| Sending message to channel: 'input'
15:03:58,055 DEBUG nnel.ChannelProducer| Message to send is:
DEFAULTMESSAGE [id: c5c61991-f567-42bd-9302-1f8e1fa16225, payload: Req][headers: {citrus_message_type=XML, citrus_message_id=c5c61991-f567-42bd-9302-1f8e1fa16225, citrus_message_timestamp=1539090238031}]
15:03:58,164 INFO nnel.ChannelProducer| Message was sent to channel: 'input'
15:03:58,165 INFO port.LoggingReporter|
15:03:58,166 DEBUG port.LoggingReporter| TEST STEP 1/2 SUCCESS
15:03:58,166 INFO port.LoggingReporter|
15:03:58,166 DEBUG port.LoggingReporter| TEST STEP 2/2: receive
15:03:58,168 DEBUG nnel.ChannelConsumer| Receiving message from: replies
15:04:03,171 INFO port.LoggingReporter|
15:04:03,172 ERROR port.LoggingReporter| TEST FAILED sendSpringIntegrationMessageTest <com.consol.citrus.samples> Nested exception is:
at com.consol.citrus.exceptions.ActionTimeoutException: Action timeout while receiving message from channel 'replies'
at com.consol.citrus.channel.ChannelConsumer.receive(ChannelConsumer.java:97)
at com.consol.citrus.messaging.AbstractSelectiveMessageConsumer.receive(AbstractSelectiveMessageConsumer.java:50)
at com.consol.citrus.actions.ReceiveMessageAction.receive(ReceiveMessageAction.java:141)
at com.consol.citrus.actions.ReceiveMessageAction.doExecute(ReceiveMessageAction.java:120)
at com.consol.citrus.actions.AbstractTestAction.execute(AbstractTestAction.java:46)
at com.consol.citrus.dsl.actions.DelegatingTestAction.doExecute(DelegatingTestAction.java:54)
at com.consol.citrus.actions.AbstractTestAction.execute(AbstractTestAction.java:46)
at com.consol.citrus.TestCase.executeAction(TestCase.java:234)
at com.consol.citrus.TestCase.doExecute(TestCase.java:153)
at com.consol.citrus.actions.AbstractTestAction.execute(AbstractTestAction.java:46)
at com.consol.citrus.Citrus.run(Citrus.java:403)
at com.consol.citrus.dsl.testng.TestNGCitrusTest.invokeTestMethod(TestNGCitrusTest.java:125)
at com.consol.citrus.dsl.testng.TestNGCitrusTestDesigner.invokeTestMethod(TestNGCitrusTestDesigner.java:73)
at com.consol.citrus.dsl.testng.TestNGCitrusTest.run(TestNGCitrusTest.java:110)
...
My context seems to be right (I'm using spring-integration-ip 5.0.8-RELEASE), there is no exception when executing the test (except using SingleTerminatior):
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
... >
<citrus:channel-endpoint id="citrusServiceInputEndpoint"
channel-name="input" />
<citrus:channel-endpoint id="citrusServiceRepliesEndpoint"
channel-name="replies" />
<int-ip:tcp-connection-factory id="client"
type="client" host="127.0.0.1"
port="33500" single-use="false"
so-timeout="10000" using-nio="true"
deserializer="javaSerializer"
serializer="javaSerializer" />
<bean id="javaSerializer"
class="org.springframework.integration.ip.tcp.serializer.ByteArrayLfSerializer" />
<int:channel id="input" />
<int:channel id="replies">
<int:queue />
</int:channel>
<int-ip:tcp-outbound-channel-adapter
id="outboundClient" channel="input" connection-factory="client" />
<int-ip:tcp-inbound-channel-adapter
id="inboundClient" channel="replies" connection-factory="client" />
</beans>
I appreciate any kind of help
Thanks
because I'm new to spring here is the dependency I added from my part:
<dependency>
<groupId>org.springframework.integration</groupId>
<artifactId>spring-integration-ip</artifactId>
<version>5.0.8.RELEASE</version>
</dependency>
This is the debug output of citrus:
22:47:41,071 DEBUG port.LoggingReporter| TEST STEP 1/2: send
22:47:41,085 DEBUG tListableBeanFactory| Returning cached instance of singleton bean 'citrusServiceInputEndpoint'
22:47:41,086 DEBUG nnel.ChannelProducer| Sending message to channel: 'input'
22:47:41,086 DEBUG nnel.ChannelProducer| Message to send is:
DEFAULTMESSAGE [id: 7d4f4c7a-92d0-462c-b695-c32fc7e697ae, payload: Req][headers: {citrus_message_type=XML, citrus_message_id=7d4f4c7a-92d0-462c-b695-c32fc7e697ae, citrus_message_timestamp=1539118061073}]
22:47:41,087 DEBUG tListableBeanFactory| Returning cached instance of singleton bean 'input'
22:47:41,091 DEBUG hannel.DirectChannel| preSend on channel 'input', message:
GenericMessage [payload=Req, headers={citrus_message_timestamp=1539118061073, citrus_message_type=XML, id=33295f63-948f-6bcf-289f-d5e1df8dc98b, citrus_message_id=7d4f4c7a-92d0-462c-b695-c32fc7e697ae, timestamp=1539118061091}]
22:47:41,092 DEBUG endingMessageHandler| org.springframework.integration.ip.tcp.TcpSendingMessageHandler#0 received message:
GenericMessage [payload=Req, headers={citrus_message_timestamp=1539118061073, citrus_message_type=XML, id=33295f63-948f-6bcf-289f-d5e1df8dc98b, citrus_message_id=7d4f4c7a-92d0-462c-b695-c32fc7e697ae, timestamp=1539118061091}]
22:47:41,092 DEBUG entConnectionFactory| Opening new socket connection to 127.0.0.1:33500
22:47:41,106 DEBUG ion.TcpNioConnection| New connection localhost:33500:55108:33be3b24-f5ae-4594-83e9-c7eb0f104b1f
22:47:41,110 DEBUG entConnectionFactory| client: Added new connection: localhost:33500:55108:33be3b24-f5ae-4594-83e9-c7eb0f104b1f
22:47:41,113 DEBUG endingMessageHandler| Got Connection localhost:33500:55108:33be3b24-f5ae-4594-83e9-c7eb0f104b1f
22:47:41,114 DEBUG ion.TcpNioConnection| localhost:33500:55108:33be3b24-f5ae-4594-83e9-c7eb0f104b1f writing 4
22:47:41,116 DEBUG ion.TcpNioConnection| localhost:33500:55108:33be3b24-f5ae-4594-83e9-c7eb0f104b1f Message sent GenericMessage [payload=Req, headers={citrus_message_timestamp=1539118061073, citrus_message_type=XML, id=33295f63-948f-6bcf-289f-d5e1df8dc98b, citrus_message_id=7d4f4c7a-92d0-462c-b695-c32fc7e697ae, timestamp=1539118061091}]
22:47:41,117 DEBUG channel.DirectChannel| postSend (sent=true) on channel 'input', message: GenericMessage [payload=Req, headers={citrus_message_timestamp=1539118061073, citrus_message_type=XML, id=33295f63-948f-6bcf-289f-d5e1df8dc98b, citrus_message_id=7d4f4c7a-92d0-462c-b695-c32fc7e697ae, timestamp=1539118061091}]
22:47:41,118 INFO nnel.ChannelProducer| Message was sent to channel: 'input'
22:47:41,118 INFO port.LoggingReporter|
22:47:41,119 DEBUG port.LoggingReporter| TEST STEP 1/2 SUCCESS
22:47:41,119 DEBUG tListableBeanFactory| Returning cached instance of singleton bean 'citrusServiceRepliesEndpoint'
22:47:41,120 INFO port.LoggingReporter|
22:47:41,121 DEBUG port.LoggingReporter| TEST STEP 2/2: receive
22:47:41,122 DEBUG tListableBeanFactory| Returning cached instance of singleton bean 'citrusServiceRepliesEndpoint'
22:47:41,124 DEBUG tListableBeanFactory| Returning cached instance of singleton bean 'response'
22:47:41,125 DEBUG nnel.ChannelConsumer| Receiving message from: response
22:47:46,130 INFO port.LoggingReporter|
22:47:46,131 ERROR port.LoggingReporter| TEST FAILED
sendSpringIntegrationMessageTest <com.consol.citrus.samples> Nested exception is:
com.consol.citrus.exceptions.ActionTimeoutException: Action timeout while receiving message from channel 'response'
...
I checked Wireshark and the payload ("Req") inclusive the linefeed was returned, which can be seen in the screenshot. Maybe I should mention I'm running this in an Ubuntu VM.
Wireshark screenshot
#CitrusTest(name = "sendSpringIntegrationMessageTest")
public void sendSpringIntegrationMessageTest() throws Exception {
send("citrusServiceInputEndpoint").payload("Req");
receive("citrusServiceRepliesEndpoint").payload("Req");
}
EDIT
ok there seem to be an issue in my python server. I changed the code a bit to see the received bytes on the console (before I was just counting the bytes) and suddenly the output of citrus changed.
11:40:54,924 DEBUG port.LoggingReporter| TEST STEP 2/2: receive
11:40:54,925 DEBUG tListableBeanFactory| Returning cached instance of singleton bean 'citrusServiceRepliesEndpoint'
11:40:54,925 DEBUG tListableBeanFactory| Returning cached instance of singleton bean 'response'
11:40:54,925 DEBUG nnel.ChannelConsumer| Receiving message from: response
11:40:54,928 DEBUG ion.TcpNioConnection| localhost:33500:55890:7537f5ed-f447-4b76-ba66-2ec59c7619a9 Reading...
11:40:54,929 DEBUG ion.TcpNioConnection| localhost:33500:55890:7537f5ed-f447-4b76-ba66-2ec59c7619a9 Running an assembler
11:40:54,929 DEBUG ion.TcpNioConnection| Read 4 into raw buffer
Exception in thread "pool-1-thread-3" java.lang.AbstractMethodError: org.springframework.integration.ip.tcp.connection.TcpMessageMapper.toMessage(Ljava/lang/Object;)Lorg/springframework/messaging/Message;
at org.springframework.integration.ip.tcp.connection.TcpNioConnection.convert(TcpNioConnection.java:358)
at org.springframework.integration.ip.tcp.connection.TcpNioConnection.run(TcpNioConnection.java:235)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
11:40:54,937 DEBUG ion.TcpNioConnection| localhost:33500:55890:7537f5ed-f447-4b76-ba66-2ec59c7619a9 Reading...
11:40:54,938 DEBUG ion.TcpNioConnection| Read 0 into raw buffer
11:40:59,930 INFO port.LoggingReporter|
11:40:59,931 ERROR port.LoggingReporter| TEST FAILED sendSpringIntegrationMessageTest <com.consol.citrus.samples> Nested exception is:
com.consol.citrus.exceptions.ActionTimeoutException: Action timeout while receiving message from channel 'response'
EDIT2
Because of the following post:
How to use Spring Integration 5 with Spring Boot 1.5.x
I switched from spring-integration-ip 5.0.8 to 4.3.9 and there the output changed one more time. Now my problem seems to move away from TCP issues to actual spring know-how.
15:04:12,318 DEBUG port.LoggingReporter| TEST STEP 2/2: receive
15:04:12,319 DEBUG tListableBeanFactory| Returning cached instance of singleton bean 'citrusServiceRepliesEndpoint'
15:04:12,319 DEBUG tListableBeanFactory| Returning cached instance of singleton bean 'replies'
15:04:12,319 DEBUG nnel.ChannelConsumer| Receiving message from: replies
15:04:12,327 DEBUG ion.TcpNioConnection| localhost:33500:56004:1174f4c6-608c-44c6-aeec-33da4074e195 Reading...
15:04:12,328 DEBUG ion.TcpNioConnection| localhost:33500:56004:1174f4c6-608c-44c6-aeec-33da4074e195 Running an assembler
15:04:12,329 DEBUG ion.TcpNioConnection| Read 4 into raw buffer
15:04:12,329 DEBUG ion.TcpNioConnection| localhost:33500:56004:1174f4c6-608c-44c6-aeec-33da4074e195 Reading...
15:04:12,330 DEBUG ion.TcpNioConnection| Read 0 into raw buffer
15:04:12,332 DEBUG yteArrayLfSerializer| Available to read:4
15:04:12,333 DEBUG tListableBeanFactory| Returning cached instance of singleton bean 'messageBuilderFactory'
15:04:12,336 DEBUG channel.QueueChannel| preSend on channel 'replies', message: GenericMessage [payload=byte[3], headers={ip_tcp_remotePort=33500, ip_connectionId=localhost:33500:56004:1174f4c6-608c-44c6-aeec-33da4074e195, ip_localInetAddress=0.0.0.0/0.0.0.0, ip_address=127.0.0.1, id=9bd90a54-cc2b-d3d0-ca63-6355f53dee7c, ip_hostname=localhost, timestamp=1539176652336}]
15:04:12,339 DEBUG channel.QueueChannel| postReceive on channel 'replies', message: GenericMessage [payload=byte[3], headers={ip_tcp_remotePort=33500, ip_connectionId=localhost:33500:56004:1174f4c6-608c-44c6-aeec-33da4074e195, ip_localInetAddress=0.0.0.0/0.0.0.0, ip_address=127.0.0.1, id=9bd90a54-cc2b-d3d0-ca63-6355f53dee7c, ip_hostname=localhost, timestamp=1539176652336}]
15:04:12,340 DEBUG nnel.ChannelConsumer| Received message from: replies
15:04:12,340 DEBUG tListableBeanFactory| Returning cached instance of singleton bean 'citrusServiceRepliesEndpoint'
15:04:12,354 DEBUG channel.QueueChannel| postSend (sent=true) on channel 'replies', message: GenericMessage [payload=byte[3], headers={ip_tcp_remotePort=33500, ip_connectionId=localhost:33500:56004:1174f4c6-608c-44c6-aeec-33da4074e195, ip_localInetAddress=0.0.0.0/0.0.0.0, ip_address=127.0.0.1, id=9bd90a54-cc2b-d3d0-ca63-6355f53dee7c, ip_hostname=localhost, timestamp=1539176652336}]
15:04:12,379 INFO port.LoggingReporter|
15:04:12,381 ERROR port.LoggingReporter| TEST FAILED sendSpringIntegrationMessageTest <com.consol.citrus.samples> Nested exception is:
com.consol.citrus.exceptions.CitrusRuntimeException: Could not find proper message validator for message type 'XML', please define a capable message validator for this message type
EDIT3
Seems like after adjusting my environment a lot for test, I was lastly missing this line in my context file:
<int:object-to-string-transformer id="transformer" input-channel="replies" output-channel="response" />
Thanks for nudging me into the right direction.
I am struggling this for days now.
I am using SI adaptor for kafka under Spring-boot container.
I have configured zookeeper and kafka on my machine.
I also created console producer and consumer tested it and everything works fine(I manage to produce vis the console messages and have the console consumer to consume them).
I tried now to produce messages via Spring integration kafka outbound adapter but the console consumer wont consume the message
SI/Spring xd xml:
<int:publish-subscribe-channel id="inputToKafka"/>
<int-kafka:outbound-channel-adapter id="kafkaOutboundChannelAdapter"
kafka-producer-context-ref="kafkaProducerContext"
auto-startup="true"
order="1"
channel="inputToKafka">
</int-kafka:outbound-channel-adapter>
<int-kafka:producer-context id="kafkaProducerContext">
<int-kafka:producer-configurations>
<int-kafka:producer-configuration broker-list="localhost:9092"
async="true"
topic="zerg.hydra"
compression-codec="default"/>
</int-kafka:producer-configurations>
</int-kafka:producer-context>
<task:executor id="taskExecutor" pool-size="5" keep-alive="120" queue-capacity="500"/>
</beans>
Java:
#Named
public class KafkaProducer {
#Autowired
#Qualifier("inputToKafka")
MessageChannel inputToKafka;
public void sendMessageToKafka(String message)
{
inputToKafka.send(
MessageBuilder.withPayload(message)
.setHeader("messageKey", "3")
.setHeader("topic", "zerg.hydra").build());
}
}
this is how I run kafka console consumer:
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic zerg.hydra --from-beginning
logs:
Testing started at 12:49 PM ...
12:49:54 PM: Executing external tasks 'cleanTest test'...
:cleanTest
:compileJava
:processResources UP-TO-DATE
:classes
:compileTestJava
:processTestResources UP-TO-DATE
:testClasses
:test
12:50:07,165 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
12:50:07,166 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
12:50:07,166 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/Users/idan/dev/Projects/CalcMicroService/build/resources/test/logback.xml]
12:50:07,167 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs multiple times on the classpath.
12:50:07,167 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [file:/Users/idan/dev/Projects/CalcMicroService/build/resources/test/logback.xml]
12:50:07,167 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [file:/Users/idan/dev/Projects/CalcMicroService/build/resources/main/logback.xml]
12:50:07,247 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
12:50:07,250 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
12:50:07,259 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [stdout]
12:50:07,281 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
12:50:07,327 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [reactor] to INFO
12:50:07,327 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.projectreactor] to INFO
12:50:07,328 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.springframework] to WARN
12:50:07,328 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.springframework.integration] to DEBUG
12:50:07,328 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
12:50:07,328 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [stdout] to Logger[ROOT]
12:50:07,331 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
12:50:07,332 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator#3de433b4 - Registering current configuration as safe fallback point
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v1.1.9.RELEASE)
12:50:09.530 [Test worker] INFO o.s.i.config.IntegrationRegistrar - No bean named 'integrationHeaderChannelRegistry' has been explicitly defined. Therefore, a default DefaultHeaderChannelRegistry will be created.
12:50:09.544 [Test worker] DEBUG o.s.i.config.IntegrationRegistrar - SpEL function '#xpath' isn't registered: there is no spring-integration-xml.jar on the classpath.
12:50:10.717 [Test worker] INFO o.s.i.c.DefaultConfiguringBeanFactoryPostProcessor - No bean named 'errorChannel' has been explicitly defined. Therefore, a default PublishSubscribeChannel will be created.
12:50:10.719 [Test worker] INFO o.s.i.c.DefaultConfiguringBeanFactoryPostProcessor - No bean named 'taskScheduler' has been explicitly defined. Therefore, a default ThreadPoolTaskScheduler will be created.
12:50:10.973 DEBUG [Test worker][org.jboss.logging] Logging Provider: org.jboss.logging.Log4jLoggerProvider
12:50:10.974 INFO [Test worker][org.hibernate.validator.internal.util.Version] HV000001: Hibernate Validator 5.0.3.Final
12:50:10.991 DEBUG [Test worker][org.hibernate.validator.internal.engine.resolver.DefaultTraversableResolver] Cannot find javax.persistence.Persistence on classpath. Assuming non JPA 2 environment. All properties will per default be traversable.
12:50:11.006 DEBUG [Test worker][org.hibernate.validator.internal.engine.ConfigurationImpl] Setting custom MessageInterpolator of type org.springframework.validation.beanvalidation.LocaleContextMessageInterpolator
12:50:11.011 DEBUG [Test worker][org.hibernate.validator.internal.engine.ConfigurationImpl] Setting custom ConstraintValidatorFactory of type org.springframework.validation.beanvalidation.SpringConstraintValidatorFactory
12:50:11.018 DEBUG [Test worker][org.hibernate.validator.internal.engine.ConfigurationImpl] Setting custom ParameterNameProvider of type com.sun.proxy.$Proxy46
12:50:11.024 DEBUG [Test worker][org.hibernate.validator.internal.xml.ValidationXmlParser] Trying to load META-INF/validation.xml for XML based Validator configuration.
12:50:11.032 DEBUG [Test worker][org.hibernate.validator.internal.xml.ValidationXmlParser] No META-INF/validation.xml found. Using annotation based configuration only.
12:50:12.089 [Test worker] INFO o.a.catalina.core.StandardService - Starting service Tomcat
12:50:12.091 [Test worker] INFO o.a.catalina.core.StandardEngine - Starting Servlet Engine: Apache Tomcat/7.0.56
12:50:14.162 [localhost-startStop-1] INFO o.a.c.c.C.[Tomcat].[localhost].[/] - Initializing Spring embedded WebApplicationContext
12:50:16.567 [Test worker] INFO o.s.i.k.support.ProducerFactoryBean - Using producer properties => {metadata.broker.list=localhost:9092, compression.codec=0, producer.type=async}
12:50:17.036 INFO [Test worker][kafka.utils.VerifiableProperties] Verifying properties
12:50:17.096 INFO [Test worker][kafka.utils.VerifiableProperties] Property compression.codec is overridden to 0
12:50:17.096 INFO [Test worker][kafka.utils.VerifiableProperties] Property metadata.broker.list is overridden to localhost:9092
12:50:17.096 INFO [Test worker][kafka.utils.VerifiableProperties] Property producer.type is overridden to async
12:50:17.591 DEBUG [Test worker][org.hibernate.validator.internal.engine.resolver.DefaultTraversableResolver] Cannot find javax.persistence.Persistence on classpath. Assuming non JPA 2 environment. All properties will per default be traversable.
12:50:17.591 DEBUG [Test worker][org.hibernate.validator.internal.engine.ConfigurationImpl] Setting custom MessageInterpolator of type org.springframework.validation.beanvalidation.LocaleContextMessageInterpolator
12:50:17.591 DEBUG [Test worker][org.hibernate.validator.internal.engine.ConfigurationImpl] Setting custom ConstraintValidatorFactory of type org.springframework.validation.beanvalidation.SpringConstraintValidatorFactory
12:50:17.591 DEBUG [Test worker][org.hibernate.validator.internal.engine.ConfigurationImpl] Setting custom ParameterNameProvider of type com.sun.proxy.$Proxy46
12:50:17.592 DEBUG [Test worker][org.hibernate.validator.internal.xml.ValidationXmlParser] Trying to load META-INF/validation.xml for XML based Validator configuration.
12:50:17.592 DEBUG [Test worker][org.hibernate.validator.internal.xml.ValidationXmlParser] No META-INF/validation.xml found. Using annotation based configuration only.
12:50:18.967 [Test worker] DEBUG o.s.i.c.GlobalChannelInterceptorProcessor - No global channel interceptors.
12:50:18.978 [Test worker] INFO o.s.i.endpoint.EventDrivenConsumer - Adding {mongo:outbound-channel-adapter:mongoAdapter.adapter} as a subscriber to the 'mongoAdapter' channel
12:50:18.979 [Test worker] INFO o.s.i.channel.DirectChannel - Channel 'application:8091.mongoAdapter' has 1 subscriber(s).
12:50:18.979 [Test worker] INFO o.s.i.endpoint.EventDrivenConsumer - started mongoAdapter.adapter
12:50:18.979 [Test worker] INFO o.s.i.endpoint.EventDrivenConsumer - Adding {mongo:outbound-channel-adapter:adapterWithConverter.adapter} as a subscriber to the 'adapterWithConverter' channel
12:50:18.979 [Test worker] INFO o.s.i.channel.DirectChannel - Channel 'application:8091.adapterWithConverter' has 1 subscriber(s).
12:50:18.979 [Test worker] INFO o.s.i.endpoint.EventDrivenConsumer - started adapterWithConverter.adapter
12:50:18.979 [Test worker] INFO o.s.i.endpoint.EventDrivenConsumer - Adding {message-handler:kafkaOutboundChannelAdapter} as a subscriber to the 'inputToKafka' channel
12:50:18.979 [Test worker] INFO o.s.i.c.PublishSubscribeChannel - Channel 'application:8091.inputToKafka' has 1 subscriber(s).
12:50:18.979 [Test worker] INFO o.s.i.endpoint.EventDrivenConsumer - started kafkaOutboundChannelAdapter
12:50:18.979 [Test worker] INFO o.s.i.endpoint.EventDrivenConsumer - Adding {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
12:50:18.979 [Test worker] INFO o.s.i.c.PublishSubscribeChannel - Channel 'application:8091.errorChannel' has 1 subscriber(s).
12:50:18.979 [Test worker] INFO o.s.i.endpoint.EventDrivenConsumer - started _org.springframework.integration.errorLogger
12:50:19.026 [Test worker] INFO o.a.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-8091"]
12:50:19.040 [Test worker] INFO o.a.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8091"]
12:50:19.047 [Test worker] INFO o.a.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read
12:50:19.338 [Test worker] DEBUG o.s.i.c.PublishSubscribeChannel - preSend on channel 'inputToKafka', message: [Payload String content=Hello Kafka From SI][Headers={messageKey=3, topic=zerg.hydra, id=be46fb68-c762-f16e-6ccb-6841ef3fe868, timestamp=1416999019338}]
12:50:19.338 [Test worker] DEBUG o.s.i.k.o.KafkaProducerMessageHandler - org.springframework.integration.kafka.outbound.KafkaProducerMessageHandler#0 received message: [Payload String content=Hello Kafka From SI][Headers={messageKey=3, topic=zerg.hydra, id=be46fb68-c762-f16e-6ccb-6841ef3fe868, timestamp=1416999019338}]
12:50:19.362 [Test worker] DEBUG o.s.i.c.PublishSubscribeChannel - postSend (sent=true) on channel 'inputToKafka', message: [Payload String content=Hello Kafka From SI][Headers={messageKey=3, topic=zerg.hydra, id=be46fb68-c762-f16e-6ccb-6841ef3fe868, timestamp=1416999019338}]
Empty test suite.
12:50:19.402 [Thread-4] INFO o.s.i.endpoint.EventDrivenConsumer - Removing {mongo:outbound-channel-adapter:mongoAdapter.adapter} as a subscriber to the 'mongoAdapter' channel
12:50:19.417 [Thread-4] INFO o.s.i.channel.DirectChannel - Channel 'application:8091.mongoAdapter' has 0 subscriber(s).
12:50:19.419 [Thread-4] INFO o.s.i.endpoint.EventDrivenConsumer - stopped mongoAdapter.adapter
12:50:19.420 [Thread-4] INFO o.s.i.endpoint.EventDrivenConsumer - Removing {mongo:outbound-channel-adapter:adapterWithConverter.adapter} as a subscriber to the 'adapterWithConverter' channel
12:50:19.422 [Thread-4] INFO o.s.i.channel.DirectChannel - Channel 'application:8091.adapterWithConverter' has 0 subscriber(s).
12:50:19.422 [Thread-4] INFO o.s.i.endpoint.EventDrivenConsumer - stopped adapterWithConverter.adapter
12:50:19.423 [Thread-4] INFO o.s.i.endpoint.EventDrivenConsumer - Removing {message-handler:kafkaOutboundChannelAdapter} as a subscriber to the 'inputToKafka' channel
12:50:19.424 [Thread-4] INFO o.s.i.c.PublishSubscribeChannel - Channel 'application:8091.inputToKafka' has 0 subscriber(s).
12:50:19.424 [Thread-4] INFO o.s.i.endpoint.EventDrivenConsumer - stopped kafkaOutboundChannelAdapter
12:50:19.425 [Thread-4] INFO o.s.i.endpoint.EventDrivenConsumer - Removing {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
12:50:19.426 [Thread-4] INFO o.s.i.c.PublishSubscribeChannel - Channel 'application:8091.errorChannel' has 0 subscriber(s).
12:50:19.426 [Thread-4] INFO o.s.i.endpoint.EventDrivenConsumer - stopped _org.springframework.integration.errorLogger
BUILD SUCCESSFUL
Total time: 25.469 secs
12:50:20 PM: External tasks execution finished 'cleanTest test'.
I tried in the same app to use the Offical Kafka client producer and it worked just fine:
#Named
public class KafkaProducerJava {
ProducerConfig config=null;
Producer<String, String> producer;
Properties props=null;
#PostConstruct
public void init()
{
props = new Properties();
props.put("metadata.broker.list", "localhost:9092");
props.put("serializer.class", "kafka.serializer.StringEncoder");
// props.put("partitioner.class", "example.producer.SimplePartitioner");
props.put("request.required.acks", "1");
}
public void sendMsgToKafka(String msg)
{
config= new ProducerConfig(props);
producer=new Producer<String, String>(config);
KeyedMessage<String, String> data = new KeyedMessage<String, String>("test", "", msg);
producer.send(data);
producer.close();
}
Any idea why the message never reached to my consumer via the Spring Integration kafka adaptor??
I just ran it in XD and it worked fine for me...
$ bin/xd-shell
_____ __ _______
/ ___| (-) \ \ / / _ \
\ `--. _ __ _ __ _ _ __ __ _ \ V /| | | |
`--. \ '_ \| '__| | '_ \ / _` | / ^ \| | | |
/\__/ / |_) | | | | | | | (_| | / / \ \ |/ /
\____/| .__/|_| |_|_| |_|\__, | \/ \/___/
| | __/ |
|_| |___/
eXtreme Data
1.1.0.BUILD-SNAPSHOT | Admin Server Target: http://localhost:9393
Welcome to the Spring XD shell. For assistance hit TAB or type "help".
xd:>stream create --name foo --definition "time | kafka --topic=test" --deploy
Created and deployed new stream 'foo'
xd:>stream destroy foo
Destroyed stream 'foo'
xd:>
.
$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
2014-11-26 10:03:09
2014-11-26 10:03:10
2014-11-26 10:03:11
2014-11-26 10:03:12
2014-11-26 10:03:13
I also wrote this test case...
public class OutboundTests {
#Test
public void test() throws Exception {
KafkaProducerContext<String, String> kafkaProducerContext = new KafkaProducerContext<String, String>();
ProducerMetadata<String, String> producerMetadata = new ProducerMetadata<String, String>("test");
producerMetadata.setValueClassType(String.class);
producerMetadata.setKeyClassType(String.class);
Encoder<String> encoder = new StringEncoder<String>();
producerMetadata.setValueEncoder(encoder);
producerMetadata.setKeyEncoder(encoder);
ProducerFactoryBean<String, String> producer = new ProducerFactoryBean<String, String>(producerMetadata, "localhost:9092");
ProducerConfiguration<String, String> config = new ProducerConfiguration<String, String>(producerMetadata, producer.getObject());
kafkaProducerContext.setProducerConfigurations(Collections.singletonMap("test", config));
KafkaProducerMessageHandler<String, String> handler = new KafkaProducerMessageHandler<String, String>(kafkaProducerContext);
handler.handleMessage(MessageBuilder.withPayload("foo")
.setHeader("messagekey", "3")
.setHeader("topic", "test")
.build());
}
}
...to simulate what you are doing and that worked too. Are you seeing any logged exceptions? I see you are not setting the key/value types and encoders.
EDIT:
As discussed in the comments, the issue is that you are using an async producer. In your test example that "works", you are closing the producer, which flushes the queue. By default, the queue won't be flushed for 5 seconds and your test case is not waiting long enough.
I updated my test to include an XML configured version, and reduced the queue.buffering.max.ms to 500ms and added code to the test to wait a couple of seconds before terminating.
See the new commit for details.
I was facing the same issue since morning and finally found the solution. If you prefix the topic with kafka as given below
.setHeader("kafka_topic", "zerg.hydra")
This is a issue with API as internally, while looking up the key from from the message header, kafka_ is getting prefixed.