I am new to Spring Batch and trust me I have read a lot those day about it to try to be familiar with its concepts. I am a bit confused about how JobInstance, RunIdIncrementer, JobParamaters work and I would like to understand some aspects :
When you run a Job and the JobInstance name is already in the BATCH_JOB_INSTANCE table, the Job is not launched. So, what is the best way to generate a new name for my JobInstance?
Is it a good practice to always generate a new name when I want to launch my job?
As job is supposed to be scheduled to run many times. What is the best practice to create a Batch (Job) to be scheduled to run many times without generating a new name?
Does the RunIdIncrementer() is supposed to create an id to generate a new JobName?
Edit : See the code below
#Bean
public Job batchExecution() {
return jobs
.get("BatchJob")
.incrementer(new JobIdIncrementer())
.start(downloadFile())
.next(archiveFile())
.next(readFile())
.build();
}
The JobIdIncrementer :
public class JobIdIncrementer implements JobParametersIncrementer {
private static String RUN_ID_KEY = "run.id";
private String key;
public JobIdIncrementer() {
this.key = RUN_ID_KEY;
}
public void setKey(String key) {
this.key = key;
}
#Override
public JobParameters getNext(JobParameters parameters) {
JobParameters params = parameters == null ? new JobParameters() : parameters;
long id = new Date().getTime();
return (new JobParametersBuilder(params)).addLong(this.key, Long.valueOf(id)).toJobParameters();
}
}
When I start the Batch the fist time, I have this log (it works fine) :
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.0.0.RELEASE)
"2018-08-08 15:36:03 - Starting Application on MA18-012.local with PID 39543
""2018-08-08 15:36:05 - HikariPool-1 - Starting...
""2018-08-08 15:36:05 - HikariPool-1 - Start completed.
""2018-08-08 15:36:06 - HHH000204: Processing PersistenceUnitInfo [
name: default
...]
""2018-08-08 15:36:06 - HHH000412: Hibernate Core {5.2.14.Final}
""2018-08-08 15:36:06 - HHH000206: hibernate.properties not found
""2018-08-08 15:36:06 - HHH80000001: hibernate-spatial integration enabled : true
""2018-08-08 15:36:06 - HCANN000001: Hibernate Commons Annotations {5.0.1.Final}
""2018-08-08 15:36:06 - HHH000400: Using dialect: org.hibernate.spatial.dialect.mysql.MySQL5SpatialDialect
""2018-08-08 15:36:06 - HHH000400: Using dialect: org.hibernate.spatial.dialect.mysql.MySQLSpatialDialect
""2018-08-08 15:36:09 - Started Application in 6.294 seconds (JVM running for 6.812)
""2018-08-08 15:36:09 - Loading the file name
""2018-08-08 15:36:23 - Downloading the file
""2018-08-08 15:36:24 - Archiving the file
""2018-08-08 15:36:24 - Unzipping the file
""2018-08-08 15:36:24 - Removing the file
""2018-08-08 15:36:51 - Reading the file
""2018-08-08 15:36:52 - HHH000397: Using ASTQueryTranslatorFactory
""2018-08-08 15:36:54 - HikariPool-1 - Shutdown initiated...
""2018-08-08 15:36:54 - HikariPool-1 - Shutdown completed.
The second time when start the Batch, I have this one (no error, but it starts and closes immediatly):
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.0.0.RELEASE)
"2018-08-08 15:38:28 - Starting Application on MA18-012.local with PID 39638
""2018-08-08 15:38:28 - No active profile set, falling back to default profiles: default
""2018-08-08 15:38:30 - HikariPool-1 - Starting...
""2018-08-08 15:38:30 - HikariPool-1 - Start completed.
""2018-08-08 15:38:30 - HHH000204: Processing PersistenceUnitInfo [
name: default
...]
""2018-08-08 15:38:30 - HHH000412: Hibernate Core {5.2.14.Final}
""2018-08-08 15:38:30 - HHH000206: hibernate.properties not found
""2018-08-08 15:38:30 - HHH80000001: hibernate-spatial integration enabled : true
""2018-08-08 15:38:30 - HCANN000001: Hibernate Commons Annotations {5.0.1.Final}
""2018-08-08 15:38:31 - HHH000400: Using dialect: org.hibernate.spatial.dialect.mysql.MySQL5SpatialDialect
""2018-08-08 15:38:31 - HHH000400: Using dialect: org.hibernate.spatial.dialect.mysql.MySQLSpatialDialect
""2018-08-08 15:38:33 - Started Application in 6.376 seconds (JVM running for 6.873)
""2018-08-08 15:38:34 - HikariPool-1 - Shutdown initiated...
""2018-08-08 15:38:34 - HikariPool-1 - Shutdown completed.
When you run a Job and the JobInstance name is already in the BATCH_JOB_INSTANCE table, the Job is not launched. So, what is the best way to generate a new name for my JobInstance?
You don't need to change the job name each time. Your job can have multiple job instances, each one is identified by a key which is the hash of the (identifying) job parameters used to run your job.
Is it a good practice to always generate a new name when I want to launch my job?
No, you don't need to generate a new name (it is the same job so the name should not change). What you need to do is to create a new job instance each time by specifying different job parameters.
As job is supposed to be scheduled to run many times. What is the best practice to create a Batch (Job) to be scheduled to run many times without generating a new name?
Depending on the frequency of your job, you can add the date as a job parameter. For example, if your job is scheduled to run daily, the current date is a good parameter. Check this section of the documentation: https://docs.spring.io/spring-batch/4.0.x/reference/html/domain.html#job it provides an example.
Does the RunIdIncrementer() is supposed to create an id to generate a new JobName?
The RunIdIncrementer increments the run.id job parameter so you get a new instance of JobParameters which will result in a new job instance. But the job name will still the same. Here is how it works: https://github.com/spring-projects/spring-batch/blob/master/spring-batch-core/src/main/java/org/springframework/batch/core/launch/support/RunIdIncrementer.java#L44
Related
I can't seem to get GATK to recognise the number of available threads. I am running GATK (4.2.4.1) in a conda environment which is part of a nextflow (v20.10.0) pipeline I'm writing. For whatever reason, I cannot get GATK to see there is more than one thread. I've tried different node types, increasing and decreasing the number of cpus available, providing java arguments such as -XX:ActiveProcessorCount=16, using taskset, but it always just detects 1.
Here is the command from the .command.sh:
gatk HaplotypeCaller \
--tmp-dir tmp/ \
-ERC GVCF \
-R VectorBase-54_AgambiaePEST_Genome.fasta \
-I AE12A_S24_BP.bam \
-O AE12A_S24_BP.vcf
And here is the top of the .command.log file:
12:10:00.695 INFO HaplotypeCaller - ------------------------------------------------------------
12:10:00.695 INFO HaplotypeCaller - The Genome Analysis Toolkit (GATK) v4.2.4.1
12:10:00.695 INFO HaplotypeCaller - For support and documentation go to https://software.broadinstitute.org/gatk/
12:10:00.696 INFO HaplotypeCaller - Executing on Linux v4.18.0-193.6.3.el8_2.x86_64 amd64
12:10:00.696 INFO HaplotypeCaller - Java runtime: OpenJDK 64-Bit Server VM v11.0.13+7-b1751.21
12:10:00.696 INFO HaplotypeCaller - Start Date/Time: 9 February 2022 at 12:10:00 GMT
12:10:00.696 INFO HaplotypeCaller - ------------------------------------------------------------
12:10:00.696 INFO HaplotypeCaller - ------------------------------------------------------------
12:10:00.697 INFO HaplotypeCaller - HTSJDK Version: 2.24.1
12:10:00.697 INFO HaplotypeCaller - Picard Version: 2.25.4
12:10:00.697 INFO HaplotypeCaller - Built for Spark Version: 2.4.5
12:10:00.697 INFO HaplotypeCaller - HTSJDK Defaults.COMPRESSION_LEVEL : 2
12:10:00.697 INFO HaplotypeCaller - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
12:10:00.697 INFO HaplotypeCaller - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
12:10:00.697 INFO HaplotypeCaller - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
12:10:00.697 INFO HaplotypeCaller - Deflater: IntelDeflater
12:10:00.697 INFO HaplotypeCaller - Inflater: IntelInflater
12:10:00.697 INFO HaplotypeCaller - GCS max retries/reopens: 20
12:10:00.698 INFO HaplotypeCaller - Requester pays: disabled
12:10:00.698 INFO HaplotypeCaller - Initializing engine
12:10:01.126 INFO HaplotypeCaller - Done initializing engine
12:10:01.129 INFO HaplotypeCallerEngine - Tool is in reference confidence mode and the annotation, the following changes will be made to any specified annotations: 'StrandBiasBySample' will be enabled. 'ChromosomeCounts', 'FisherStrand', 'StrandOddsRatio' and 'QualByDepth' annotations have been disabled
12:10:01.143 INFO HaplotypeCallerEngine - Standard Emitting and Calling confidence set to 0.0 for reference-model confidence output
12:10:01.143 INFO HaplotypeCallerEngine - All sites annotated with PLs forced to true for reference-model confidence output
12:10:01.162 INFO NativeLibraryLoader - Loading libgkl_utils.so from jar:file:/home/anaconda3/envs/NF_GATK/share/gatk4-4.2.4.1-0/gatk-package-4.2.4.1-local.jar!/com/intel/gkl/native/libgkl_utils.so
12:10:01.169 INFO NativeLibraryLoader - Loading libgkl_pairhmm_omp.so from jar:file:/home/anaconda3/envs/NF_GATK/share/gatk4-4.2.4.1-0/gatk-package-4.2.4.1-local.jar!/com/intel/gkl/native/libgkl_pairhmm_omp.so
12:10:01.209 INFO IntelPairHmm - Flush-to-zero (FTZ) is enabled when running PairHMM
12:10:01.210 INFO IntelPairHmm - Available threads: 1
12:10:01.210 INFO IntelPairHmm - Requested threads: 4
12:10:01.210 WARN IntelPairHmm - Using 1 available threads, but 4 were requested
12:10:01.210 INFO PairHMM - Using the OpenMP multi-threaded AVX-accelerated native PairHMM implementation
12:10:01.271 INFO ProgressMeter - Starting traversal
I found a thread on the broad institute website suggesting it might be the OMP library, but this is seemingly loaded, and I'm using the version they suggested updating to...
Needless to say, this is a little slow. I can always parallelise by using the -L option, but this doesn't solve that every step in the pipeline will be very slow.
Thanks in advance.
In case anyone else has the same problem, it turned out I had to configure the submission as an MPI job.
So on the HPC I use, here is the nextflow process:
process DNA_HCG {
errorStrategy { sleep(Math.pow(2, task.attempt) * 600 as long); return 'retry' }
maxRetries 3
maxForks params.HCG_Forks
tag { SampleID+"-"+chrom }
executor = 'pbspro'
clusterOptions = "-lselect=1:ncpus=${params.HCG_threads}:mem=${params.HCG_memory}gb:mpiprocs=1:ompthreads=${params.HCG_threads} -lwalltime=${params.HCG_walltime}:00:00"
publishDir(
path: "${params.HCDir}",
mode: 'copy',
)
input:
each chrom from chromosomes_ch
set SampleID, path(bam), path(bai) from processed_bams
path ref_genome
path ref_dict
path ref_index
output:
tuple chrom, path("${SampleID}_${chrom}.vcf") into HCG_ch
path("${SampleID}_${chrom}.vcf.idx") into idx_ch
beforeScript 'module load anaconda3/personal; source activate NF_GATK'
script:
"""
mkdir tmp
n_slots=`expr ${params.GVCF_threads} / 2 - 3`
if [ \$n_slots -le 0 ]; then n_slots=1; fi
taskset -c 0-\${n_slots} gatk --java-options \"-Xmx${params.HCG_memory}G -XX:+UseParallelGC -XX:ParallelGCThreads=\${n_slots}\" HaplotypeCaller \\
--tmp-dir tmp/ \\
--pair-hmm-implementation AVX_LOGLESS_CACHING_OMP \\
--native-pair-hmm-threads \${n_slots} \\
-ERC GVCF \\
-L ${chrom} \\
-R ${ref_genome} \\
-I ${bam} \\
-O ${SampleID}_${chrom}.vcf ${params.GVCF_args}
"""
}
I think I solved this problem (at least for me, it worked well on SLURM). This comes from how GATK is configured for parallelizing jobs: it's based on OpenMP, so you should add to the beginning of your script something like this:
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
source
I'm trying to use Citrus to build a configurable mockup that I'd like to run from the command line passing different parameters to the test each time I run it.
I tried to use this as a reference to get started https://github.com/citrusframework/citrus/issues/325 but could not get it to work in my case.
I've got a running test running with one test case inside and I can run it like that
mvn clean verify -Dit.test=myTest#myTestCase
But when trying to run the following app:
package com.grge.citrus;
import com.consol.citrus.dsl.design.DefaultTestDesigner;
import com.consol.citrus.Citrus;
import com.consol.citrus.context.TestContext;
import org.springframework.context.ConfigurableApplicationContext;
import com.grge.citrus.*;
import com.consol.citrus.dsl.testng.TestNGCitrusTestDesigner;
public class App {
public static void main(String[] args) {
String suiteName = "mySuite";
Citrus citrus = Citrus.newInstance();
// my test with one testcase inside
TestNGCitrusTestDesigner myTest = new MyTest();
try {
citrus.beforeSuite(suiteName);
// This works it come from the sample
citrus.run(new SampleJavaDslTest().getTestCase());
// fails null pointer exception
citrus.run(myTest.getTestCase());
citrus.afterSuite(suiteName);
} finally {
((ConfigurableApplicationContext) citrus.getApplicationContext()).close();
}
}
private static class SampleJavaDslTest extends DefaultTestDesigner {
public SampleJavaDslTest() {
super();
echo("Hello from Java DSL!");
}
}
}
It fails with the following error
22:08:21,691 DEBUG citrus.Citrus| Loading Citrus application properties
22:08:21,696 DEBUG citrus.Citrus| Setting application property citrus.spring.java.config=com.grge.citrus.VCenterActorConfigSSL
22:08:22,025 DEBUG BeanDefinitionReader| Loaded 0 bean definitions from location pattern [classpath*:citrus-context.xml]
22:08:23,059 DEBUG server.HttpServer| Starting server: vCenterServer ...
22:08:23,217 DEBUG CachingServletFilter| Initializing filter 'request-caching-filter'
22:08:23,220 DEBUG CachingServletFilter| Filter 'request-caching-filter' configured successfully
22:08:23,220 DEBUG et.GzipServletFilter| Initializing filter 'gzip-filter'
22:08:23,220 DEBUG et.GzipServletFilter| Filter 'gzip-filter' configured successfully
22:08:23,222 DEBUG rusDispatcherServlet| Initializing servlet 'vCenterServer-servlet'
22:08:23,241 INFO rusDispatcherServlet| FrameworkServlet 'vCenterServer-servlet': initialization started
22:08:23,250 DEBUG rusDispatcherServlet| Servlet with name 'vCenterServer-servlet' will try to create custom WebApplicationContext context of class 'org.springframework.web.context.support.XmlWebApplicationContext', using parent context [null]
22:08:23,865 DEBUG rusDispatcherServlet| Unable to locate MultipartResolver with name 'multipartResolver': no multipart request handling provided
22:08:23,869 DEBUG rusDispatcherServlet| Unable to locate LocaleResolver with name 'localeResolver': using default [org.springframework.web.servlet.i18n.AcceptHeaderLocaleResolver#16fb356]
22:08:23,873 DEBUG rusDispatcherServlet| Unable to locate ThemeResolver with name 'themeResolver': using default [org.springframework.web.servlet.theme.FixedThemeResolver#1095f122]
22:08:23,894 DEBUG rusDispatcherServlet| No HandlerExceptionResolvers found in servlet 'vCenterServer-servlet': using default
22:08:23,896 DEBUG rusDispatcherServlet| Unable to locate RequestToViewNameTranslator with name 'viewNameTranslator': using default [org.springframework.web.servlet.view.DefaultRequestToViewNameTranslator#733c423e]
22:08:23,910 DEBUG rusDispatcherServlet| No ViewResolvers found in servlet 'vCenterServer-servlet': using default
22:08:23,915 DEBUG rusDispatcherServlet| Unable to locate FlashMapManager with name 'flashMapManager': using default [org.springframework.web.servlet.support.SessionFlashMapManager#681aad3b]
22:08:23,922 DEBUG rusDispatcherServlet| Published WebApplicationContext of servlet 'vCenterServer-servlet' as ServletContext attribute with name [org.springframework.web.servlet.FrameworkServlet.CONTEXT.vCenterServer-servlet]
22:08:23,922 INFO rusDispatcherServlet| FrameworkServlet 'vCenterServer-servlet': initialization completed in 678 ms
22:08:23,923 DEBUG rusDispatcherServlet| Servlet 'vCenterServer-servlet' configured successfully
22:08:54,266 INFO server.HttpServer| Started server: vCenterServer
22:08:54,362 INFO port.LoggingReporter|
22:08:54,362 INFO port.LoggingReporter| ------------------------------------------------------------------------
22:08:54,362 INFO port.LoggingReporter| .__ __
22:08:54,362 INFO port.LoggingReporter| ____ |__|/ |________ __ __ ______
22:08:54,362 INFO port.LoggingReporter| _/ ___\| \ __\_ __ \ | \/ ___/
22:08:54,362 INFO port.LoggingReporter| \ \___| || | | | \/ | /\___ \
22:08:54,362 INFO port.LoggingReporter| \___ >__||__| |__| |____//____ >
22:08:54,362 INFO port.LoggingReporter| \/ \/
22:08:54,362 INFO port.LoggingReporter|
22:08:54,363 INFO port.LoggingReporter| C I T R U S T E S T S 2.7.8
22:08:54,363 INFO port.LoggingReporter|
22:08:54,363 INFO port.LoggingReporter| ------------------------------------------------------------------------
22:08:54,363 DEBUG port.LoggingReporter| BEFORE TEST SUITE
22:08:54,363 INFO port.LoggingReporter|
22:08:54,363 INFO port.LoggingReporter|
22:08:54,363 INFO port.LoggingReporter| BEFORE TEST SUITE: SUCCESS
22:08:54,363 INFO port.LoggingReporter| ------------------------------------------------------------------------
22:08:54,363 INFO port.LoggingReporter|
22:08:54,370 DEBUG t.TestContextFactory| Created new test context - using global variables: '{}'
22:08:54,370 INFO port.LoggingReporter|
22:08:54,370 INFO port.LoggingReporter| ------------------------------------------------------------------------
22:08:54,370 DEBUG port.LoggingReporter| STARTING TEST SampleJavaDslTest <com.grge.citrus>
22:08:54,370 INFO port.LoggingReporter|
22:08:54,370 DEBUG citrus.TestCase| Initializing test case
22:08:54,372 DEBUG context.TestContext| Setting variable: citrus.test.name with value: 'SampleJavaDslTest'
22:08:54,372 DEBUG context.TestContext| Setting variable: citrus.test.package with value: 'com.grge.citrus'
22:08:54,372 DEBUG citrus.TestCase| Test variables:
22:08:54,372 DEBUG citrus.TestCase| citrus.test.name = SampleJavaDslTest
22:08:54,372 DEBUG citrus.TestCase| citrus.test.package = com.grge.citrus
22:08:54,373 INFO actions.EchoAction| Hello from Java DSL!
22:08:54,493 INFO port.LoggingReporter|
22:08:54,493 INFO port.LoggingReporter| TEST SUCCESS SampleJavaDslTest (com.grge.citrus)
22:08:54,493 INFO port.LoggingReporter| ------------------------------------------------------------------------
22:08:54,493 INFO port.LoggingReporter|
Exception in thread "main" java.lang.NullPointerException
at com.consol.citrus.dsl.testng.TestNGCitrusTestDesigner.getTestCase(TestNGCitrusTestDesigner.java:113)
at com.grge.citrus.App.main(App.java:25)
Effectively using the debugger I can see that when entering the try section, myTest has:
applicationContext
testDesigner
and many other attributes set to null and so the testDesigner.getTestCase() raise an Exception
Thx
Since Citrus 2.7 the framework provides a main CLI class that does exactly what you want:
https://github.com/citrusframework/citrus/blob/10d3954fd6c051d32e9cd974f685322c6454c779/modules/citrus-core/src/main/java/com/consol/citrus/main/CitrusApp.java#L38
Citrus application option usage:
-h or --help = Displays cli option usage
-d or --duration = Maximum time in milliseconds the server should be up and running - server will terminate automatically when time exceeds
-c or --config = Custom Spring configuration class
-s or --skipTests = Skip test execution
-p or --package = Test package to execute
-D or --properties = Default system properties to set
-e or --exit = Force system exit when finished
-t or --test = Test class/method to execute
-j or --jar = External test jar to load tests from
In the process of making a straightforward change to data in the region (cleared the region and re-loaded the new dataset). The data in the region had an updated class structure, so I updated the domain class / serializedID and repackaged the jar that I add to the the server classpath to help out with queries in pulse.
I have a simple setup, 1 locator 2 server/members
gfsh>list members
Name | Id
-------- | -------------------------------------------------
locator1 | 10.32.XX.XXX(locator1:11077:locator)<ec><v0>:1024
server1 | 10.32.XX.XXX(server1:21045)<v22>:1025
With a replicated persistent region:
gfsh>describe region --name=NPI
..........................................................
Name : NPI
Data Policy : persistent replicate
Hosting Members : server1
Non-Default Attributes Shared By Hosting Members
Type | Name | Value
------ | ----------- | --------------------
Region | data-policy | PERSISTENT_REPLICATE
| size | 5069949
| scope | distributed-ack
I brought down sever2 when I tried to restart I got this error:
gfsh>start server --name=server2 --server-port=40412 --group=NPI_GRP --classpath /opt/app/proj/npidomain.jar
Starting a Geode Server in /opt/app/proj/server2...
The Cache Server process terminated unexpectedly with exit status 1. Please refer to the log file in /opt/app/proj/server2 for full details.
Exception in thread "main" org.apache.geode.cache.CacheXmlException: Error while parsing XML, caused by org.xml.sax.SAXParseException; lineNumber: 2; columnNumber: 419; cvc-complex-type.3.1: Value '1.0' of attribute 'version' of element 'cache' is not valid with respect to the corresponding attribute use. Attribute 'version' has a fixed value of '8.1'.
at org.apache.geode.internal.cache.xmlcache.CacheXml.error(CacheXml.java:907)
at org.apache.geode.internal.cache.xmlcache.CacheXmlParser$DefaultHandlerDelegate.error(CacheXmlParser.java:3613)
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.error(ErrorHandlerWrapper.java:137)....
This is Apache Geode 1.0.0-incubating running on RHEL6 Java 1.8.0_101. Neither of these variables have change since the error started.
Not even sure where to start looking as very little changed between deployments.
Even though both gemfire and geode are installed on the same machine I am certain that I'm in the correct gfsh:
[geodeusr#hostname proj]$ ./gfsh
_________________________ __
/ _____/ ______/ ______/ /____/ /
/ / __/ /___ /_____ / _____ /
/ /__/ / ____/ _____/ / / / /
/______/_/ /______/_/ /_/ 1.0.0-incubating
Monitor and Manage Apache Geode (incubating)
gfsh>
I am getting this error while running the application. I am using postgres 9.4 database. I ensured that DataSource.groovy exists. I am on MAC OS X 10.10
grails> r-a
| Starting daemon...
| Compiling 450 source files
| Parent process shutdown. Exiting...
| Error Forked Grails VM exited with error
| Starting daemon.....
| Warning No config found for the application.
| Warning DataSource.groovy not found, assuming dataSource bean is configured by Spring
| Running Grails application
objc[2087]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.7.0_75.jdk/Contents/Home/bin/java and /Library/Java/JavaVirtualMachines/jdk1.7.0_75.jdk/Contents/Home/jre/lib/libinstrument.dylib. One of the two will be used. Which one is undefined.
| Error 2015-04-10 12:24:40,201 [localhost-startStop-1] ERROR context.ContextLoader - Context initialization failed
Message: Error creating bean with name 'grailsApplication' defined in ServletContext resource [/WEB-INF/applicationContext.xml]: Invocation of init method failed; nested exception is org.codehaus.groovy.grails.exceptions.GrailsConfigurationException: Class not found loading Grails application: BootStrap
Line | Method
->> 262 | run in java.util.concurrent.FutureTask
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
| 1145 | runWorker in java.util.concurrent.ThreadPoolExecutor
| 615 | run . . . in java.util.concurrent.ThreadPoolExecutor$Worker
^ 745 | run in java.lang.Thread
Caused by GrailsConfigurationException: Class not found loading Grails application: BootStrap
->> 262 | run in java.util.concurrent.FutureTask
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
| 1145 | runWorker in java.util.concurrent.ThreadPoolExecutor
| 615 | run . . . in java.util.concurrent.ThreadPoolExecutor$Worker
^ 745 | run in java.lang.Thread
Caused by ClassNotFoundException: BootStrap
->> 366 | run in java.net.URLClassLoader$1
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
| 355 | run in ''
| 354 | findClass in java.net.URLClassLoader
| 425 | loadClass in java.lang.ClassLoader
| 262 | run . . . in java.util.concurrent.FutureTask
| 1145 | runWorker in java.util.concurrent.ThreadPoolExecutor
| 615 | run . . . in java.util.concurrent.ThreadPoolExecutor$Worker
^ 745 | run in java.lang.Thread
| Error 2015-04-10 12:24:40,261 [localhost-startStop-1] ERROR context.GrailsContextLoaderListener - Error initializing the application: Error creating bean with name 'grailsApplication' defined in ServletContext resource [/WEB-INF/applicationContext.xml]: Invocation of init method failed; nested exception is org.codehaus.groovy.grails.exceptions.GrailsConfigurationException: Class not found loading Grails application: BootStrap
Message: Error creating bean with name 'grailsApplication' defined in ServletContext resource [/WEB-INF/applicationContext.xml]: Invocation of init method failed; nested exception is org.codehaus.groovy.grails.exceptions.GrailsConfigurationException: Class not found loading Grails application: BootStrap
Line | Method
->> 262 | run in java.util.concurrent.FutureTask
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
| 1145 | runWorker in java.util.concurrent.ThreadPoolExecutor
| 615 | run . . . in java.util.concurrent.ThreadPoolExecutor$Worker
^ 745 | run in java.lang.Thread
Caused by GrailsConfigurationException: Class not found loading Grails application: BootStrap
->> 262 | run in java.util.concurrent.FutureTask
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
| 1145 | runWorker in java.util.concurrent.ThreadPoolExecutor
| 615 | run . . . in java.util.concurrent.ThreadPoolExecutor$Worker
^ 745 | run in java.lang.Thread
Caused by ClassNotFoundException: BootStrap
->> 366 | run in java.net.URLClassLoader$1
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
| 355 | run in ''
| 354 | findClass in java.net.URLClassLoader
| 425 | loadClass in java.lang.ClassLoader
| 262 | run . . . in java.util.concurrent.FutureTask
| 1145 | runWorker in java.util.concurrent.ThreadPoolExecutor
| 615 | run . . . in java.util.concurrent.ThreadPoolExecutor$Worker
^ 745 | run in java.lang.Thread
| Error 2015-04-10 12:24:40,277 [localhost-startStop-1] ERROR core.StandardContext - Error listenerStart
| Error 2015-04-10 12:24:40,288 [localhost-startStop-1] ERROR core.StandardContext - Context [/yazh] startup failed due to previous errors
| Server running. Browse to http://localhost:8080/yazh
| Application loaded in interactive mode. Type 'stop-app' to shutdown.
| Enter a script name to run. Use TAB for completion:
| Parent process shutdown. Exiting...
| Error Forked Grails VM exited with error
This usually happens when there's an error in the Domain, Controller or Service classes. run with --stacktrace and you'll find the root cause
This isn't much of an answer, but I just experienced this error and was able to get rid of it with a grails clean and grails compile.
I am struggling this for days now.
I am using SI adaptor for kafka under Spring-boot container.
I have configured zookeeper and kafka on my machine.
I also created console producer and consumer tested it and everything works fine(I manage to produce vis the console messages and have the console consumer to consume them).
I tried now to produce messages via Spring integration kafka outbound adapter but the console consumer wont consume the message
SI/Spring xd xml:
<int:publish-subscribe-channel id="inputToKafka"/>
<int-kafka:outbound-channel-adapter id="kafkaOutboundChannelAdapter"
kafka-producer-context-ref="kafkaProducerContext"
auto-startup="true"
order="1"
channel="inputToKafka">
</int-kafka:outbound-channel-adapter>
<int-kafka:producer-context id="kafkaProducerContext">
<int-kafka:producer-configurations>
<int-kafka:producer-configuration broker-list="localhost:9092"
async="true"
topic="zerg.hydra"
compression-codec="default"/>
</int-kafka:producer-configurations>
</int-kafka:producer-context>
<task:executor id="taskExecutor" pool-size="5" keep-alive="120" queue-capacity="500"/>
</beans>
Java:
#Named
public class KafkaProducer {
#Autowired
#Qualifier("inputToKafka")
MessageChannel inputToKafka;
public void sendMessageToKafka(String message)
{
inputToKafka.send(
MessageBuilder.withPayload(message)
.setHeader("messageKey", "3")
.setHeader("topic", "zerg.hydra").build());
}
}
this is how I run kafka console consumer:
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic zerg.hydra --from-beginning
logs:
Testing started at 12:49 PM ...
12:49:54 PM: Executing external tasks 'cleanTest test'...
:cleanTest
:compileJava
:processResources UP-TO-DATE
:classes
:compileTestJava
:processTestResources UP-TO-DATE
:testClasses
:test
12:50:07,165 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
12:50:07,166 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
12:50:07,166 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/Users/idan/dev/Projects/CalcMicroService/build/resources/test/logback.xml]
12:50:07,167 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs multiple times on the classpath.
12:50:07,167 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [file:/Users/idan/dev/Projects/CalcMicroService/build/resources/test/logback.xml]
12:50:07,167 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [file:/Users/idan/dev/Projects/CalcMicroService/build/resources/main/logback.xml]
12:50:07,247 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
12:50:07,250 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
12:50:07,259 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [stdout]
12:50:07,281 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
12:50:07,327 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [reactor] to INFO
12:50:07,327 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.projectreactor] to INFO
12:50:07,328 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.springframework] to WARN
12:50:07,328 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.springframework.integration] to DEBUG
12:50:07,328 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
12:50:07,328 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [stdout] to Logger[ROOT]
12:50:07,331 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
12:50:07,332 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator#3de433b4 - Registering current configuration as safe fallback point
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v1.1.9.RELEASE)
12:50:09.530 [Test worker] INFO o.s.i.config.IntegrationRegistrar - No bean named 'integrationHeaderChannelRegistry' has been explicitly defined. Therefore, a default DefaultHeaderChannelRegistry will be created.
12:50:09.544 [Test worker] DEBUG o.s.i.config.IntegrationRegistrar - SpEL function '#xpath' isn't registered: there is no spring-integration-xml.jar on the classpath.
12:50:10.717 [Test worker] INFO o.s.i.c.DefaultConfiguringBeanFactoryPostProcessor - No bean named 'errorChannel' has been explicitly defined. Therefore, a default PublishSubscribeChannel will be created.
12:50:10.719 [Test worker] INFO o.s.i.c.DefaultConfiguringBeanFactoryPostProcessor - No bean named 'taskScheduler' has been explicitly defined. Therefore, a default ThreadPoolTaskScheduler will be created.
12:50:10.973 DEBUG [Test worker][org.jboss.logging] Logging Provider: org.jboss.logging.Log4jLoggerProvider
12:50:10.974 INFO [Test worker][org.hibernate.validator.internal.util.Version] HV000001: Hibernate Validator 5.0.3.Final
12:50:10.991 DEBUG [Test worker][org.hibernate.validator.internal.engine.resolver.DefaultTraversableResolver] Cannot find javax.persistence.Persistence on classpath. Assuming non JPA 2 environment. All properties will per default be traversable.
12:50:11.006 DEBUG [Test worker][org.hibernate.validator.internal.engine.ConfigurationImpl] Setting custom MessageInterpolator of type org.springframework.validation.beanvalidation.LocaleContextMessageInterpolator
12:50:11.011 DEBUG [Test worker][org.hibernate.validator.internal.engine.ConfigurationImpl] Setting custom ConstraintValidatorFactory of type org.springframework.validation.beanvalidation.SpringConstraintValidatorFactory
12:50:11.018 DEBUG [Test worker][org.hibernate.validator.internal.engine.ConfigurationImpl] Setting custom ParameterNameProvider of type com.sun.proxy.$Proxy46
12:50:11.024 DEBUG [Test worker][org.hibernate.validator.internal.xml.ValidationXmlParser] Trying to load META-INF/validation.xml for XML based Validator configuration.
12:50:11.032 DEBUG [Test worker][org.hibernate.validator.internal.xml.ValidationXmlParser] No META-INF/validation.xml found. Using annotation based configuration only.
12:50:12.089 [Test worker] INFO o.a.catalina.core.StandardService - Starting service Tomcat
12:50:12.091 [Test worker] INFO o.a.catalina.core.StandardEngine - Starting Servlet Engine: Apache Tomcat/7.0.56
12:50:14.162 [localhost-startStop-1] INFO o.a.c.c.C.[Tomcat].[localhost].[/] - Initializing Spring embedded WebApplicationContext
12:50:16.567 [Test worker] INFO o.s.i.k.support.ProducerFactoryBean - Using producer properties => {metadata.broker.list=localhost:9092, compression.codec=0, producer.type=async}
12:50:17.036 INFO [Test worker][kafka.utils.VerifiableProperties] Verifying properties
12:50:17.096 INFO [Test worker][kafka.utils.VerifiableProperties] Property compression.codec is overridden to 0
12:50:17.096 INFO [Test worker][kafka.utils.VerifiableProperties] Property metadata.broker.list is overridden to localhost:9092
12:50:17.096 INFO [Test worker][kafka.utils.VerifiableProperties] Property producer.type is overridden to async
12:50:17.591 DEBUG [Test worker][org.hibernate.validator.internal.engine.resolver.DefaultTraversableResolver] Cannot find javax.persistence.Persistence on classpath. Assuming non JPA 2 environment. All properties will per default be traversable.
12:50:17.591 DEBUG [Test worker][org.hibernate.validator.internal.engine.ConfigurationImpl] Setting custom MessageInterpolator of type org.springframework.validation.beanvalidation.LocaleContextMessageInterpolator
12:50:17.591 DEBUG [Test worker][org.hibernate.validator.internal.engine.ConfigurationImpl] Setting custom ConstraintValidatorFactory of type org.springframework.validation.beanvalidation.SpringConstraintValidatorFactory
12:50:17.591 DEBUG [Test worker][org.hibernate.validator.internal.engine.ConfigurationImpl] Setting custom ParameterNameProvider of type com.sun.proxy.$Proxy46
12:50:17.592 DEBUG [Test worker][org.hibernate.validator.internal.xml.ValidationXmlParser] Trying to load META-INF/validation.xml for XML based Validator configuration.
12:50:17.592 DEBUG [Test worker][org.hibernate.validator.internal.xml.ValidationXmlParser] No META-INF/validation.xml found. Using annotation based configuration only.
12:50:18.967 [Test worker] DEBUG o.s.i.c.GlobalChannelInterceptorProcessor - No global channel interceptors.
12:50:18.978 [Test worker] INFO o.s.i.endpoint.EventDrivenConsumer - Adding {mongo:outbound-channel-adapter:mongoAdapter.adapter} as a subscriber to the 'mongoAdapter' channel
12:50:18.979 [Test worker] INFO o.s.i.channel.DirectChannel - Channel 'application:8091.mongoAdapter' has 1 subscriber(s).
12:50:18.979 [Test worker] INFO o.s.i.endpoint.EventDrivenConsumer - started mongoAdapter.adapter
12:50:18.979 [Test worker] INFO o.s.i.endpoint.EventDrivenConsumer - Adding {mongo:outbound-channel-adapter:adapterWithConverter.adapter} as a subscriber to the 'adapterWithConverter' channel
12:50:18.979 [Test worker] INFO o.s.i.channel.DirectChannel - Channel 'application:8091.adapterWithConverter' has 1 subscriber(s).
12:50:18.979 [Test worker] INFO o.s.i.endpoint.EventDrivenConsumer - started adapterWithConverter.adapter
12:50:18.979 [Test worker] INFO o.s.i.endpoint.EventDrivenConsumer - Adding {message-handler:kafkaOutboundChannelAdapter} as a subscriber to the 'inputToKafka' channel
12:50:18.979 [Test worker] INFO o.s.i.c.PublishSubscribeChannel - Channel 'application:8091.inputToKafka' has 1 subscriber(s).
12:50:18.979 [Test worker] INFO o.s.i.endpoint.EventDrivenConsumer - started kafkaOutboundChannelAdapter
12:50:18.979 [Test worker] INFO o.s.i.endpoint.EventDrivenConsumer - Adding {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
12:50:18.979 [Test worker] INFO o.s.i.c.PublishSubscribeChannel - Channel 'application:8091.errorChannel' has 1 subscriber(s).
12:50:18.979 [Test worker] INFO o.s.i.endpoint.EventDrivenConsumer - started _org.springframework.integration.errorLogger
12:50:19.026 [Test worker] INFO o.a.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-8091"]
12:50:19.040 [Test worker] INFO o.a.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8091"]
12:50:19.047 [Test worker] INFO o.a.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read
12:50:19.338 [Test worker] DEBUG o.s.i.c.PublishSubscribeChannel - preSend on channel 'inputToKafka', message: [Payload String content=Hello Kafka From SI][Headers={messageKey=3, topic=zerg.hydra, id=be46fb68-c762-f16e-6ccb-6841ef3fe868, timestamp=1416999019338}]
12:50:19.338 [Test worker] DEBUG o.s.i.k.o.KafkaProducerMessageHandler - org.springframework.integration.kafka.outbound.KafkaProducerMessageHandler#0 received message: [Payload String content=Hello Kafka From SI][Headers={messageKey=3, topic=zerg.hydra, id=be46fb68-c762-f16e-6ccb-6841ef3fe868, timestamp=1416999019338}]
12:50:19.362 [Test worker] DEBUG o.s.i.c.PublishSubscribeChannel - postSend (sent=true) on channel 'inputToKafka', message: [Payload String content=Hello Kafka From SI][Headers={messageKey=3, topic=zerg.hydra, id=be46fb68-c762-f16e-6ccb-6841ef3fe868, timestamp=1416999019338}]
Empty test suite.
12:50:19.402 [Thread-4] INFO o.s.i.endpoint.EventDrivenConsumer - Removing {mongo:outbound-channel-adapter:mongoAdapter.adapter} as a subscriber to the 'mongoAdapter' channel
12:50:19.417 [Thread-4] INFO o.s.i.channel.DirectChannel - Channel 'application:8091.mongoAdapter' has 0 subscriber(s).
12:50:19.419 [Thread-4] INFO o.s.i.endpoint.EventDrivenConsumer - stopped mongoAdapter.adapter
12:50:19.420 [Thread-4] INFO o.s.i.endpoint.EventDrivenConsumer - Removing {mongo:outbound-channel-adapter:adapterWithConverter.adapter} as a subscriber to the 'adapterWithConverter' channel
12:50:19.422 [Thread-4] INFO o.s.i.channel.DirectChannel - Channel 'application:8091.adapterWithConverter' has 0 subscriber(s).
12:50:19.422 [Thread-4] INFO o.s.i.endpoint.EventDrivenConsumer - stopped adapterWithConverter.adapter
12:50:19.423 [Thread-4] INFO o.s.i.endpoint.EventDrivenConsumer - Removing {message-handler:kafkaOutboundChannelAdapter} as a subscriber to the 'inputToKafka' channel
12:50:19.424 [Thread-4] INFO o.s.i.c.PublishSubscribeChannel - Channel 'application:8091.inputToKafka' has 0 subscriber(s).
12:50:19.424 [Thread-4] INFO o.s.i.endpoint.EventDrivenConsumer - stopped kafkaOutboundChannelAdapter
12:50:19.425 [Thread-4] INFO o.s.i.endpoint.EventDrivenConsumer - Removing {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
12:50:19.426 [Thread-4] INFO o.s.i.c.PublishSubscribeChannel - Channel 'application:8091.errorChannel' has 0 subscriber(s).
12:50:19.426 [Thread-4] INFO o.s.i.endpoint.EventDrivenConsumer - stopped _org.springframework.integration.errorLogger
BUILD SUCCESSFUL
Total time: 25.469 secs
12:50:20 PM: External tasks execution finished 'cleanTest test'.
I tried in the same app to use the Offical Kafka client producer and it worked just fine:
#Named
public class KafkaProducerJava {
ProducerConfig config=null;
Producer<String, String> producer;
Properties props=null;
#PostConstruct
public void init()
{
props = new Properties();
props.put("metadata.broker.list", "localhost:9092");
props.put("serializer.class", "kafka.serializer.StringEncoder");
// props.put("partitioner.class", "example.producer.SimplePartitioner");
props.put("request.required.acks", "1");
}
public void sendMsgToKafka(String msg)
{
config= new ProducerConfig(props);
producer=new Producer<String, String>(config);
KeyedMessage<String, String> data = new KeyedMessage<String, String>("test", "", msg);
producer.send(data);
producer.close();
}
Any idea why the message never reached to my consumer via the Spring Integration kafka adaptor??
I just ran it in XD and it worked fine for me...
$ bin/xd-shell
_____ __ _______
/ ___| (-) \ \ / / _ \
\ `--. _ __ _ __ _ _ __ __ _ \ V /| | | |
`--. \ '_ \| '__| | '_ \ / _` | / ^ \| | | |
/\__/ / |_) | | | | | | | (_| | / / \ \ |/ /
\____/| .__/|_| |_|_| |_|\__, | \/ \/___/
| | __/ |
|_| |___/
eXtreme Data
1.1.0.BUILD-SNAPSHOT | Admin Server Target: http://localhost:9393
Welcome to the Spring XD shell. For assistance hit TAB or type "help".
xd:>stream create --name foo --definition "time | kafka --topic=test" --deploy
Created and deployed new stream 'foo'
xd:>stream destroy foo
Destroyed stream 'foo'
xd:>
.
$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
2014-11-26 10:03:09
2014-11-26 10:03:10
2014-11-26 10:03:11
2014-11-26 10:03:12
2014-11-26 10:03:13
I also wrote this test case...
public class OutboundTests {
#Test
public void test() throws Exception {
KafkaProducerContext<String, String> kafkaProducerContext = new KafkaProducerContext<String, String>();
ProducerMetadata<String, String> producerMetadata = new ProducerMetadata<String, String>("test");
producerMetadata.setValueClassType(String.class);
producerMetadata.setKeyClassType(String.class);
Encoder<String> encoder = new StringEncoder<String>();
producerMetadata.setValueEncoder(encoder);
producerMetadata.setKeyEncoder(encoder);
ProducerFactoryBean<String, String> producer = new ProducerFactoryBean<String, String>(producerMetadata, "localhost:9092");
ProducerConfiguration<String, String> config = new ProducerConfiguration<String, String>(producerMetadata, producer.getObject());
kafkaProducerContext.setProducerConfigurations(Collections.singletonMap("test", config));
KafkaProducerMessageHandler<String, String> handler = new KafkaProducerMessageHandler<String, String>(kafkaProducerContext);
handler.handleMessage(MessageBuilder.withPayload("foo")
.setHeader("messagekey", "3")
.setHeader("topic", "test")
.build());
}
}
...to simulate what you are doing and that worked too. Are you seeing any logged exceptions? I see you are not setting the key/value types and encoders.
EDIT:
As discussed in the comments, the issue is that you are using an async producer. In your test example that "works", you are closing the producer, which flushes the queue. By default, the queue won't be flushed for 5 seconds and your test case is not waiting long enough.
I updated my test to include an XML configured version, and reduced the queue.buffering.max.ms to 500ms and added code to the test to wait a couple of seconds before terminating.
See the new commit for details.
I was facing the same issue since morning and finally found the solution. If you prefix the topic with kafka as given below
.setHeader("kafka_topic", "zerg.hydra")
This is a issue with API as internally, while looking up the key from from the message header, kafka_ is getting prefixed.