I am running a scheduled job in a web application which sometimes (could notreproduce it ) results in the following exception:
[WebappClassLoader] Illegal access: this web application instance has been stopped already. Could not load org.quartz.StatefulJob. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1244)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1204)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at org.quartz.JobDetail.class$(JobDetail.java:279)
at org.quartz.JobDetail.isStateful(JobDetail.java:425)
at org.quartz.simpl.RAMJobStore.triggerFired(RAMJobStore.java:1313)
at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:342)
13:41:00,083 ERROR [STDERR] Exception in thread "DefaultQuartzScheduler_QuartzSchedulerThread"
13:41:00,083 ERROR [STDERR] java.lang.NoClassDefFoundError: org.quartz.StatefulJob
13:41:00,083 ERROR [STDERR] at org.quartz.JobDetail.class$(JobDetail.java:279)
13:41:00,083 ERROR [STDERR] at org.quartz.JobDetail.isStateful(JobDetail.java:425)
13:41:00,083 ERROR [STDERR] at org.quartz.simpl.RAMJobStore.triggerFired(RAMJobStore.java:1313)
13:41:00,083 ERROR [STDERR] at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:342)
There also a NoClassDefFoundError. It says that org.quartz.StatefulJob is not found.
Here is how the job is scheduled:
Scheduler sched = StdSchedulerFactory.getDefaultScheduler();
if (!sched.isStarted()){
sched.start();
}
String konf = MyConfigClass.getRow(25).getKonfiguration();
Calendar cal = Calendar.getInstance();
cal.setTime(MyParser.stf.parse(konf));
String expression = "0 " + cal.get(Calendar.MINUTE) + " " + cal.get(Calendar.HOUR_OF_DAY) + " ? * MON-FRI";
CronTrigger ct = new CronTrigger(triggerName, group, jobName, group, expression);
if (sched.getTrigger(triggerName, Scheduler.DEFAULT_GROUP) != null) {
sched.rescheduleJob(triggerName, group, ct);
} else {
JobDetail jd = new JobDetail(jobName, group, MyJob.class);
sched.scheduleJob(jd, ct);
}
I do not know what could be the problem. First I thought that the session dies before the job executes, but I've tried it and that's no the case. The interesting thing is that after this exception, the job runs again with no problem.
Do you have any ideas?
It looks like your quartz job is trying to do something after your app was stopped. It might be trying to access some resources that were available while your app was running, but not anymore (something like getResourceAsStream maybe).
It might help if you post more info: stacktraces, code of your job, etc.
After update:
Here is what I think is happening:
You schedule your job successfully
For whatever reason your webapp is stopped. I would search JBoss logs for a reason.
At this moment time comes to execute your job (org.quartz.simpl.RAMJobStore.triggerFired(RAMJobStore.java:1313))
This call goes down to isStateful method of JobDetail class
The method has this code in it return (StatefulJob.class.isAssignableFrom(jobClass));
StatefulJob class was not loaded before. Your classLoader tries to load it (org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1204))
Since your webapp was stopped, you are getting exception
The way I see it, you have two choices:
Ignore this problem, since there is nothing you can do to make quartz work after your webapp was stopped
Try to figure out what causes your webapp to stop and fix it
Related
We've been using
jaxws-spring
WebSphere 8.5.5.2
for our web-services interactions, and now we get stuck with such "cause" when we call any web-method on the client-side -
ServletWrappe E com.ibm.ws.webcontainer.servlet.ServletWrapper service SRVE0014E: Uncaught service() exception root cause appServlet:
org.springframework.web.util.NestedServletException: Request processing failed; nested exception is javax.xml.ws.soap.SOAPFaultException: java.lang.NullPointerException
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:948)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:827)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:812)
.....
Caused by: javax.xml.ws.soap.SOAPFaultException: java.lang.NullPointerException
at org.apache.axis2.jaxws.marshaller.impl.alt.MethodMarshallerUtils.createSystemException(MethodMarshallerUtils.java:1363)
at org.apache.axis2.jaxws.marshaller.impl.alt.MethodMarshallerUtils.demarshalFaultResponse(MethodMarshallerUtils.java:1089)
at org.apache.axis2.jaxws.marshaller.impl.alt.DocLitWrappedMethodMarshaller.demarshalFaultResponse(DocLitWrappedMethodMarshaller.java:680)
at org.apache.axis2.jaxws.client.proxy.JAXWSProxyHandler.getFaultResponse(JAXWSProxyHandler.java:626)
....
Caused by: java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:946)
at org.apache.axis2.jaxws.message.databinding.JAXBUtils.getJAXBContext(JAXBUtils.java:445)
at org.apache.axis2.datasource.jaxb.JAXBDSContext.getJAXBContext(JAXBDSContext.java:228)
at org.apache.axis2.datasource.jaxb.JAXBDSContext.getJAXBContext(JAXBDSContext.java:161)
at org.apache.axis2.datasource.jaxb.JAXBDSContext.marshal(JAXBDSContext.java:396)
at org.apache.axis2.jaxws.message.databinding.impl.JAXBBlockImpl._outputFromBO(JAXBBlockImpl.java:189)
at org.apache.axis2.jaxws.message.impl.BlockImpl.outputTo(BlockImpl.java:371)
at org.apache.axis2.jaxws.message.impl.BlockImpl.serialize(BlockImpl.java:295)
at org.apache.axiom.om.impl.llom.OMSourcedElementImpl.internalSerialize(OMSourcedElementImpl.java:781)
at org.apache.axiom.om.impl.llom.OMElementImpl.internalSerialize(OMElementImpl.java:967)
at org.apache.axiom.soap.impl.llom.SOAPEnvelopeImpl.serializeInternally(SOAPEnvelopeImpl.java:283)
at org.apache.axiom.soap.impl.llom.SOAPEnvelopeImpl.internalSerialize(SOAPEnvelopeImpl.java:245)
at org.apache.axiom.om.impl.llom.OMSerializableImpl.serializeAndConsume(OMSerializableImpl.java:207)
at org.apache.axis2.transport.http.SOAPMessageFormatter.writeTo(SOAPMessageFormatter.java:74)
Web-service client initialisation comes out with no any errors -
URL service_endpoint;
service_endpoint = new URL(endpoint);
service = new UDHandlersServiceImpl(service_endpoint, SERVICE_QNAME).getUDHandlersServiceImplPort();
NullPointerException occurs after calling any method of "service". And it doesn't even reach server side - because it doesn't have any logs there. Furthermore - such situation occurs only on the WebSphere - we deployed our client application on the Tomcat (server remained on websphere) and it worked fine. One more thing - SoapUI does not have any problems interacting with server side too.
We tried to switch classLoadings to parentLast for client application - with no success.
Also I can't even find sources for "org.apache.axis2.jaxws.message.databinding.JAXBUtils.getJAXBContext(JAXBUtils.java:445)". I googled it and didn't find this version of JAXBUtils. But even if i did, i think, it wouldnt be useful...
I hope someone can help us with that unobvious problem.
Thanks beforehand.
===== updated======
When we remove jaxb libraries from endorsed dir (or appServer/classes) problem seems to disappear but in case of arising webExceptions(faults) server throws
Caused by: java.lang.ClassCastException: com.ibm.xml.xlxp2.jaxb.JAXBContextImpl incompatible with com.sun.xml.bind.api.JAXBRIContext
at com.sun.xml.ws.fault.SOAPFaultBuilder.<clinit>(SOAPFaultBuilder.java:565)
at java.lang.J9VMInternals.initializeImpl(Native Method)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:237)
and returns no answer to the client.
We've realized that our exceptions didn't contain 3 required methods -
//============required==================
public DocumentValidationException(String message, ErrorsBean errorsBean, Throwable cause) {
super(message, cause);
this.errorsBean = errorsBean;
}
public ErrorsBean getFaultInfo() {
return errorsBean;
}
public DocumentValidationException(String message, ErrorsBean errorsBean) {
super(message);
this.errorsBean = errorsBean;
}
//======================================
so, now it does.
And also we had some wrong target namespaces in it at the #WebFault annotations.
Here is the link with similar problem - https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000014076927#77777777-0000-0000-0000-000014083314
My application is running for months and working very well. Then suddenly I get the following error:
com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.spi.impl.ProxyServiceImpl$ProxyRegistry.<init>(ProxyServiceImpl.java:220)
at com.hazelcast.spi.impl.ProxyServiceImpl$ProxyRegistry.<init>(ProxyServiceImpl.java:207)
at com.hazelcast.spi.impl.ProxyServiceImpl$1.createNew(ProxyServiceImpl.java:69)
at com.hazelcast.spi.impl.ProxyServiceImpl$1.createNew(ProxyServiceImpl.java:67)
at com.hazelcast.util.ConcurrencyUtil.getOrPutIfAbsent(ConcurrencyUtil.java:47)
at com.hazelcast.spi.impl.ProxyServiceImpl.getDistributedObject(ProxyServiceImpl.java:101)
at com.hazelcast.instance.HazelcastInstanceImpl.getDistributedObject(HazelcastInstanceImpl.java:285)
at com.hazelcast.instance.HazelcastInstanceImpl.getLock(HazelcastInstanceImpl.java:183)
at com.hazelcast.instance.HazelcastInstanceProxy.getLock(HazelcastInstanceProxy.java:77)
at br.com.xyz.lock.hazelcast.HazelcastLockManager.lock(HazelcastLockManager.java:37)
at br.com.xyz.lock.hazelcast.LockManagerFacade.lock(LockManagerFacade.java:24)
at br.com.xyz.recebe.negocio.NProcessadorMensagemRecebida.processamentoLock(NProcessadorMensagemRecebida.java:85)
at br.com.xyz.recebe.negocio.NProcessadorMensagemRecebida.processaArquivo(NProcessadorMensagemRecebida.java:74)
at br.com.xyz.recebe.processador.ProcessadorBase.processaArquivo(ProcessadorBase.java:75)
at br.com.xyz.recebe.processador.ProcessadorXml.processaArquivo(ProcessadorXml.java:16)
at br.com.xyz.recebe.processador.ProcessadorFacade.processaArquivo(ProcessadorFacade.java:34)
at br.com.xyz.recebe.mail.pdes.ProcessadorPDESMeRecebida.processar(ProcessadorPDESMeRecebida.java:77)
at gov.sefaz.util.pdes.ProcessadorDiretorioEntradaSaidaDaemon.processar(ProcessadorDiretorioEntradaSaidaDaemon.java:575)
at gov.sefaz.util.pdes.ProcessadorDiretorioEntradaSaidaDaemon.varrerDiretorioUsingStrategy(ProcessadorDiretorioEntradaSaidaDaemon.java:526)
at gov.sefaz.util.pdes.ProcessadorDiretorioEntradaSaidaDaemon.run(ProcessadorDiretorioEntradaSaidaDaemon.java:458)
at java.lang.Thread.run(Unknown Source)
I found some issues on Google that says it shutdown because other errors. But in my case there isn't any.
It shutdown without reason.
Has anyone seen this before?
The problem occurs if a hazelcast member doesn't shutdown normally (terminate), When you stop HazelcastInstance which queue proxy is bound to; then any operation on that queue after instance stopped should throw HazelcastInstanceNotActiveException.
You just need to wait for a few minutes and rerun your job.
I am attempting to start up my hadoop application, however upon startup i am seeing this in the log files, does anyone have a clue as to what the problem is?
Creating filesystem for hdfs://10.170.4.141:9000
java.io.IOException: config()
at org.apache.hadoop.conf.Configuration.(Configuration.java:229)
at org.apache.hadoop.conf.Configuration.(Configuration.java:216)
at org.apache.hadoop.security.SecurityUtil.(SecurityUtil.java:60)
at org.apache.hadoop.net.NetUtils.makeSocketAddr(NetUtils.java:188)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:168)
at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:198)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:88)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1413)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:68)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1431)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:256)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:125)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:240)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:372)
at org.blismedia.VolumeReportGenerateUpdates.main(VolumeReportGenerateUpdates.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:187)
I think you're running into HADOOP-2851. This "error" can safely be ignored.
Apparently, Configuration's constructor logs an exception to the debug log, despite no exception actually being thrown. Why? Your guess is as good as mine. But the issue is resolved in their project as won't fix. "It's a feature, not a bug."
public Configuration(boolean loadDefaults) {
if (LOG.isDebugEnabled()) {
LOG.debug(StringUtils.stringifyException(new IOException("config()")));
}
// ...
}
I have a task scheduler implemented in my application. Basically, what i do is schedule some task's to be executed 4 times in a day (like 6 in 6 hours), so the system schedules it to: 00:00, 06:00, 12:00, 18:00.
Ok, i have a class (FlowJobController) which extends the Thread class and in the run() implementation i keep sleeping the thread in 60 to 60 seconds, so it's executed again to check if there's any task to be executed. If true, i run my Job.
Basically the main part it:
rSet = pStmt.executeQuery();
while (rSet.next()) {
long jobId = rSet.getLong("trf_codigo");
String ruleName = rSet.getString("reg_nome");
String ruleParameters = rSet.getString("trf_regra_parametros");
Job job = new Job();
job.setId(jobId);
job.setRuleName(ruleName);
job.setParameters(Functions.stringToList(ruleParameters, "\n"));
FlowJob flowJob = new FlowJob(this, job);
flowJob.start();
}
} catch (Exception ex) {
logger.error(WFRSystem.DEFAULT_USER, system.getCode(), ex);
} finally {
try {
DBConnection.close(pStmt);
DBConnection.close(rSet);
// executede 60 in 60 sec
Thread.sleep(60 * 1000);
} catch (InterruptedException ex) {
logger.error(WFRSystem.DEFAULT_USER, system.getCode(), ex);
}
}
The thing is: When the pStmt.executeQUery() returns records to be executed, it goes into the while and the error appears into the line: Job job = new Job();
The error is:
Exception in thread "FlowJobController" java.lang.NoClassDefFoundError: wfr/com/Job
at wfr.com.FlowJobController.run(FlowJobController.java:112)
Before this error i got this error:
25/09/2012 12:00:09 org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load wfr.com.Job.
The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1566)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at wfr.com.FlowJobController.run(FlowJobController.java:112)
The FlowJobController.java:112 is the Job job = new Job();
What am i doing wrong?
The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
If you are saying that your code deliberately threw the IllegalStateException exception, then that is the likely cause of the subsequent problems. If you throw an uncheck exception in static initialization, and that exception propagates, then the class initialization fails, and the class (and all classes that depend on it) are unusable. In this case, it looks like it also caused the webapp to be stopped.
If this is the problem, the solution is "don't do that". Fix whatever is causing the exception to be thrown.
The other possibility is that this is simply a missing class problem:
Well, that sounds not possible (to not be in the war) since i'm running into the eclipse environment and it's in the class folder directory and no errors is shown about not finding those classes...
The class loader won't be looking in the Eclipse classes folder. It will be looking in the webapp directory on your web container (Tomcat, Jetty, whatever).
I have a java webapp using an ibatis row handler to load a very large dataset (1 million rows in an innodb table). The process is run as a nightly cron job by quartz scheduler. However, after it processes for 6 minutes, it dies with the following stack trace:
WARN [DefaultQuartzScheduler_Worker-8] MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(168) | Could not invoke method 'doBatch' on target object [org.myCron#4adb34]
org.springframework.jdbc.UncategorizedSQLException: SqlMapClient operation: encountered SQLException [
--- The error occurred in org/myCron/mySqlMap.xml.
--- The error occurred while applying a result map.
--- Check the mySqlMap.outputMapping.
--- The error happened while setting a property on the result object.
--- Cause: com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception:
** BEGIN NESTED EXCEPTION **
java.io.EOFException
STACKTRACE:
java.io.EOFException
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1903)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2402)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2860)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
at com.mysql.jdbc.ResultSet.next(ResultSet.java:6106)
at org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:168)
at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:592)
at com.ibatis.common.jdbc.logging.ResultSetLogProxy.invoke(ResultSetLogProxy.java:47)
at $Proxy10.next(Unknown Source)
at com.ibatis.sqlmap.engine.execution.SqlExecutor.handleResults(SqlExecutor.java:380)
at com.ibatis.sqlmap.engine.execution.SqlExecutor.handleMultipleResults(SqlExecutor.java:301)
at com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:190)
at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteQuery(GeneralStatement.java:205)
at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithCallback(GeneralStatement.java:173)
at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithRowHandler(GeneralStatement.java:133)
at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryWithRowHandler(SqlMapExecutorDelegate.java:649)
at com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryWithRowHandler(SqlMapSessionImpl.java:156)
at com.ibatis.sqlmap.engine.impl.SqlMapClientImpl.queryWithRowHandler(SqlMapClientImpl.java:133)
at org.springframework.orm.ibatis.SqlMapClientTemplate$5.doInSqlMapClient(SqlMapClientTemplate.java:267)
at org.springframework.orm.ibatis.SqlMapClientTemplate.execute(SqlMapClientTemplate.java:165)
at org.springframework.orm.ibatis.SqlMapClientTemplate.queryWithRowHandler(SqlMapClientTemplate.java:265)
at org.myCron.doBatch(MyCron.java:57)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:592)
at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:248)
at org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:165)
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:66)
at org.quartz.core.JobRunShell.run(JobRunShell.java:191)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:516)
** END NESTED EXCEPTION **
The stack trace is very vague. The only hints that I see are 'the error happened while setting a property on the result object'. There are only two properties on the result object: a String and an Integer. Both of them permit null values, but my select statements indicate that neither of them have any null values. They both have a proper gettter/setter (which makes sense since the process runs for a while successfully before dying). Every time that the cron runs, it dies at a random point (so it isn't stuck on a particular row).
Note - The method 'doBatch' does exist since that is the method that starts the cron process. If it couldn't find doBatch, it couldn't successfully process the first thousand rows.
I've also tried runnning the job outside of quartz and it also fails there as well. We tried increasing our MySQL net_read_timeout, net_write_timeout, and delayed_insert_timeout but none of these settings helped with the problem. I also tried setting my log4j setting to DEBUG and I did not get any helpful info.
Any other ideas about what I could try?
Sounds like MySQL closed the connection for some reason. Check the MySQL log see if anything shows up. Turn on various logging options for MySQL if necessary.
Also, start printing debug data (including timestamp) from your app - just print everything, then see what the last action was - perhaps you have some rarely triggered conditions in your code that has a bug.
I.e. every single time you talk to MySQL log it before AND after.