We Keep Coding

How to stop log4j2 load class in exception stack?

Log4j2 will load class when print exception stack trace, but it will use the incorrect class loader(org.eclipse.jetty.webapp.WebAppClassLoader, extends URLClassLoader, load class synchronized) to load the class generated by aop (such as sun.reflect.GeneratedMethodAccessor<\digit>, only can load by DelegateClassLoader), and it can never load successfully.
Here is the exception stack trace in my service:
2018-11-30 14:43:09.425 - - [ERROR] com.myservice.mobile.message.group.deal.RPCGroupDealService-2-thread-30 ThriftServerInvoker #XMDT#{__ptTest__=true __traceId__=4869019290298993826}#XMDT# mtthrift server invoker Exception:null, cause:
Error querying database. Cause: java.sql.SQLException: com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 50, active 20, maxActive 20
at com.alibaba.druid.pool.DruidDataSource.getConnectionInternal(DruidDataSource.java:1155)
at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:977)
at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:957)
at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:947)
at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:102)
at com.mycompany.zebra.single.jdbc.SingleDataSource.getConnectionOrigin(SingleDataSource.java:212)
at com.mycompany.zebra.single.jdbc.SingleDataSource.access$000(SingleDataSource.java:31)
at com.mycompany.zebra.single.jdbc.SingleDataSource$2.getSingleConnection(SingleDataSource.java:197)
at com.mycompany.zebra.filter.wall.WallFilter.getSingleConnection(WallFilter.java:120)
at com.mycompany.zebra.single.jdbc.SingleDataSource$2.getSingleConnection(SingleDataSource.java:195)
at com.mycompany.zebra.filter.DefaultJdbcFilter.getSingleConnection(DefaultJdbcFilter.java:75)
at com.mycompany.zebra.single.jdbc.SingleDataSource$2.getSingleConnection(SingleDataSource.java:195)
at com.mycompany.zebra.filter.DefaultJdbcFilter.getSingleConnection(DefaultJdbcFilter.java:75)
at com.mycompany.zebra.single.jdbc.SingleDataSource$2.getSingleConnection(SingleDataSource.java:195)
at com.mycompany.zebra.filter.DefaultJdbcFilter.getSingleConnection(DefaultJdbcFilter.java:75)
at com.mycompany.zebra.single.jdbc.SingleDataSource$2.getSingleConnection(SingleDataSource.java:195)
at com.mycompany.zebra.monitor.filter.CatFilter.getSingleConnection(CatFilter.java:114)
at com.mycompany.zebra.single.jdbc.SingleDataSource$2.getSingleConnection(SingleDataSource.java:195)
at com.mycompany.zebra.single.jdbc.SingleDataSource.getConnection(SingleDataSource.java:201)
at com.mycompany.zebra.single.jdbc.SingleDataSource.getConnection(SingleDataSource.java:184)
at com.mycompany.zebra.group.datasources.LoadBalancedDataSource.getConnection(LoadBalancedDataSource.java:114)
at com.mycompany.zebra.group.datasources.LoadBalancedDataSource.getConnection(LoadBalancedDataSource.java:80)
at com.mycompany.zebra.group.jdbc.GroupConnection.getReadConnection(GroupConnection.java:176)
at com.mycompany.zebra.group.jdbc.GroupConnection.getRealConnection(GroupConnection.java:192)
at com.mycompany.zebra.group.jdbc.GroupPreparedStatement.executeQuery(GroupPreparedStatement.java:140)
at com.mycompany.zebra.group.jdbc.GroupPreparedStatement.execute(GroupPreparedStatement.java:97)
at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:39)
at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:55)
at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:41)
at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:243)
at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:117)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:79)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:73)
at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:350)
at com.sun.proxy.$Proxy44.selectList(Unknown Source)
at org.mybatis.spring.SqlSessionTemplate.selectList(SqlSessionTemplate.java:190)
at com.myservice.mobile.group.biz.domain.dao.DealPoiDao.selectByPoiidsAndDpStatus(DealPoiDao.java:89)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at com.myservice.deal.aspects.aj.MonitorAspect.wrap(MonitorAspect.java:72)
at com.myservice.deal.aspects.aj.MonitorAspect.wrapMethod(MonitorAspect.java:41)
at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at com.myservice.mobile.group.biz.common.aspect.DaoFlowControlAspect.doFlowControl(DaoFlowControlAspect.java:25)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy45.selectByPoiidsAndDpStatus(Unknown Source)
at com.myservice.mobile.group.biz.service.PoiDealService.getPoiDealFromDB(PoiDealService.java:118)
at com.myservice.mobile.group.biz.service.PoiDealService.getPoiDealMap(PoiDealService.java:91)
at com.myservice.mobile.group.octo.DealThriftProcessor.listDidByEqualsPoiidsAndDpStatus(DealThriftProcessor.java:283)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at com.myservice.deal.aspects.aj.MonitorAspect.wrap(MonitorAspect.java:72)
at com.myservice.deal.aspects.aj.MonitorAspect.wrapMethod(MonitorAspect.java:41)
at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy48.listDidByEqualsPoiidsAndDpStatus(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.myservice.mobile.mtthrift.proxy.ThriftServerInvoker.invoke(ThriftServerInvoker.java:187)
at com.sun.proxy.$Proxy48.listDidByEqualsPoiidsAndDpStatus(Unknown Source)
at com.myservice.mobile.message.group.deal.RPCGroupDealService$Processor$listDidByEqualsPoiidsAndDpStatus.getResult(RPCGroupDealService.java:726)
at com.myservice.mobile.message.group.deal.RPCGroupDealService$Processor$listDidByEqualsPoiidsAndDpStatus.getResult(RPCGroupDealService.java:715)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at com.myservice.mobile.mtthrift.proxy.ThriftServerPublisher$MtTProcessor.process(ThriftServerPublisher.java:416)
at com.myservice.mobile.mtthrift.server.netty.DefaultServerHandler.handleRequest(DefaultServerHandler.java:138)
at com.myservice.mobile.mtthrift.server.netty.DefaultServerHandler$1.run(DefaultServerHandler.java:106)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
When massive exceptions occurred suddenly, worker thread will blocked on the log4j2 class loader load the GeneratedMethodAccessor<\digit>.
Here is my worker thread stack locked on the method loadClass:

This issue is caused by the class loading that happens while formatting "extended stack traces", which appears as ThrowableProxy.toExtendedStackTrace in your example.
You can disable extended stack traces by adding alwaysWriteExceptions="false" to your PatternLayout (and then adding your own exception formatter, such as %ex, instead), e.g.:
<PatternLayout pattern="%d{ISO8601}{GMT} %m%n%ex" alwaysWriteExceptions="false"/>
I have verified that this fixes the problem for me.
Note that not only does this problem cause locking, it can also cause poor performance due to the JAR unzipping that happens inside e.g. Tomcat's classloader.

Memory leak because of HibernateSessionImpl in Spring batch

I've written a spring batch application that uses JPA+ Hibernate model for read-write operations.
I'm getting OutOfmemory error due to hibernateSessionImpl object.
I'm using JPA entity manager factory: org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean
as a singleton bean configured in spring.xml
Error:
java.lang.OutOfMemoryError
at java.util.HashMap.createHashedEntry(HashMap.java:650)
at java.util.HashMap.putImpl(HashMap.java:629)
at java.util.HashMap.put(HashMap.java:605)
at org.hibernate.event.def.AbstractFlushingEventListener.postFlush(AbstractFlushingEventListener.java:366)
at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:52)
at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1216)
at org.hibernate.ejb.AbstractEntityManagerImpl.flush(AbstractEntityManagerImpl.java:960)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:600)
at org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:240)
at $Proxy43.flush(Unknown Source)
at com.bunge.portal.core.dao.jpa.RssGenericDaoJpaImpl.create(RssGenericDaoJpaImpl.java:200)
at com.bunge.portal.rss.repository.RssFeedContentCacheRepositoryImpl.create(RssFeedContentCacheRepositoryImpl.java:17)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:600)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy44.create(Unknown Source)
at com.bunge.portal.rss.service.RssFeedServiceImpl.createRssFeedContentCache(RssFeedServiceImpl.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:48)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:600)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
15:22:08,023 ERROR AbstractStep:212 - Encountered an error executing the step
org.springframework.batch.core.JobExecutionException: Partition handler returned an unsuccessful step
at org.springframework.batch.core.partition.support.PartitionStep.doExecute(PartitionStep.java:110)
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:195)
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:135)
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:61)
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:60)
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:144)
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:124)
at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:135)
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:281)
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:120)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:48)
at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:114)
at org.springframework.batch.core.launch.support.CommandLineJobRunner.start(CommandLineJobRunner.java:349)
at org.springframework.batch.core.launch.support.CommandLineJobRunner.main(CommandLineJobRunner.java:574)

The problem was we were fetching 8000 records in one of the queries and using only first record. The batch was running successfully as the number of records in table were less, but as time passed the total records were too many and it was overflowing the heap memory,i changed the query to fetch only one record the latest one and it works fine....
Thanks for you help everyone..!!

DynamicJasper Issue - Cannot run program "javac": java.io.IOException: error=24, Too many open files

When generating the reports using the DynamicJasper API, the below exception is getting thrown. In my logic, around 5000 to 10000 reports will be generating in single execution.
Due to this exception, the report generation exception is totally blocked. Please suggest how to fix this solution without increasing the ulimit option as we can not increase depends on the no.of reports.
net.sf.jasperreports.engine.JRException: Error compiling report java source files : /home/gscsfo/Report_1409558503418_133890.java
com.bnp.scm.services.common.exception.BNPApplicationException: net.sf.jasperreports.engine.JRException: Error compiling report java source files : /home/gscsfo/Report_1409558503418_133890.java
at com.bnp.scm.services.scheduler.common.SystemLogger.profile(SystemLogger.java:54)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
at com.bnp.scm.services.scheduler.events.EmailAlertAttachment$$EnhancerByCGLIB$$3c6f687d.getPDFAttachment(<generated>)
at com.bnp.scm.services.scheduler.events.ScheduleEmailEvent.processEvent(ScheduleEmailEvent.java:1039)
at com.bnp.scm.services.scheduler.events.ScheduleEmailEvent$$FastClassByCGLIB$$84dbc7ef.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)
at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at com.bnp.scm.services.scheduler.common.SystemLogger.profile(SystemLogger.java:54)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
at com.bnp.scm.services.scheduler.events.ScheduleEmailEvent$$EnhancerByCGLIB$$f22c0d43.processEvent(<generated>)
at com.bnp.scm.scheduler.job.handler.EmailNotificationJob.run(EmailNotificationJob.java:52)
at com.bnp.scm.scheduler.job.AbstractJob.execute(AbstractJob.java:79)
at com.bnp.scm.scheduler.job.AbstractJob$$FastClassByCGLIB$$d1c345af.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)
at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at com.bnp.scm.services.scheduler.common.SystemLogger.profile(SystemLogger.java:54)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
at com.bnp.scm.scheduler.job.handler.EmailNotificationJob$$EnhancerByCGLIB$$b1534990.execute(<generated>)
at org.quartz.core.JobRunShell.run(JobRunShell.java:206)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:548)
Caused by: net.sf.dynamicreports.report.exception.DRException: net.sf.jasperreports.engine.JRException: Error compiling report java source files : /home/gscsfo/Report_1409558503418_133890.java
at net.sf.dynamicreports.jasper.builder.JasperReportBuilder.toJasperReport(JasperReportBuilder.java:279)
at net.sf.dynamicreports.jasper.builder.JasperReportBuilder.toJasperPrint(JasperReportBuilder.java:309)
at com.bnp.scm.services.scheduler.events.EmailAlertAttachment.getPDFAttachment(EmailAlertAttachment.java:183)
... 56 more
Caused by: net.sf.jasperreports.engine.JRException: Error compiling report java source files : /home/gscsfo/Report_1409558503418_133890.java
at net.sf.jasperreports.engine.design.JRJavacCompiler.compileClasses(JRJavacCompiler.java:89)
at net.sf.jasperreports.engine.design.JRAbstractClassCompiler.compileUnits(JRAbstractClassCompiler.java:63)
at net.sf.jasperreports.engine.design.JRAbstractCompiler.compileReport(JRAbstractCompiler.java:188)
at net.sf.jasperreports.engine.JasperCompileManager.compileReport(JasperCompileManager.java:212)
at net.sf.dynamicreports.jasper.builder.JasperReportBuilder.toJasperReport(JasperReportBuilder.java:276)
... 58 more
Caused by: java.io.IOException: Cannot run program "javac": java.io.IOException: error=24, Too many open files
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at java.lang.Runtime.exec(Runtime.java:593)
at java.lang.Runtime.exec(Runtime.java:466)
at net.sf.jasperreports.engine.design.JRJavacCompiler.compileClasses(JRJavacCompiler.java:58)
... 62 more
Caused by: java.io.IOException: java.io.IOException: error=24, Too many open files
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
... 65 more

java.lang.NoClassDefFoundError: java/lang/ProcessBuilder$Redirect

I am getting this error when I launch my job. It works on my local machine but it fails if I upload war file to tomcat7 on server. The application is started but it fails at run time.
This is the code,
ProcessBuilder pb = new ProcessBuilder(commands);
File log = new File(this.logFilePath);
if (!log.exists()) {
log.getParentFile().mkdirs();
}
pb.redirectErrorStream(true);
pb.redirectOutput(Redirect.appendTo(log));
Process p = pb.start();
and the stack trace
ERROR: org.springframework.batch.core.step.AbstractStep - Encountered an error executing step eu.createBatchStep in job eu
java.lang.NoClassDefFoundError: java/lang/ProcessBuilder$Redirect
at com.stockopedia.symfony.Symfony.run(Symfony.java:105)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:132)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:120)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at sun.proxy.$Proxy29.run(Unknown Source)
at com.stockopedia.batch.foundation.tasklets.SystemCommandTasklet.execute(SystemCommandTasklet.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:132)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:120)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at sun.proxy.$Proxy22.execute(Unknown Source)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:402)
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:326)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:131)
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:267)
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:77)
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:368)
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215)
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:144)
at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:253)
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:198)
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148)
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:64)
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:67)
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:162)
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:141)
at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:134)
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:304)
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:135)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Initially I though it could be because file does not exist, but I tried creating file manually, but still getting same error.

The errors tells that it can't find class Redirect in class path.
This class is available from Java 1.7 you can check it there, so you can't use it in earlier versions on Java.
I suppose you use Java 1.7 or higher in your development environment, but tomcat is run under Java 1.6 where class Redirect and method redirectOutput(Redirect) does not exist.
So try to upgrade your java on server or use API specific to Java 1.6.

Could not initialize class com.sun.xml.internal.ws.fault.SOAPFaultBuilder

I am using Jaxb-impl-2.1.3.jar in the maven library.. every thing looks fine in pom entry.. if anyone come across with this error pleas give me an idea..
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:42)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at
Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.sun.xml.internal.ws.fault.SOAPFaultBuilder
at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:107)
at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:78)
at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(SEIStub.java:107)

You need to add following jars into your application's classpath to make it work:
jaxws-api.jar
jaxws-rt.jar
jaxws-tools.jar