We are running SpringMVC with Hibernate on a clustered Weblogic environment. Our application worked well, until we run into database space problems. Due to that, our DB provider extended the database storage.
After that change the Weblogic system status is complaining about the thread pool "Thread pool has stucked threads".
The stack-trace in the log file looks as follows:
<Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '12' for queue: 'weblogic.ke
rnel.Default (self-tuning)' has been busy for "641" seconds working on the request "weblogic.servlet.internal.ServletReq
uestImpl#17782ff3[
GET /default/default/applicationPreloader/load.do?_=1331887527968 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:10.0.2) Gecko/20100101 Firefox/10.0.2
Accept: */*
Accept-Language: en-gb,en;q=0.5
Accept-Encoding: gzip, deflate
x-requested-with: XMLHttpRequest
Referer: http://novintegweb01.eu.acme.net:9136/default/default/applicationPreloader.do
Cookie: JSESSIONID1=NhBdPv9Hx8Wj5lnr5vnGhLSbR2GGM1DPhqT2YGzWCGFvhN3WJWVH!700169023!1391220923
Connection: Keep-Alive
Proxy-Client-IP: 192.168.1.1
X-Forwarded-For: 192.168.1.1
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Request-ClusterInfo: true
x-weblogic-cluster-hash: G4kGT1kfuDBbtj5w//SYNt6hlVo
]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
oracle.net.ns.Packet.receive(Packet.java:293)
oracle.net.ns.DataPacket.receive(DataPacket.java:104)
oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)
oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
oracle.net.ns.NetInputStream.read(NetInputStream.java:102)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1136)
oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1113)
oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:288)
oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523)
oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1009)
oracle.jdbc.driver.OraclePreparedStatement.executeForRowsWithTimeout(OraclePreparedStatement.java:10117)
oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:10221)
oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:216)
weblogic.jdbc.wrapper.PreparedStatement.executeBatch(PreparedStatement.java:210)
org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:48)
org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:247)
org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:93)
org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:88)
org.hibernate.jdbc.AbstractBatcher.prepareBatchStatement(AbstractBatcher.java:223)
org.hibernate.persister.collection.AbstractCollectionPersister.recreate(AbstractCollectionPersister.java:1119)
org.hibernate.action.CollectionRecreateAction.execute(CollectionRecreateAction.java:39)
org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279)
org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:263)
org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:171)
org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298)
org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27)
org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1001)
org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:339)
org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:106)
org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:656)
org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:754)
org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:120)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
com.acme.superproject.dao.CertificationDaoImpl$$EnhancerByCGLIB$$bf3135a5.getById(<generated>)
com.acme.superproject.integration.UserProfileSabaServiceImpl.determineTargetDateFromWs(UserProfileSabaServiceImpl.java:436)
com.acme.superproject.integration.UserProfileSabaServiceImpl.determineTargetDate(UserProfileSabaServiceImpl.java:401)
com.acme.superproject.integration.UserProfileSabaServiceImpl.createCertificationRelationObject(UserProfileSabaServiceImpl.java:373)
com.acme.superproject.integration.UserProfileSabaServiceImpl.getRequiredLearning(UserProfileSabaServiceImpl.java:334)
com.acme.superproject.integration.UserProfileSabaServiceImpl.extractUserProfile(UserProfileSabaServiceImpl.java:203)
com.acme.superproject.integration.UserProfileSabaServiceImpl.getUserProfile(UserProfileSabaServiceImpl.java:73)
com.acme.superproject.integration.UserProfileSabaServiceImpl.getUserProfile(UserProfileSabaServiceImpl.java:114)
com.acme.superproject.integration.UserProfileSabaServiceImpl$$FastClassByCGLIB$$e48c61c9.invoke(<generated>)
net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
com.acme.superproject.techarch.TracingAspect.trace(TracingAspect.java:31)
sun.reflect.GeneratedMethodAccessor5382.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
com.acme.superproject.integration.UserProfileSabaServiceImpl$$EnhancerByCGLIB$$5d7b85df.getUserProfile(<generated>)
com.acme.superproject.service.UserProfileServiceImpl.getUserProfile(UserProfileServiceImpl.java:26)
com.acme.superproject.service.UserProfileServiceImpl$$FastClassByCGLIB$$70cfe815.invoke(<generated>)
net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
com.acme.superproject.techarch.TracingAspect.trace(TracingAspect.java:31)
sun.reflect.GeneratedMethodAccessor5382.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
com.acme.superproject.service.UserProfileServiceImpl$$EnhancerByCGLIB$$5bcb212b.getUserProfile(<generated>)
com.acme.superproject.web.session.SessionDataProvider.fetchUserProfile(SessionDataProvider.java:97)
com.acme.superproject.web.session.SessionDataProvider.getUserProfile(SessionDataProvider.java:78)
com.acme.superproject.web.session.SessionDataProvider.fetchUserRegistrationCollection(SessionDataProvider.java:155)
com.acme.superproject.web.session.SessionDataProvider.getUserRegistrationCollection(SessionDataProvider.java:141)
com.acme.superproject.web.ApplicationPreloaderController.loadAction(ApplicationPreloaderController.java:84)
sun.reflect.GeneratedMethodAccessor6506.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176)
org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:426)
org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:414)
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:368)
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:97)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:89)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:79)
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:169)
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2277)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2183)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1454)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
weblogic.work.ExecuteThread.run(ExecuteThread.java:176)
>
We already have restarted Weblogic, the database server and made a new deployment. The problem is always the same: When we try to access the login page of the application it hangs up - as far as I can see while trying to perform a DB transaction.
In addition to that, when I try to connect to our database with "Oracle SQL Developer" we also get the error "ORA-00257: archiver error. Connect internal only, until freed."
Unfortunately we have no SYSDBA rights (because managed by an external provider).
My question are:
Has anyone of you had a similar problem before?
Do you think the root cause of such problems is more on application side, or more on DB side?
How can I at least get rid of the stucked threads?
Chances are that the database storage is full, or that the archiver cannot move or create new archive logs. This is almost definitely an issue at the database level and not the app server level. Since you are getting this message that means you already have a connection to the database, so it is not sitting and waiting for a connection.
Stuck Threads in WebLogic don't always need to be 'fixed' and are not always a problem. The purpose of the stuck thread mechanism is to notify you when a thread is taking an abnormally long time to do something (like access the database). The stuck thread notification is just that - a notification that something might not be right, and in this case it is serving that purpose.
If you have threads that routinely take 20 minutes to do something but want to be notified if it takes longer than 25 minutes, then have to increase the Stuck Thread Limit to 25 minutes and get the notification at that point then you must put this limit from 600 seconds to 1500 seconds (25*60).
You can edit stuck thread limit in weblogic admin base_domain > Environment > Servers > Admin Server > Configuration/Tuning.
refer below link :
https://docs.oracle.com/cd/E19316-01/820-3745/ghudn/index.html
This seems to be an issue with the database. The threads are stuck on the DB. The performance of your seems to have degraded. I would suggest to refer the case to your DBA for futher investigation. Also, try recreating your indexes on the tables being accessed.
We have faced this issue when the Oracle DB Server was on a VirtualBox, in a developer environment. Pointing out the issue was bit difficult. We noticed this when the ThreadPool did not have any more threads to handle the requests and the requests were rejected.
Related
Dears,
we're running into a 'too many open files' issue.
We've made some progress but the issue is still there...
We use cxf on weblogic on aix.
relevant cxf configuration:
<http-conf:conduit name="*.http-conduit">
<http-conf:client Connection="close" />
</http-conf:conduit>
When executing this code:
for(int i=0;i<1300;i++){
response = callService(request).call();
}
(we don't intend to use this code in production but it's to troubleshoot the issue)
-> the first run succeeds;
-> the second run exposes the problem
output of:
netstat| wc -l
first showed that too many sockets remained open (hence the cxf config change)
now the open sockets remain stable +- 400 lines where before the config change >1000
but...
lsof 2> /dev/null|grep TCP|wc -l
exposes the issue we run into for the present moment
the number of open sockets is stable (so sockets get closed) but the number of open files doesn't...
from 319 just after managed server boot until +-1688 when following exception is raised:
javax.xml.ws.soap.SOAPFaultException: NullPointerException invoking https://tst-cjcsr.just.fgov.be/cjcs-cg-ws/1.0-rc2/: null
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:158)
at com.sun.proxy.$Proxy302.requestCriminalRecord(Unknown Source)
at be.fgov.just.cjr.application.dossier.DossierBean$1.call(DossierBean.java:226)
at be.fgov.just.cjr.application.dossier.DossierBean$1.call(DossierBean.java:222)
at be.fgov.just.cjr.application.dossier.DossierBean.callService(DossierBean.java:153)
at be.fgov.just.cjr.application.dossier.DossierBean.search(DossierBean.java:99)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at org.jboss.el.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:329)
at org.jboss.el.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:342)
at org.jboss.el.parser.AstPropertySuffix.invoke(AstPropertySuffix.java:58)
at org.jboss.el.parser.AstValue.invoke(AstValue.java:96)
at org.jboss.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:276)
at com.sun.faces.facelets.el.TagMethodExpression.invoke(TagMethodExpression.java:105)
at javax.faces.component.MethodBindingMethodExpressionAdapter.invoke(MethodBindingMethodExpressionAdapter.java:88)
at com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:102)
at javax.faces.component.UICommand.broadcast(UICommand.java:315)
at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:794)
at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:1259)
at com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:81)
at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101)
at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:118)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:409)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at be.fgov.just.cjr.application.authentication.UserbeanEnforceFilter.doFilter(UserbeanEnforceFilter.java:64)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at be.fgov.just.cjr.application.authentication.openam.OpenAmPingFilter.doFilter(OpenAmPingFilter.java:88)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at be.fgov.just.cjr.application.authentication.openam.OpenAmReverifyFilter.doFilter(OpenAmReverifyFilter.java:94)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at be.fgov.just.cjr.application.authentication.openam.OpenAmSessionFilter.doFilter(OpenAmSessionFilter.java:166)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:259)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3748)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3714)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2283)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2182)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1491)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused by: java.lang.NullPointerException: NullPointerException invoking https://tst-cjcsr.just.fgov.be/cjcs-cg-ws/1.0-rc2/: null
at sun.reflect.GeneratedConstructorAccessor108.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:39)
at java.lang.reflect.Constructor.newInstance(Constructor.java:527)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1346)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1335)
at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:632)
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272)
at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:572)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:481)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:382)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:335)
at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:96)
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:136)
... 56 more
Caused by: java.lang.NullPointerException
at weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:459)
at weblogic.net.http.SOAPHttpsURLConnection.getInputStream(SOAPHttpsURLConnection.java:37)
at weblogic.net.http.HttpURLConnection.getResponseCode(HttpURLConnection.java:1038)
at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.getResponseCode(URLConnectionHTTPConduit.java:266)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.doProcessResponseCode(HTTPConduit.java:1550)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1579)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1520)
at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1317)
... 66 more
if more info is to be provided, just ask,
any aid in troubleshooting this issue is greatly appreciated,
restarting the managed server 'solves' the problem but this is no option in production...
TIME_WAIT is set to 1 (15 seconds) so no solution there...
S.
edit:
one partial solution is described at DZone; indeed:
ulimit -S -n
outputs 2000 and
procfiles 15728790 | grep rlimit
outputs
Current rlimit: 2000 file descriptors
We're implementing their solution now (add ulimit -S -n 65536 to the end of resetFd()) but the root cause (socket leak) is still not solved :-/
Any help still greatly appreciated...
S.
I'm trying to debug a critical issue where a JSP application is seeing request timeouts when making call to a remote web service.
I should clarify the timeout doesn't happen at all time, for smaller queries the web service does return response.
I have tried increasing the timeout value in WebLogic server > Protocols > HTTP > Post timeout and Duration. I have also tried to set JAX-WS properties in my spring.xml as follows:
<util:map id="jaxwsCustomProperties">
<entry key="com.sun.xml.ws.request.timeout">
<value type="java.lang.Integer">650000</value>
</entry>
<entry key="com.sun.xml.ws.connect.timeout">
<value type="java.lang.Integer">650000</value>
</entry>
</util:map>
Console output:
org.springframework.remoting.jaxws.JaxWsSoapFaultException: Request timeout.
Read timed out; nested exception is javax.xml.ws.soap.SOAPFaultException: Request timeout.
Read timed out
at org.springframework.remoting.jaxws.JaxWsPortClientInterceptor.doInvoke(JaxWsPortClientInterceptor.java:395)
at org.springframework.remoting.jaxws.JaxWsPortClientInterceptor.invoke(JaxWsPortClientInterceptor.java:379)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy72.getDeviceList(Unknown Source)
at com.company.msr.web.actionbean.DeviceListProcessor.getDeviceList(DeviceListProcessor.java:223)
at com.company.msr.web.actionbean.MsrMainActionBean.listToJson(MsrMainActionBean.java:287)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at net.sourceforge.stripes.controller.DispatcherHelper$6.intercept(DispatcherHelper.java:467)
at net.sourceforge.stripes.controller.ExecutionContext.proceed(ExecutionContext.java:158)
at net.sourceforge.stripes.controller.BeforeAfterMethodInterceptor.intercept(BeforeAfterMethodInterceptor.java:113)
at net.sourceforge.stripes.controller.ExecutionContext.proceed(ExecutionContext.java:155)
at net.sourceforge.stripes.controller.ExecutionContext.wrap(ExecutionContext.java:74)
at net.sourceforge.stripes.controller.DispatcherHelper.invokeEventHandler(DispatcherHelper.java:465)
at net.sourceforge.stripes.controller.DispatcherServlet.invokeEventHandler(DispatcherServlet.java:278)
at net.sourceforge.stripes.controller.DispatcherServlet.service(DispatcherServlet.java:160)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
at net.sourceforge.stripes.controller.StripesFilter.doFilter(StripesFilter.java:247)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
at com.company.msr.web.actionbean.UserAccessFilter.doFilter(UserAccessFilter.java:53)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
at com.company.msr.web.actionbean.filter.AjaxFilter.doFilter(AjaxFilter.java:29)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3496)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(Unknown Source)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
Caused by: javax.xml.ws.soap.SOAPFaultException: Request timeout.
Read timed out
at com.sun.xml.ws.fault.SOAP11Fault.getProtocolException(SOAP11Fault.java:188)
at com.sun.xml.ws.fault.SOAPFaultBuilder.createException(SOAPFaultBuilder.java:108)
at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:119)
at com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:89)
at com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:118)
at $Proxy108.getDeviceList(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.remoting.jaxws.JaxWsPortClientInterceptor.doInvoke(JaxWsPortClientInterceptor.java:416)
at org.springframework.remoting.jaxws.JaxWsPortClientInterceptor.doInvoke(JaxWsPortClientInterceptor.java:392)
I have been unsuccessful with all attempts. Any ideas on how to resolve this are deeply appreciated.
Thanks.
Try setting the following for your webserver (Server Start tab or put it into your startWebLogic.sh script):
Specifies, in seconds, how long a client application that is attempting to invoke a Web service waits to make a connection. After the specified time elapses, if a connection hasn't been made, the attempt times out.
-Dweblogic.http.client.defaultConnectTimeout=650000
Specifies, in seconds, how long a client application waits for a response from a Web service it is invoking. After the specified time elapses, if a response hasn't arrived, the client times out.
-Dweblogic.http.client.defaultReadTimeout=650000
References here:
http://docs.oracle.com/cd/E14571_01/web.1111/e13760/client.htm#i227120
http://docs.oracle.com/cd/E29597_01/fusionapps.1111/e14496/compliance.htm
We're getting the below exception when calling JasperFillManager.fillReport(). The confusing aspect is that it happens in our "production" environment, but works fine in our TEST environment.
Apparently this issue happens during initialization of a static property, and we think it may be related to the multi-threading Jasper does (ThreadLocal-stuff).
We've tried the java.awt.headless=true JVM setting as some suggestions have indicated, but there was no change. We're really in need of suggestions for troubleshooting, or clues as to what might cause this in one environment but not the other. As far as we can tell, they're the same, but obviously something is different and we're out of ideas for things to check.
So the question really is...
how can we troubleshoot an issue like this?
Jasper Reports 4.7 (Also occurs with 3.7)
Java 1.6, Flex (GraniteDS on server)/Spring/Hibernate
(we're not using EJB, and we deploy via a simple war file)
FunctionName: com.ibm.ws.webcontainer.servlet.ServletWrapper.service
SourceId: com.ibm.ws.webcontainer.servlet
Category: SEVERE
ExtendedMessage: SRVE0068E: Uncaught exception created in one of the service methods of the servlet ReportServlet in application B
AO. Exception created : java.lang.ExceptionInInitializerError
at java.lang.J9VMInternals.initialize(J9VMInternals.java:222)
at net.sf.jasperreports.engine.fill.JRBaseFiller.<init>(JRBaseFiller.java:121)
at net.sf.jasperreports.engine.fill.JRVerticalFiller.<init>(JRVerticalFiller.java:88)
at net.sf.jasperreports.engine.fill.JRVerticalFiller.<init>(JRVerticalFiller.java:103)
at net.sf.jasperreports.engine.fill.JRVerticalFiller.<init>(JRVerticalFiller.java:61)
at net.sf.jasperreports.engine.fill.JRFiller.createFiller(JRFiller.java:153)
at net.sf.jasperreports.engine.fill.JRFiller.fill(JRFiller.java:55)
at net.sf.jasperreports.engine.JasperFillManager.fill(JasperFillManager.java:446)
at net.sf.jasperreports.engine.JasperFillManager.fill(JasperFillManager.java:407)
at net.sf.jasperreports.engine.JasperFillManager.fillReport(JasperFillManager.java:824)
at gov.bao.util.JasperWork.execute(JasperWork.java:36)
at org.hibernate.impl.SessionImpl.doWork(SessionImpl.java:1997)
at gov.bao.dao.ReportDao.generateJasperReport(ReportDao.java:246)
at gov.bao.service.ReportService.getTracerReport(ReportService.java:258)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy65.getTracerReport(Unknown Source)
at gov.bao.servlet.ReportServlet.doGet(ReportServlet.java:110)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:718)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1657)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1597)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:131)
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:359)
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:275)
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:344)
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:272)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:81)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:77)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:908)
at com.ib***BUFFER OVERFLOW***Y
Trace: 2012/09/06 08:43:55.556 01 t=8C9CF0 c=UNK key=P8 (00000011)
Description: Log Java Message
Message: SRVE0068E: Uncaught exception created in one of the service methods of the servlet ReportServlet in application BAO. Exce
ption created : java.lang.ExceptionInInitializerError
at java.lang.J9VMInternals.initialize(J9VMInternals.java:222)
at net.sf.jasperreports.engine.fill.JRBaseFiller.<init>(JRBaseFiller.java:121)
at net.sf.jasperreports.engine.fill.JRVerticalFiller.<init>(JRVerticalFiller.java:88)
at net.sf.jasperreports.engine.fill.JRVerticalFiller.<init>(JRVerticalFiller.java:103)
at net.sf.jasperreports.engine.fill.JRVerticalFiller.<init>(JRVerticalFiller.java:61)
at net.sf.jasperreports.engine.fill.JRFiller.createFiller(JRFiller.java:153)
at net.sf.jasperreports.engine.fill.JRFiller.fill(JRFiller.java:55)
at net.sf.jasperreports.engine.JasperFillManager.fill(JasperFillManager.java:446)
at net.sf.jasperreports.engine.JasperFillManager.fill(JasperFillManager.java:407)
at net.sf.jasperreports.engine.JasperFillManager.fillReport(JasperFillManager.java:824)
at gov.bao.util.JasperWork.execute(JasperWork.java:36)
at org.hibernate.impl.SessionImpl.doWork(SessionImpl.java:1997)
at gov.bao.dao.ReportDao.generateJasperReport(ReportDao.java:246)
at gov.bao.service.ReportService.getTracerReport(ReportService.java:258)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy65.getTracerReport(Unknown Source)
at gov.bao.servlet.ReportServlet.doGet(ReportServlet.java:110)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:718)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1657)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1597)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:131)
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:359)
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:275)
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:344)
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:272)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:81)
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:188)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:116)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:77)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:908)
at com.ib***BUFFER OVERFLOW***Y com.ibm.ws.webcontainer.servlet.ServletWrapper.service
To troubleshoot the issue, you need to get the original exception thrown by the static initializer. Normally that exception is linked to the ExceptionInInitializerError, the first time an attempt is made to load the class that contains the static initializer.
Therefore you need to do two things:
Ensure that you are able to get the entire stack trace. The stack trace you have posted seems to be truncated (see the ***BUFFER OVERFLOW*** marker).
Get the stack trace of the exception thrown when the first attempt to initialize the class is made. If necessary, restart the application or server.
The original answer of post's author (#Mifune):
Solved by adding two libs:
jasperreports-javaflow.jar
Apache commons-javaflow.jar
I tried to run a stress test using a JMeter test on an API which producing messages toward a RabbitMQ queue. The ressource call a service which open a connection to RabbitMQ, publish a message on a queue via a binded exchange and then close the connection.
When I run a test with a single thread calling the API to produce 45000 messages, It works perfectly.
A problem appears when I try to run a test with five thread producing 9000 messages : a java.net.NoRouteToHostException is thrown (see the trace below).
Running this test I checked connections number on my server and I found more than 30000 TIME_WAIT connections, I decided to reduce the time_wait time out in my sysctl.conf to 20 seconds instead of 120 seconds.
But nothing changed. There are less connections in TIME_WAIT state (around 5000 now) but exception is still thrown.
I tried to increase file descriptors limit in RabbitMQ for the same result.
SEVERE: Mapped exception to response: 500 (Internal Server Error)
fr.company.rest.InternalServerError: java.net.NoRouteToHostException: Cannot assign requested address
at fr.company.webResources.EmailResource.create(EmailResource.java:66)
at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:240)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:164)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:462)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:164)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:563)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:403)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:301)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:162)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:140)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:309)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.net.NoRouteToHostException: Cannot assign requested address
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
at java.net.Socket.connect(Socket.java:546)
at com.rabbitmq.client.ConnectionFactory.createFrameHandler(ConnectionFactory.java:362)
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:400)
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:423)
at fr.company.services.MessageService.init(MessageService.java:41)
at fr.company.webResources.EmailResource.create(EmailResource.java:62)
... 41 more
If I right understand you, i see two ways. First - you must close connection when don't need in it.
connect();
some_action();
disconnect();
Second - connect in setup; send queries, parse responses and others in runTest; disconnect in tearDown.
If you load MQ, i think second way is yours.
I am working on tomcat 7.1, after 4,5 deployment I am getting following error:
SEVERE: Catalina.stop: java.net.SocketException: No buffer space available (maximum connections reached?): connect
Then I need to restart my system, because my internet also stopped working when this error happens. How is this problem caused and how can I solve it?
I just found the issue again, pasting the log. please help me.
INFO [http-apr-8080-exec-14] (OrderDAO_MySqlImpl.java:287) - INSIDE THE ORDER DAO IMPL: SUBCAT IS: Cpmuter Services
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
Last packet sent to the server was 0 ms ago.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:353)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1074)
at com.mysql.jdbc.Connection.createNewIO(Connection.java:2001)
at com.mysql.jdbc.Connection.<init>(Connection.java:716)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:353)
at com.mysql.jdbc.Connection.getInstance(Connection.java:295)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266)
at java.sql.DriverManager.getConnection(DriverManager.java:582)
at java.sql.DriverManager.getConnection(DriverManager.java:185)
at bseller.Order.dao.mysql.OrderDAO_MySqlImpl.getConnection(OrderDAO_MySqlImpl.java:264)
at bseller.Order.dao.mysql.OrderDAO_MySqlImpl.getAllOrder(OrderDAO_MySqlImpl.java:294)
at bseller.postadd.utils.OrderControlService.getAllOrders(OrderControlService.java:33)
at bseller.postadd.utils.OrderManager.PopulateOrderDetail(OrderManager.java:50)
at org.apache.jsp.SubCategoryDeails_jsp._jspService(SubCategoryDeails_jsp.java:182)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:433)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:389)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:333)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:304)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:684)
at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:593)
at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:530)
at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:927)
at org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:652)
at org.apache.struts.tiles.TilesUtilImpl.doInclude(TilesUtilImpl.java:137)
at org.apache.struts.tiles.TilesUtil.doInclude(TilesUtil.java:177)
at org.apache.struts.taglib.tiles.InsertTag.doInclude(InsertTag.java:756)
at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag(InsertTag.java:881)
at org.apache.struts.taglib.tiles.InsertTag.doEndTag(InsertTag.java:473)
at org.apache.jsp.Layout_jsp._jspx_meth_tiles_005finsert_005f1(Layout_jsp.java:244)
at org.apache.jsp.Layout_jsp._jspService(Layout_jsp.java:137)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:433)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:389)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:333)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:304)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:684)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:471)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:402)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:329)
at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1069)
at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:274)
at org.apache.struts.tiles.TilesRequestProcessor.processTilesDefinition(TilesRequestProcessor.java:254)
at org.apache.struts.tiles.TilesRequestProcessor.processForwardConfig(TilesRequestProcessor.java:309)
at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:279)
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:507)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:304)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:462)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:164)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:851)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:405)
at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:257)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515)
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1764)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.SocketException: No buffer space available (maximum connections reached?): connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:519)
at java.net.Socket.connect(Socket.java:469)
at java.net.Socket.<init>(Socket.java:366)
at java.net.Socket.<init>(Socket.java:209)
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:168)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:262)
at com.mysql.jdbc.Connection.createNewIO(Connection.java:1926)
... 72 more
Sep 11, 2011 2:46:24 PM org.apache.catalina.core.ApplicationDispatcher invoke
SEVERE: Servlet.service() for servlet jsp threw exception
java.net.SocketException: No buffer space available (maximum connections reached?): connect
In plain english, this is the operating system telling you it can't create an additional TCP socket due to resource congestion. Most operating systems have a hard limit on how many connections can be open concurrently. It is possible you have hit that limit.
To avoid issues like that, database connections are usually pooled, i.e. rather than dispose of a no-longer needed connection, the connection is kept around, to be reused for future requests to the database. There are a lot of connection pooling libraries. Pooling capability is also provided by most application servers, if you obtain connections from a DataSource mapped in JNDI.
Another possible cause is that you are not closing all connections after you have used them.
Does your application use directly socket connections or databases? Maybe your application creates many connections and does not close them.
You may try to enter in cmd
netstat -a
and analyze what foreign addres occurs most times and do you know it - check is it database address or sth else familiar to you.