I'm having trouble in mule service. Daily, the service crashes, about 4 times a day, making it necessary to redeploy my service.
Here's my stack trace.
ERROR 2016-02-04 10:30:38,063 [[app_service].NoSessionConnector.receiver.03] org.mule.exception.DefaultMessagingExceptionStrategy:
********************************************************************************
Message : Timeout for connection (java.net.SocketException). Message payload is of type: HttpResponse
Code : MULE_ERROR--2
--------------------------------------------------------------------------------
Exception stack is:
1. Timeout for connection (java.net.SocketException)
java.net.SocketOutputStream:-2 (null)
2. Timeout for connection (java.net.SocketException). Message payload is of type: HttpResponse (org.mule.execution.ResponseDispatchExcepti
on)
org.mule.transport.http.HttpMessageProcessTemplate:141 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/execution/ResponseDispatc
hException.html)
--------------------------------------------------------------------------------
Root Exception stack trace:
java.net.SocketException: Tempo esgotado para conexão
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
+ 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************
I've tried changing the number of threads in maxThreadsActive and maxBufferSize, but is still occurring the same error
Seems like the tcp connection has not been usable for a long time.
Use the the Socket.setKeepAlive() or apply the heart beat mechanism.
Related
Hope you are doing well. Basically I am calling Java API from .NET .
Most of the time it is working absolutely fine but the thing it sometimes once or twice in a day it gives me Socket exception. I am putting down the calling code and the exception stack trace. Any suggestion to fix this once and for all will be helpful.
Thanks in advance.!!
Code :
var client = new HttpClient{Timeout = TimeSpan.FromMinutes(30), BaseAddress = new Uri(AppSettingsConfiguration.GetConfig<string>("appSettings:Generic"))};
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.Add("key", AppSettingsConfiguration.GetConfig<string>("appSettings:JavaAPIKey"));
var response = client.PostAsync(AppSettingsConfiguration.GetConfig<string>("appSettings:Generic"), new StringContent(JsonConvert.SerializeObject(order), Encoding.UTF8, "application/json")).Result;
Stack Trace :
Error reported : One or more errors occurred. Inner Exception
System.Net.Http.HttpRequestException: An error occurred while sending the request. --->
System.Net.WebException: Unable to connect to the remote server --->
System.Net.Sockets.SocketException: A connection attempt failed because the connected party did
not properly respond after a period of time, or established connection failed because connected host has failed to respond {IP} at
System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult asyncResult) at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6,
Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception) --- End of inner exception stack trace --- at
System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context)
at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar) --- End of inner
exception stack trace --- Stack trace :
I know there are a ton of these posts, but this is a little different. We are using vended code for part of our data processing system, and part of the system sends emails to clients if certain events take place on data insertion or deletion. Recently we have started getting address already in use exceptions. We checked the repository history, and nothing has changed in our code in the last 6 months for this system. We have already tried the typical solutions for this issue including increasing the number of connections allowed to the port with little success. We had a meeting with the vendor, and I asked if anything had changed in their code, and if they would assure that all connections in their code are explicitly closed. They indicated that they are explicitly closing all sockets. However they didn't show us the code so there is no way for us to know if this is true other than taking their word for it. So, the only thing I can think of to do is continue to increase the number of connections to the port until we stop getting bind exceptions. So, what is the industry standard for max number of connections to port 25; is there one? Also if anyone has any other suggestions I would greatly appreciate it? Thanks so much in advance, Robert
20210505112127.716 ERROR m.fiserv.ppx.business.notification.EmailNotifier : MessagingException from notify
javax.mail.MessagingException: Could not connect to SMTP host: SERVER.URL.COM, port: 25;
nested exception is:
java.net.BindException: Address already in use: connect
at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:1545)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:453)
Caused by:
java.net.BindException: Address already in use: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:90)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:380)
20210505131529.950 ERROR erv.ppx.web.controller.AuditReportViewController : Error while generating HTML
net.sf.jasperreports.engine.JRException: Error writing to OutputStream writer : CorpAdminAuditReport
at net.sf.jasperreports.engine.export.JRHtmlExporter.exportReport(JRHtmlExporter.java:496)
at com.fiserv.ppx.web.controller.AuditReportViewController.generateReport(AuditReportViewController.java:184)
Caused by:
com.ibm.wsspi.webcontainer.ClosedConnectionException: OutputStream encountered error during write
at com.ibm.ws.webcontainer.channel.WCCByteBufferOutputStream.write(WCCByteBufferOutputStream.java:188)
at com.ibm.ws.webcontainer.srt.SRTOutputStream.write(SRTOutputStream.java:97)
20210505140706.240 ERROR com.fiserv.ppx.business.db.DBConnectionUtil : Exception in getting for AppServer connection from DataSource.
com.ibm.websphere.ce.cm.ConnectionWaitTimeoutException: J2CA1010E: Connection not available; timed out waiting for 180,005 seconds.
at com.ibm.ws.rsadapter.AdapterUtil.toSQLException(AdapterUtil.java:1680)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:661)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:611)
Caused by:
com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException: J2CA1010E: Connection not available; timed out waiting for 180,005 seconds.
at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1781)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3834)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3082)
20210505140731.341 ERROR com.fiserv.ppx.business.db.DBConnectionUtil : Exception in getting for AppServer connection from DataSource.
com.ibm.websphere.ce.cm.ConnectionWaitTimeoutException: J2CA1010E: Connection not available; timed out waiting for 180,010 seconds.
at com.ibm.ws.rsadapter.AdapterUtil.toSQLException(AdapterUtil.java:1680)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:661)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:611)
Caused by:
com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException: J2CA1010E: Connection not available; timed out waiting for 180,010 seconds.
at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1781)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3904)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:3082)
20210505140731.341 ERROR com.fiserv.ppx.sso.controller.SSOController : SSO Configuration error
java.lang.NullPointerException
at com.fiserv.ppx.business.db.PPXDbTransactionManager.<init>(PPXDbTransactionManager.java:60)
at com.fiserv.ppx.sso.impl.SSOLoginAuthenticator.authenticateSSOUser(SSOLoginAuthenticator.java:157)
I got strange behavior when my application tries to send an email.
20:59:08,926 ERROR [release.com.mycompany.mail.GenericMail] (EJB default - 5) [MY_EJB INBOUND] Sending message failed!: javax.mail.MessagingException: Can't send command to SMTP host;
nested exception is:
java.net.SocketException: Broken pipe
at com.sun.mail.smtp.SMTPTransport.sendCommand(SMTPTransport.java:2106) [mail-1.4.4-redhat-2.jar:1.4.4-redhat-2]
at com.sun.mail.smtp.SMTPTransport.sendCommand(SMTPTransport.java:2093) [mail-1.4.4-redhat-2.jar:1.4.4-redhat-2]
at com.sun.mail.smtp.SMTPTransport.close(SMTPTransport.java:1184) [mail-1.4.4-redhat-2.jar:1.4.4-redhat-2]
at javax.mail.Transport.send0(Transport.java:197) [mail-1.4.4-redhat-2.jar:1.4.4-redhat-2]
at javax.mail.Transport.send(Transport.java:124) [mail-1.4.4-redhat-2.jar:1.4.4-redhat-2]
at com.mycompany.MailUtils.sendMail(MailUtils.java:258) [classes:]
Before this exception some timeout exception has been thrown:
20:57:50,291 ERROR [org.jboss.as.ejb3] (EJB default - 9) [ ] JBAS014122: Error during
retrying timeout for timer: [id=6be904b5-c1ef-4f0e-a277-d4c9f93e21b3 timedObjectId=SOME_EJB
auto-timer?:false persistent?:false timerService=org.jboss.as.ejb3.timerservice.TimerServiceImpl#99fdab1 initialExpiration=/* date */ 00:00:00 UTC 2015 intervalDuration(in milli sec)=0 nextExpiration=/* other date */ 21:00:00 UTC 2015
timerState=RETRY_TIMEOUT: javax.ejb.EJBTransactionRolledbackException: JBAS014373:
EJB 3.1 PFD2 4.8.5.5.1 concurrent access timeout on org.jboss.invocation.InterceptorContext$Invocation#66668127 - could not obtain lock within 5000MILLISECONDS
To be honest i don't know whats happened. I got these kind of exception for 5 hours.
I want to know whats happened and be able to avoid exceptions in future.
UPDATE 1
SOME_EJB is an ejb with works with timerService. Runs every 3 minutes, when the conditions are met sends an email.
My only idea is that there was some network/database issue and it caused that single execution of task took more than 3 minutes.
MailUtils is a #Stateless ejb
Is the mail sent or not?
It looks like JavaMail is closing the connection to the mail server when the exception occurs. If a previous error caused the server to drop the connection immediately, JavaMail may be getting this exception while trying to send the SMTP BYE command before closing the connection.
Turn on JavaMail Session debugging to see what error the server might have reported before this exception occurs.
I have the following problem:
I try to connect to an ActiveMQ broker (which is now down) using the following piece of code
connectionFactory = new ActiveMQConnectionFactory(this.url + "?timeout=2000");
connection = connectionFactory.createConnection();
connection.start();
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
LOGGER.info("Connected to " + this.url);
The problem is that the timeout does not have any effect
connection.start()
is blocked forever.
I inspected ActiveMQ log and found the following info:
2013-12-20 01:49:03,149 DEBUG [ActiveMQ Task-1] (FailoverTransport.java:786) - urlList connectionList:[tcp://localhost:61616?timeout=2000], from: [tcp://localhost:61616?timeout=2000]
2013-12-20 01:49:03,149 DEBUG [ActiveMQ Task-1] (FailoverTransport.java:1040) - Connect fail to: tcp://localhost:61616?timeout=2000, reason: java.lang.IllegalArgumentException: Invalid connect parameters: {timeout=2000}
The timeout parameter is specified here http://activemq.apache.org/cms/configuring.html
Has anybody any idea how to pass timeout argument to ActiveMQConnectionFactory?
Or how to set a timeout for connection.start() ?
Thank you!
Update: I found this on Stackoverflow: ActiveMQ - CreateSession failover timeout after a connection is resumed . I tried it but the following exception is thrown:
javax.jms.JMSException: Could not create Transport. Reason: java.lang.IllegalArgumentException: Invalid connect parameters: {transport.timeout=5000}
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:35)
I use ActiveMQ 5.8.0 from maven repo
It appears that your url is invalid still in both cases when attempting to set the timeout property.
If you're trying to have a failover URL, which it looks like you are since it is getting in to the Failover code then you're probably looking for initialReconnectDelay (and possibly maxReconnectAttempts which would throw an exception if the server is still down after the number of attempts is reached).
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory("failover://(tcp://localhost:61616)?initialReconnectDelay=2000&maxReconnectAttempts=2");
I'm using a Jetty based servlet to do RPC and I'm having an issue where a request that takes a long time throws the following exception on the server:
2012-02-11 21:07:07,673 [btpool0-4] DEBUG org.mortbay.log - EXCEPTION
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
at org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:99)
at org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector
.java:190)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:277)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:203)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:357)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.
java:217)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool
.java:475) 2012-02-11 21:07:07,674 [btpool0-4] DEBUG org.mortbay.log
- EOF
I tried setting the Connection,Keep-Alive http request property but that had no effect and from what I can gather, http 1.1 (which I'm pretty sure I'm using) is persistent by default.
So I think there are 2 ways I can try to address this:
figure out how to prevent the timeout exception from being
thrown at all
Have the client issue the initial request without waiting
for a response, and then ping with separate requests to check when
the server is done.
Update (2/12/2012): I set the maxIdleTime as Tim suggested and that did extend the time before the timeout occurred, but then I started getting a new exception:
2012-02-11 23:24:01,187 [btpool0-1] DEBUG org.mortbay.log - EXCEPTION
java.io.IOException: An existing connection was forcibly closed by the
remote host at sun.nio.ch.SocketDispatcher.read0(Native Method) at
sun.nio.ch.SocketDispatcher.read(Unknown Source) at
sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source) at
sun.nio.ch.IOUtil.read(Unknown Source) at
sun.nio.ch.SocketChannelImpl.read(Unknown Source) at
org.mortbay.io.nio.ChannelEndPoint.fill(ChannelEndPoint.java:129) at
org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:277) at
org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:203) at
org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:357) at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:329)
at
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
So something outside of Jetty was killing the connection, I suspect most likely a firewall. So what I ended up doing was making the server process the request with multiple threads; the original thread would immediately respond to the http request and a second thread would be kicked off to perform the action that was taking a long time. The client would then poll with http requests to check when the action on the server was complete.
This is a socket timeout, so nothing you do at the HTTP level can fix it - hence your keep alive not achieving anything.
Try setting the maxIdleTime on the SocketConnector
See here: http://docs.codehaus.org/display/JETTY/Configuring+Connectors ( archive link )