We have a microservice developed using Sprig Boot. This microservice will called an external Rest API. This is deployed on multiple servers. On one of the servers we are seeing the following error
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:743)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:669)
at org.springframework.web.client.RestTemplate.postForObject(RestTemplate.java:413)
Caused by: javax.net.ssl.SSLException: Connection reset
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:127)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:320)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:258)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:137)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402)
....
Suppressed: java.net.SocketException: Broken pipe (Write failed)
at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
at java.base/sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:81)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:351)
....
Caused by: java.net.SocketException: Connection reset
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:186)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:448)
at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:165)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:108)
on the other servers the same service is running fine.
We tried traceroute on this server and other servers but the output on both are same. We restarted the service but same error.
Any pointers on what we can debug and find where the issue is?
This thread can be closed as after the restart of that service, its working fine.
Related
It is an swing desktop application we have functionality to login.
In one of our client machine with window 7 we are not able to connect to the our auth service due to the following error :-
org.apache.oltu.oauth2.common.exception.OAuthSystemException: java.net.SocketException: Permission denied: connect
at org.apache.oltu.oauth2.client.URLConnectionClient.execute(URLConnectionClient.java:108)
at org.apache.oltu.oauth2.client.OAuthClient.accessToken(OAuthClient.java:65)
at org.apache.oltu.oauth2.client.OAuthClient.accessToken(OAuthClient.java:55)
at org.apache.oltu.oauth2.client.OAuthClient.accessToken(OAuthClient.java:71)
at database.TvilightIO.getAccessToken(TvilightIO.java:53)
at gui.MainScreen.showLoginScreen(MainScreen.java:218)
at gui.MainScreen.<init>(MainScreen.java:48)
at tester.Tester.<init>(Tester.java:112)
at tester.Tester.main(Tester.java:312)
Caused by: java.net.SocketException: Permission denied: connect
at java.base/java.net.PlainSocketImpl.connect0(Native Method)
at java.base/java.net.PlainSocketImpl.socketConnect(PlainSocketImpl.java:101)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:
242)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
at java.base/java.net.Socket.connect(Socket.java:608)
at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:285)
at java.base/sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:182)
at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:265)
at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:372)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(
AbstractDelegateHttpsURLConnection.java:191)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.jav
a:1187)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java
:1081)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractD
elegateHttpsURLConnection.java:177)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.
java:1362)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.j
ava:1337)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConne
ctionImpl.java:246)
at org.apache.oltu.oauth2.client.URLConnectionClient.execute(URLConnectionClient.java:85)
... 8 more
As we have migrated the our application from Java 8 to java 11.
This issue was related to firewall restriction applied by the network.
So once host was permitted through firewall, then it started working perfectly.
Encountered Internal Server Exception
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "localhost:8034/v1/users/81ebc4ca-2496-4cc3-b533-311f237b6fe9/crop_details/10/summary": Read timed out; nested exception is java.net.SocketTimeoutException: Read timed out
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:666)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:621)
at org.springframework.web.client.RestTemplate.getForEntity(RestTemplate.java:321)
at ...
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1535)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:84)
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:652)
... 91 common frames omitted
Scenario - When I am trying to load testing using JMeter for Rest API from one service(running on 8033 port). It internally calls using RestTemplate to other service rest API(above one on 8034). For few request it works fine but as I go beyond just 10 concurrent requests it start giving above error.
What I tried till now
I already tried using Different ClientManagers to set both connection and read time out values.
Also tried using PoolConnectionMangers.
Custom Keep Alive Strategy.
Setting JDK's values for above time out properties.
But Still getting same problem.
Links referred
http://www.baeldung.com/httpclient-connection-management
http://www.baeldung.com/rest-template
http:/blog.technogemsinc.com/2010/08/java-url-connection-timeout-default.html
Same thing works on my local machine for 10000 request without any problem.
Box/Server details - 3.7 GB/50GB single core CPU
We have abrupt connection close issue to rabbitmq on localhost dev/testing scenario. In our development environment we have rabbitmq installed on each developers Windows 7 machine, and we connect to it using java client via Spring AMQP library. Everything works fine for a while but at some point of time the connection is dropped with the following message in rabbitmq log:
=WARNING REPORT==== 4-Jan-2016::14:39:37 ===
closing AMQP connection <0.3731.0> (127.0.0.1:50792 -> 127.0.0.1:5672):
connection_closed_abruptly
In the client log we have this exception:
04/01/2016 14:39:37.181 (AMQP Connection 127.0.0.1:5672) ERROR [CachingConnectionFactory] Channel shutdown: connection error
04/01/2016 14:39:38.188 (SimpleAsyncTaskExecutor-2) WARN [SimpleMessageListenerContainer] Consumer raised exception, processing can restart if the connection factory supports it
com.rabbitmq.client.ShutdownSignalException: connection error
at com.rabbitmq.client.impl.AMQConnection.startShutdown(AMQConnection.java:723)
at com.rabbitmq.client.impl.AMQConnection.shutdown(AMQConnection.java:713)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:571)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: Unrecognized Windows Sockets error: 0: recv failed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:288)
at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95)
at com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:139)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:536)
... 1 more
In the client start up we create a dynamic queue and attach several consumers to it. This connection drop causes the queue to be removed and any subsequent attempt to recreate it fails with the following exception:
04/01/2016 14:39:38.239 (SimpleAsyncTaskExecutor-8) WARN [BlockingQueueConsumer] Queue declaration failed; retries left=3
org.springframework.amqp.rabbit.listener.BlockingQueueConsumer$DeclarationException: Failed to declare queue(s):[MY_TEST_QU]
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.attemptPassiveDeclarations(BlockingQueueConsumer.java:571)
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:470)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1165)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:106)
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:102)
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:124)
at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:885)
at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:704)
at com.sun.proxy.$Proxy77.queueDeclarePassive(Unknown Source)
at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.attemptPassiveDeclarations(BlockingQueueConsumer.java:550)
... 3 more
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no queue 'MY_TEST_QU' in vhost '/', class-id=50, method-id=10)
at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67)
at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:33)
at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:361)
at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:226)
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:118)
... 12 more
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no queue 'MY_TEST_QU' in vhost '/', class-id=50, method-id=10)
at com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:484)
at com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:321)
at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:144)
at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:91)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:554)
... 1 more
We have had this problem with rabbitmq 3.4.x and 3.5.x and Spring AMQP 1.3.x and 1.5.x. The interesting thing is this never happens in our QA and production environment where rabbitmq is installed on a separate server.
Any help is much appreciated.
The org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.attemptPassiveDeclarations() fails when there is no the queue to listen to anymore.
It isn't queue recreation it is just PassiveDeclarations. So, if you drop that "dynamic queue" manually, you should recreate it manually as well.
I mean if you don't follow with the RabbitAdmin bean contract, you don't have choice unless re-create it manually. You don't need to worry about the listener in that case: it will reconnect correctly, when the queue comes back.
UPDATE
Starting with version 1.5 Spring AMQP provides ListenerContainerConsumerFailedEvent, which is emitted exactly after all attemptPassiveDeclarations() attempts. So, you can catch that ApplicationEvent, stop() your listener container, declare queue back and start() the container again.
If your queue isn't a bean, it won't be redeclared automatically.
Here is the scenario
I am trying to connect to mysql database server from the core java app using spring. My application and server runs on different box i.e. Have limited ports open
Here is what I have done soo far
I am able to telnet to port 5506 (on which my database server run ) from the client machine
I am able to connect to database server from command prompt(mysql client) from client machine
3.I tried all driver version from 5.0.8 to latest
and off course same code works on everywhere except production :)
Somehow I started thinking that Java process required more ports to be open
Here is the exception
org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception:
** BEGIN NESTED EXCEPTION **
java.net.ConnectException
MESSAGE: Connection refused
STACKTRACE:
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
at java.net.Socket.connect(Socket.java:546)
at java.net.Socket.connect(Socket.java:495)
at java.net.Socket.<init>(Socket.java:392)
at java.net.Socket.<init>(Socket.java:235)
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:256)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:271)
at com.mysql.jdbc.Connection.createNewIO(Connection.java:2771)
at com.mysql.jdbc.Connection.<init>(Connection.java:1555)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:285)
at java.sql.DriverManager.getConnection(DriverManager.java:620)
at java.sql.DriverManager.getConnection(DriverManager.java:169)
at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriverManager(DriverManagerDataSource.java:173)
at org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:164)
at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnectionFromDriver(AbstractDriverBasedDataSource.java:149)
at org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnection(AbstractDriverBasedDataSource.java:119)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:113)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:381)
at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:455)
at org.springframework.jdbc.core.JdbcTemplate.queryForRowSet(JdbcTemplate.java:498)
at
Have you specified the 5506 port in the connection URL? This port is not the default one for MySQL (3306).
When I call, the service's method, I have Web Service Exception happening which I couldn't figure out for now.
The issue is this, I am running NetBeans 6.8 against IIS 7.0 with the Web Service written using WCF:
javax.xml.ws.WebServiceException: java.net.SocketException: Connection reset
at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.readResponseCodeAndMessage(HttpClientTransport.java:201)
at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:151)
at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:83)
at com.sun.xml.internal.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:105)
at com.sun.xml.internal.ws.api.pipe.Fiber.__doRun(Fiber.java:587)
at com.sun.xml.internal.ws.api.pipe.Fiber._doRun(Fiber.java:546)
at com.sun.xml.internal.ws.api.pipe.Fiber.doRun(Fiber.java:531)
at com.sun.xml.internal.ws.api.pipe.Fiber.runSync(Fiber.java:428)
at com.sun.xml.internal.ws.client.Stub.process(Stub.java:211)
at com.sun.xml.internal.ws.client.sei.SEIStub.doProcess(SEIStub.java:124)
at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:98)
at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:78)
at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(SEIStub.java:107)
at $Proxy30.getTwo(Unknown Source)
at HighBeamWcfClient.Main.main(Main.java:28)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:652)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1072)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:373)
at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.readResponseCodeAndMessage(HttpClientTransport.java:198)
How to fix it?
"Connection reset" typically means that someone/something between you and the server is resetting the connection or somehow blocking it. Check that you can connect to the server on the port correctly without any issue, and that there are no firewalls in the way.
A very simple test that you can connect to the port in question is to simply telnet to it with telnet serverhostname portnum, for example telnet stackoverflow.com 80.