The below log messages indicate that the connection between my client and the server is being closed by the server since i am getting "Read thread is stopped for socket" from the Wireshark the message indicates that the server is closing the connection after it has sent the final "ACK" message indeed , the server is sending a FIN,ACK after the client sends a Capabilities-Exchange message. This suggests that the server is properly receiving the client's message, but for some reason it is not responding with a message of its own and instead is closing the connection. the localhost is my laptop , below is the logs
2023-01-09 13:11:43,946 INFO DictionaryImpl - Mobicents Diameter Dictionary loaded in 188ms -- Vendors[10] Commands[52] Types[20] AVPs[510]
2023-01-09 13:11:44,044 INFO StackImpl - (-)(-)(-)(-)(-) Started Mobicents DIAMETER Stack v1.5.10.0-build639 (-)(-)(-)(-)(-)
2023-01-09 13:11:44,544 INFO StackCreator - Diameter CLIENT :: Adding Listener for [AppId [Vendor-Id:0; Auth-Application-Id:4; Acct-Application-Id:0]].
2023-01-09 13:11:44,547 INFO StackCreator - Diameter CLIENT :: Supporting 1 applications.
2023-01-09 13:11:45,152 INFO NetworkGuard - Open server socket ServerSocket[addr=/127.0.0.1,localport=8080]
2023-01-09 13:12:00,267 INFO TCPTransportClient - Read thread is stopped for socket [Socket[addr=10.10.10.113/10.10.203.113,port=3868,localport=60884]]
2023-01-09 13:12:00,267 INFO TCPTransportClient - Read thread is stopped for socket [Socket[addr=10.10.10.113/10.105.10.113,port=3868,localport=60884]]
2023-01-09 13:12:00,267 INFO TCPTransportClient - Read thread is stopped for socket [Socket[addr=10.10.10.113/10.105.10.113,port=3868,localport=60884]]
do you have any idea please
I am expecting to connect to the server and send sendInitial request
Related
I have a Netty application that keeps track of connected web socket connections. When a web socket disconnects, I need to perform some logic that cleans up the stored data on that connection. I'm trying to be a good Netty citizen and not block the event loop with this cleanup logic, so I'm running everything in a CompletableFuture. The logic looks vaguely like this:
#Override
public void channelInactive(ChannelHandlerContext ctx) {
String connectionId = ctx.channel().id().asLongText();
LOGGER.info("Web socket disconnected: {}.", connectionId);
connectionStore
.deleteConnection(connectionId)
.thenCompose(
(connection) → {
LOGGER.info("Deleted connection {}, publishing event.", connectionId);
return messageBusPublisher.publish(new DisconnectEvent(connectionId));
})
.thenAccept((v) -> LOGGER.info("Done cleaning up connection {}.", connectionId));
}
This results in log statements like this:
2020-02-10 17:47:23,091 [nioEventLoopGroup-6-2] INFO c.e.WebSocketFrameHandler - Web socket disconnected: acde48fffe001122-000080ec-00000004-d86b4b1c219b503a-d8239aa2.
2020-02-10 17:47:23,091 [nioEventLoopGroup-6-2] INFO c.e.store.ConnectionStore - Attempting to delete connection for acde48fffe001122-000080ec-00000004-d86b4b1c219b503a-d8239aa2
2020-02-10 17:47:23,092 [nioEventLoopGroup-6-1] INFO c.e.WebSocketFrameHandler - Web socket disconnected: acde48fffe001122-000080ec-00000003-60f2c3402164ad79-a7e03a19.
2020-02-10 17:47:23,092 [nioEventLoopGroup-6-1] INFO c.e.store.ConnectionStore - Attempting to delete connection for acde48fffe001122-000080ec-00000003-60f2c3402164ad79-a7e03a19
2020-02-10 17:47:23,093 [nioEventLoopGroup-6-3] INFO c.e.WebSocketFrameHandler - Web socket disconnected: acde48fffe001122-000080ec-00000005-6962838a219b52e9-e6ceac8b.
2020-02-10 17:47:23,093 [nioEventLoopGroup-6-3] INFO c.e.store.ConnectionStore - Attempting to delete connection for acde48fffe001122-000080ec-00000005-6962838a219b52e9-e6ceac8b
2020-02-10 17:47:23,093 [nioEventLoopGroup-6-5] INFO c.e.WebSocketFrameHandler - Web socket disconnected: acde48fffe001122-000080ec-00000007-ccfcc4a5219b57eb-d78fc5f5.
2020-02-10 17:47:23,093 [nioEventLoopGroup-6-4] INFO c.e.WebSocketFrameHandler - Web socket disconnected: acde48fffe001122-000080ec-00000006-6c933ab1219b554f-2b3cf759.
2020-02-10 17:47:23,093 [nioEventLoopGroup-6-5] INFO c.e.store.ConnectionStore - Attempting to delete connection for acde48fffe001122-000080ec-00000007-ccfcc4a5219b57eb-d78fc5f5
2020-02-10 17:47:23,093 [nioEventLoopGroup-6-4] INFO c.e.store.ConnectionStore - Attempting to delete connection for acde48fffe001122-000080ec-00000006-6c933ab1219b554f-2b3cf759
2020-02-10 17:47:23,097 [lettuce-kqueueEventLoop-4-1] INFO c.e.store.ConnectionStore - Successfully deleted connection for acde48fffe001122-000080ec-00000004-d86b4b1c219b503a-d8239aa2
2020-02-10 17:47:23,102 [lettuce-kqueueEventLoop-4-1] INFO c.e.WebSocketFrameHandler - Deleted connection acde48fffe001122-000080ec-00000004-d86b4b1c219b503a-d8239aa2, publishing event.
2020-02-10 17:47:23,127 [lettuce-kqueueEventLoop-4-1] INFO c.e.service.MessageBusPublisher - Attempting to publish message for acde48fffe001122-000080ec-00000004-d86b4b1c219b503a-d8239aa2
2020-02-10 17:47:23,866 [sdk-async-response-0-0] INFO c.e.service.MessageBusPublisher - Successfully published message for acde48fffe001122-000080ec-00000004-d86b4b1c219b503a-d8239aa2
2020-02-10 17:47:23,866 [sdk-async-response-0-0] INFO c.e.WebSocketFrameHandler - Done cleaning up connection acde48fffe001122-000080ec-00000004-d86b4b1c219b503a-d8239aa2
The problem I'm running into is that when multiple disconnects happen in short order, it seems that the full cleanup logic is only run for the first one or two connections, and beyond that, nothing after the initial call is processed. This means the connections are properly deleted from the store, but no follow up actions are processed beyond that. My assumption is that this is because the CompletableFuture that represents the full chain of calls is not bound to anything, so when the first calls in the chain complete, the callbacks are not executed.
I've tried various things like using ctx.channel().eventLoop().execute(...) to run the clean up code (and theoretically bind the execution to the channel's event loop), but I see the same results.
My question then is: what is the appropriate way to fire off I/O bound tasks in response to Netty channel events?
As discussed above in the comments, this had nothing to do w/ how I was handling the long running work, but instead was the lack of error handling on my futures.
Remember to always catch exceptions from your futures, kids!
I have a frequent Channel shutdown: connection error issues (under 24.133.241:5671 thread, name is truncated) in RabbitMQ Java client (my producer and consumer are far apart). Most of the time consumer is automatically restarted as I have enabled heartbeat (15 seconds). However, there were some instances only Channel shutdown: connection error but no Consumer raised exception and no Restarting Consumer (under cTaskExecutor-4 thread).
My current workaround is to restart my application. Anyone can shed some light on this matter?
2017-03-20 12:42:38.856 ERROR 24245 --- [24.133.241:5671] o.s.a.r.c.CachingConnectionFactory
: Channel shutdown: connection error
2017-03-20 12:42:39.642 WARN 24245 --- [cTaskExecutor-4] o.s.a.r.l.SimpleMessageListenerCont
ainer : Consumer raised exception, processing can restart if the connection factory supports
it
...
2017-03-20 12:42:39.642 INFO 24245 --- [cTaskExecutor-4] o.s.a.r.l.SimpleMessageListenerCont
ainer : Restarting Consumer: tags=[{amq.ctag-4CqrRsUP8plDpLQdNcOjDw=21-05060179}], channel=Ca
ched Rabbit Channel: AMQChannel(amqp://21-05060179#10.24.133.241:5671/,1), conn: Proxy#7ec317
54 Shared Rabbit Connection: SimpleConnection#44bac9ec [delegate=amqp://21-05060179#10.24.133
.241:5671/], acknowledgeMode=NONE local queue size=0
Generally, this is due to the consumer thread being "stuck" in user code somewhere, so it can't react to the broken connection.
If you have network issues, perhaps it's stuck reading or writing to a socket; make sure you have timeouts set for any I/O operations.
Next time it happens take a thread dump to see what the consumer threads are doing.
A code that worked me for a while now on various platforms started to suddenly to fail, there were changed in the code but not in an area that related to this place.
The line of code that throws the exceptions is:
(SocketChannel) selectionKey.channel()).finishConnect()
And it throws:
java.net.SocketException: Permission denied: no further information
what may be the issue?
Some log lines that might show more info, it is a test code:
2016-12-04 22:50:10,585 [main] DEBUG Socket - Socket-Client-1-1
register to connect to address: 0.0.0.0/0.0.0.0:8080
2016-12-04 22:50:10,585 [main] DEBUG SocketSelector - Registering keys:
OP_CONNECT (for Socket-Client-1-1)
2016-12-04 22:50:10,595 [main] DEBUG SocketSelector - Registered to: OP_CONNECT > 2016-12-04 22:50:10,595 [NetworkThread] DEBUG SocketSelector -Got selected keys
for channel (java.nio.channels.SocketChannel[connection-pending
remote=0.0.0.0/0.0.0.0:8080])
The issue is that you are attempting to connect to 0.0.0.0, which is not a valid TCP target address. This causes EPERM for the reasons described in man connect.
Use 127.0.0.1 or a proper IP address.
I'm having trouble in mule service. Daily, the service crashes, about 4 times a day, making it necessary to redeploy my service.
Here's my stack trace.
ERROR 2016-02-04 10:30:38,063 [[app_service].NoSessionConnector.receiver.03] org.mule.exception.DefaultMessagingExceptionStrategy:
********************************************************************************
Message : Timeout for connection (java.net.SocketException). Message payload is of type: HttpResponse
Code : MULE_ERROR--2
--------------------------------------------------------------------------------
Exception stack is:
1. Timeout for connection (java.net.SocketException)
java.net.SocketOutputStream:-2 (null)
2. Timeout for connection (java.net.SocketException). Message payload is of type: HttpResponse (org.mule.execution.ResponseDispatchExcepti
on)
org.mule.transport.http.HttpMessageProcessTemplate:141 (http://www.mulesoft.org/docs/site/current3/apidocs/org/mule/execution/ResponseDispatc
hException.html)
--------------------------------------------------------------------------------
Root Exception stack trace:
java.net.SocketException: Tempo esgotado para conexão
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
+ 3 more (set debug level logging or '-Dmule.verbose.exceptions=true' for everything)
********************************************************************************
I've tried changing the number of threads in maxThreadsActive and maxBufferSize, but is still occurring the same error
Seems like the tcp connection has not been usable for a long time.
Use the the Socket.setKeepAlive() or apply the heart beat mechanism.
I've been pounding my head on this for a while, I hope someone can help?
I have some JavaPNS code which pushes fine to my device when running from my local machine, however, when I copy that code over to my server, everything runs fine, no errors, but I never get the alert on my device?
After examining the logs from my server compared to my local box, I noticed I never get the flushing message on my server, I am using the JavaPNS queue, with 30 threads. In both cases, local box and server, I am sending less than 30 alerts.
public class PushWorker
{
private PushQueue queue = null;
public PushWorker(File keystore, String password, boolean production) throws KeystoreException
{
BasicConfigurator.configure();
int threads = 30;
this.queue = Push.queue(keystore, password, production, threads);
queue.start();
}
public void push(String message, String sound, String token, String eventId) throws JSONException
{
BasicDevice bd = new BasicDevice();
bd.setToken(token);
PushNotificationPayload payload = PushNotificationPayload.complex();
payload.addAlert(message);
payload.addSound(sound);
payload.addCustomDictionary("eid", eventId);
push(payload, bd);
}
private void push(Payload payload, Device device)
{
queue.add(payload, device);
}
}
----BELOW is flush message from my local box --------------
4872 [JavaPNS grouped notification thread in QUEUE mode] DEBUG javapns.notification.PushNotificationManager - Flushing
4872 [JavaPNS grouped notification thread in QUEUE mode] DEBUG javapns.notification.PushNotificationManager - At this point, the entire 139-bytes message has been streamed out successfully through the SSL connection
4872 [JavaPNS grouped notification thread in QUEUE mode] DEBUG javapns.notification.PushNotificationManager - Notification sent on first attempt
Can I force the flush somehow from the queue?
----------------------BELOW is the server JavaPNS logging-----------------------
0 [JavaPNS grouped notification thread in QUEUE mode] DEBUG javapns.communication.ConnectionToAppleServer - Creating SSLSocketFactory
16 [JavaPNS grouped notification thread in QUEUE mode] DEBUG javapns.communication.ConnectionToAppleServer - Creating SSLSocketFactory
49 [JavaPNS grouped notification thread in QUEUE mode] DEBUG javapns.communication.ConnectionToAppleServer - Creating SSLSocket to gateway.sandbox.push.apple.com:2195
49 [JavaPNS grouped notification thread in QUEUE mode] DEBUG javapns.communication.ConnectionToAppleServer - Creating SSLSocket to gateway.sandbox.push.apple.com:2195
177 [JavaPNS grouped notification thread in QUEUE mode] DEBUG javapns.notification.PushNotificationManager - Initialized Connection to Host: [gateway.sandbox.push.apple.com] Port: [2195]: 5117f31e[SSL_NULL_WITH_NULL_NULL: Socket[addr=gateway.sandbox.push.apple.com/17.172.233.65,port=2195,localport=56015]]
...
...
DEBUG javapns.notification.Payload - Adding alert [blah, blah my alert]
14767 [main] DEBUG javapns.notification.Payload - Adding sound [default]
14767 [main] DEBUG javapns.notification.Payload - Adding custom Dictionary [eid] = [193790]
14776 [JavaPNS grouped notification thread in QUEUE mode] DEBUG javapns.notification.PushNotificationManager - Building Raw message from deviceToken and payload
14776 [JavaPNS grouped notification thread in QUEUE mode] DEBUG javapns.notification.PushNotificationManager - Built raw message ID 16777217 of total length 135
14777 [JavaPNS grouped notification thread in QUEUE mode] DEBUG javapns.notification.PushNotificationManager - Attempting to send notification: {"aps":{"sound":"default","alert":"blah, blah my alert"},"eid":"193790"}
14777 [JavaPNS grouped notification thread in QUEUE mode] DEBUG javapns.notification.PushNotificationManager - to device: [my device number]
And that's it, no flush...
Do you have iptables configured on the server or some other firewall device?
Maybe port 2195 needs to be allowed.
Try using telnet to test:
$ telnet gateway.sandbox.push.apple.com 2195
Do you have SELinux enabled? check the following setting:
$ getsebool -a | grep httpd
httpd_can_network_connect --> ?
make sure it is on.
Did you check the documentation on the javapns repo site?
Let us know how it goes.
Bill