I have an issue with one of my AMQ producers, I have networks issues with it, consequently it fails to send messages to the Q.
However, the server opens a connections to the Q machine and maintain them intact. The number of connections raises, the Q machine is suffocated and cannot serve the rest of its producers and consumers.
Here is the exception I catch in my "bad" producer:
Caused by: javax.jms.JMSException: Wire format negotiation timeout: peer did not send his wire format.
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72)
at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1413)
at org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1478)
at org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527)
at com.queue.service.ActiveMQ.ActiveMQWriter.initSession(ActiveMQWriter.java:163)
... 4 more
Caused by: java.io.IOException: Wire format negotiation timeout: peer did not send his wire format.
at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:99)
at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1388)
... 7 more
Running netstat command on my "bad" producer gives me State ESTABLISHED for every failed request.
Running netstat command on my Q server gives me State ESTABLISHED from the "bad" producer, for every failed request.
Question: what parameter I need to modify in the producer and/or Q in order to "break" those connections after a configurable time?
The best solution is to use failover transport, so that your clients can try connecting again if the first attempt fails.
You can also try extending wire format negotiation period (default 10 sec).
You can do that by using wireFormat.maxInactivityDurationInitalDelay property on the connection URL in your client to increase to 30 sec timeout
ActiveMQConnectionFactory cf = new ActiveMQConnectionFactory("failover:(tcp://localhost:61616?wireFormat.maxInactivityDurationInitalDelay=30000)");
http://activemq.apache.org/javaxjmsjmsexception-wire-format-negociation-timeout-peer-did-not-send-his-wire-format.html
Related
I get the following exception at times:
ERROR : 07.16.2021:0709 (05.988) [[]http-nio-8080-exec-4] ShippingSatelliteForm: /addLeadTimeForShippingBakeries.xhtml, user - The bean encountered a non-application exception; nested exception is: ERROR : 07.16.2021:0709 (05.988) [[]http-nio-8080-exec-4] ShippingSatelliteForm: /addLeadTimeForShippingBakeries.xhtml, user - The bean encountered a non-application exception; nested exception is: org.hibernate.exception.JDBCConnectionException: could not execute query
Caused by: com.ibm.db2.jcc.am.DisconnectNonTransientConnectionException: [jcc][t4][2030][11211][4.21.29] A communication error occurred during operations on the connection's underlying socket, socket input stream, Caused by: com.ibm.db2.jcc.am.DisconnectNonTransientConnectionException: [jcc][t4][2030][11211][4.21.29] A communication error occurred during operations on the connection's underlying socket, socket input stream, or socket output stream. Error location: Reply.fill() - insufficient data (-1). Message: Insufficient data. ERRORCODE=-4499, SQLSTATE=08001
I have deployed the application in Azure Kubernetes and DB2 is in on-premises - is this happening due to different networks?
Guessing here, but I would try to add hibernate.c3p0.max_idle_time=300 to have the connection pool recycle connections every 5th minute. This prevents connections from living long enough for someone else (database, firewall etc) to decide to kill it...
Try testing whether you can still use an idle connection after:
15 seconds
30 seconds
60 seconds
etc.
Most probably you use some kind of NAT-routing, virtual (overlay) network, load-balancer etc. that removes idle connections to avoid running out of memory.
Connection Pools are important to achieve good performance if you run many queries per second, but and if you don't need the connection for many seconds you can either:
close the connection
retest whether it still is valid
configure TCP Keep-Alive (on server or client side)
Default TCP Keep-Alive settings depend on OS configuration and usually require both enabling and setting the correct frequency because the default frequency is usually to late to avoid a timeout. Setting these from Java has only recently become practical and is probably not yet support by the DB2 driver.
See: How to set socket option (TCP_KEEPCNT, TCP_KEEPIDLE, TCP_KEEPINTVL) in java or netty?
However your application should be prepared for all kinds of trouble because of: https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing so you may want to add some kind of retry logic to take care of any kind of network issues that may arise now or in the future.
I am Using Amazon Mq as my Mqtt broker and when around 1000 requests are received simultaneously the mqtt broker breaks and disconnects. Can Anyone tell me how to use Amazon Mq as my broker & simultaneously solve the scaling problem also.
I'm assuming that you have created ActiveMQ as a singleton class. Right?
-For producing a message, you create an instance of PooledConnectionFactory like
-------//some code here
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory(MQTT_END_POINT);
connectionFactory.setUserName();
connectionFactory.setPassword();
PooledConnectionFactory pooledConnectionFactory = getActiveMQInstance().configurePooledConnectionFactory(activeMQConnectionFactory);
-------
This pooledConnectionFactory is used to create a connection then session and then destination is entered (as mentioned on AmazonMQ documentation). You send the message using MessageProducer object and close the MessageProducer, session and connection
-For consumption, there will be an always-alive-listener that is always ready for message to arrive. The consumer part, it follows the same process like consumerConnection, then session and then destination queue to listen on.
As far as I remember, this part is also mentioned in amazonMQ documentation.
There is one problem that the connection to broker is lost for consumer sometimes, (since producer reopens the connections, produces and closes, it is not observed in it). Remember, you will have to reestablish the connection for consumer.
If there is any variance from the above approach please mention. Also, add your amazonMQ broker picture showing the connection, queue, active consumers.
Just out of curiosity, what are the maximum connections you have set for the PooledConnectionFactory?
I have to call a web service that takes 40 seconds to respond. I'm in JBoss-EAP 6.4 and I use this code:
IService svc = service.getBasicHttpBindingIService();
[...]
((BindingProvider) svc).getRequestContext().put("com.sun.xml.ws.connect.timeout",timeout)
[...]
svc.processMessage(req);
When the timeout is less then 20000 (20 seconds), I get the expected exception:
HTTP transport error: java.net.SocketTimeoutException: connect timed out
If the timeout is bigger than 20000, i get another exeption after 20 second:
HTTP transport error: java.net.ConnectException: Connection timed out: connect
So, I can't get the response from the server (40 seconds to respond), I always get the exception. What kind of timeout I have to increase?
Why do you want the timeout to be 40 seconds? There are two timeouts represented by "com.sun.xml.internal.ws.connect.timeout" and "com.sun.xml.internal.ws.request.timeout". You could have been connected to the server, but the request is taking long to process in which case, the latter setting would take effect. Note: I believe these settings are for JBoss (I could be wrong, cross check it).
The design of my current stomp client process is as follows:
Open stomp connection (sending CONNECT frame)
Subscribe to a feed (send a SUBSCRIBE frame)
Do a loop to continually receive feed:
while (true) {
connection.begin("txt1");
StompFrame message = connection.receive();
System.out.println("message get header"+message.toString());
LOG.info(message.getBody());
connection.ack(message, "txt1");
connection.commit("txt1");
}
My problem with this process is that I get
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)...
and I think the cause of this is mostly because the feed I am subscribed to gives information slower on certain times (as I normally get this error when the weekend comes, holidays or evenings).
I have been reading up on this here and I think this would help with my problem. However, I'm not so sure how to incorporate it with the current layout of my stomp client. Would I have to send a CONNECT header within Step 3?
I am currently using activemq to create my stomp client if that helps.
In the stomp spec we have:
Regarding the heart-beats themselves, any new data received over the
network connection is an indication that the remote end is alive. In a
given direction, if heart-beats are expected every milliseconds:
the sender MUST send new data over the network connection at least every milliseconds
if the sender has no real STOMP frame to send, it MUST send a single newline byte (0x0A)
if, inside a time window of at least milliseconds, the receiver did not receive any new data, it CAN consider the
connection as dead
because of timing inaccuracies, the receiver SHOULD be tolerant and take into account an error margin
Would that mean my client would need to send a newline bye every n seconds?
The stomp server you are connected to has timed out your connection due to innactivity.
Providing the server supports Stomp version 1.1 or newer, the easiest solution for your client is to include a heart-beat instruction in the header of your CONNECT, such as "0,10000". This tells the server that you cannot send heart-beats, but you want it to send one every 10 seconds. This way you don't need to implement them, and the server will keep the connection active by sending them to you.
Of course the server will have its own requirements of the client. In your comment it responds to your request with "1000,0". This indicates that it will send a heart-beat every 1000 millisecs, and it expects you to send one every 0 millisecs, 0 indicating none at all. So your job will be minimal.
I'm issuing a socket connection, using the following snippet
Socket socket = new Socket();
InetSocketAddress endPoint = new InetSocketAddress("localhost", 1234);
try
{
socket.connect(endPoint, 30000);
}
catch (IOException e)
{
e.printStackTrace();
// Logging
}
The endpoint it is trying to connect to is offline, what I want it to do is to attempt to connect, and using the 30000ms timeout, wait for that period of time before it concludes a result
Currently, that 30000 parameter doesn't seem to be applied, as from the timestamp on my logging it appears that it is determining within 1 second that a connection failed.
How can I force the connect to wait for a set amount of time before giving up?
13:13:57,685 6235 DEBUG [Thread-7] - Unable to connect to [localhost:1234]
13:13:58,685 7235 DEBUG [Thread-7] - Unable to connect to [localhost:1234]
13:13:59,695 8245 DEBUG [Thread-7] - Unable to connect to [localhost:1234]
13:14:00,695 9245 DEBUG [Thread-7] - Unable to connect to [localhost:1234]
EDIT : The API does state Connects this socket to the server with a specified timeout value. A timeout of zero is interpreted as an infinite timeout. The connection will then block until established or an error occurs. however it appears I'm not experiencing such behaviour, or am not catering to it, most likely the latter
What you're getting here is correct. connect won't sit on a socket waiting until it sees a server, it will attempt to connect and wait for a response. if there is nothing to connect to, it returns. if there is something to connect to, it will wait timeout seconds for a response and fail if none is received.
You need to distinguish among several possible exception conditions.
ConnectException with the text 'connection refused', which means the host was up and reachable and nothing was listening at the port. This happens very quickly and cannot be subjected to a timeout.
NoRouteToHostException: this indicates a connectivity issue. Again it happens immediately and cannot be subjected to a timeout.
UnknownHostException: the host names cannot be resolved via DNS. This happens immediately, or rather after a generally short DNS delay, and cannot be subjected to a timeout.
ConnectException with any other text: this can indicate a failure to respond by the target system. Usually happens when firewalls are present. Can be subjected to a timeout.
You are doing the correct thing by calling Socket.connect() with a timeout parameter. If you don't do this, or if you specify a zero timeout, the default system timeout is used, which is of the order of 60-75 seconds depending on the platform. This is contrary to the Javadoc's statement about an 'infinite timeout', which is not correct. Also you cannot increase the timeout beyond this limit via Socket.connect() witha a timeout parameter. Alternatively you can use java.nio socket channels in non-blocking mode with a select() to administer the timeout for you, but you still can't increase the timeout beyond the platform default via this or any other method.
When the timeout occurs, a SocketTimeoutException exception is thrown which you do not catch and log. The IOException is fired when "an error occurs during the connection". The timeout is never applied because there's an error beforehand.
Edit: Just to clarify: TCP/IP as a suite has many specifics that could prevent a packet from reaching it's desired outcome (a SYN/ACK packet). If a computer responds to your SYN packet by an informing your application that the port is closed (i.e. there's no application running/listening there), it would fire an exception telling you that it is impossible to connect to that port. If you wish to send and re-send SYN packets either way with the knowledge that an application will come online listening on that port, this is done on a different network layer (and, as far as I know, is not accessible with Java out-of-the-box).
Try scocket.setSoTimeout(timeout) before connecting.