I have an application with client server architecture. The client
use Java Web Start with Java Swing / AWT and the sert uses HTTP server / Servlet with
Tomcat.
The communication is made from the serialization of objects, create a
ObjectOutput serializes a byte array and send to the server
respectively called the ObjectInputStream and deserializes.
The application follows communicating correctly to a certain
time of concurrency where starting to show error
"SocketException read timeout". The erro happens when the server invoke the method
ObjectInputStream.getObject() in my servlet doPost method.
The tomcat will come slow and the errors start to decrease server response time until the crash time where i must restart the server and after everything works.
Someone went through this problem ?
Client Code
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
OutputStream os = conn.getOutputStream();
ObjectOutputStream oss = new ObjectOutputStream(os);
oss.writeUTF("protocol header sample");
oss.writeObject(_parameters);
oss.flush();
oss.close();
Server Code
ObjectInputStream input = new ObjectInputStream(_request.getInputStream());
String method = input.readUTF();
parameters = input.readObject();
input.readObject() is where the error is
You haven't given us much information to go on, especially about the client side. But my suspicion is that the client side is:
failing to setting the Content-length header (or setting it to the wrong value),
failing to flush the output stream, and/or
not closing the output side of the socket.
Mysterious.
Based on your updated question, it looks like none of the above. Here are a couple of other possibilities:
For some reason the client side is either locking up entirely during serialization or taking a VERY LONG TIME.
There is a proxy between the client and server that is causing problems.
You are experiencing load-related network problems, or network hardware problems.
Another possible explanation is that you have a memory leak, and that the slowdown is caused by the GC taking more and more time as you run out of memory. This will show up in the GC logs if you have them enabled.
I think During high Concurrency, the Socket Timeout set in Tomcat is Expired and the connection is closed. The next read by Tomcat for that connection is greater than the server socket timeout specified in the server.
If you want to avoid this problem you have to increase the timeout on the server-side which is expired in your case. But not advisable.
BTW you did not give enough information. Did you increase the no of threads for connection in Tomcat? If you did, this surely would happen.
Related
I have Java web socket web application. The websocket endpoint interacts with mobile clients. In one of the use case, the web application needs to write bytes of the size of 10MB or more to the web-socket outputstream. Following is the code which writes to the output stream :
if (webSocSession.isOpen()) {
webSocSession.getBasicRemote().sendBinary(byteBuffer);
byteBuffer.clear();
}
I am getting the following exception at times when the writing to the web-socket :
IOException writing to web-socket
java.io.IOException: java.net.SocketTimeoutException: Write timeout
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlock(WsRemoteEndpointImplBase.java:324)
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlock(WsRemoteEndpointImplBase.java:259)
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendBytes(WsRemoteEndpointImplBase.java:131)
at org.apache.tomcat.websocket.WsRemoteEndpointBasic.sendBinary(WsRemoteEndpointBasic.java:43)
at test.web.websocket.LIMSEndpoint$SocketWorker.writeToWebSocket(LIMSEndpoint.java:1188)
at test.web.websocket.LIMSEndpoint$SocketWorker.run(LIMSEndpoint.java:1127)
Caused by: java.net.SocketTimeoutException: Write timeout
at org.apache.tomcat.util.net.SocketWrapperBase.vectoredOperation(SocketWrapperBase.java:1458)
at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:1376)
at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:1347)
at org.apache.tomcat.websocket.server.WsRemoteEndpointImplServer.doWrite(WsRemoteEndpointImplServer.java:93)
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.writeMessagePart(WsRemoteEndpointImplBase.java:509)
at org.apache.tomcat.websocket.WsRemoteEndpointImplBase.sendMessageBlock(WsRemoteEndpointImplBase.java:311)
I tried setting the up the following SEND timeout property to 0 ( infinite write timeout ) , but it does not help. The following Apache Tomcat doc helped - https://tomcat.apache.org/tomcat-8.5-doc/web-socket-howto.html
org.apache.tomcat.websocket.BLOCKING_SEND_TIMEOUT
The web socket session max idle timeout has been set to 0 ( not to timeout )
Session.setMaxIdleTimeout(0)
Any help in this regard will be great help.
I would suspect that the clients receive buffer is filled up and it cuts the connection. I know this will happen the other way around, when a client sends large data to an apache server and the server isn't configured to have an extended binary buffer. See: org.apache.tomcat.websocket.binaryBufferSize
Perhaps you can get around the problem by sending the data in smaller chunks, sendBytes have a partial version.
https://tomcat.apache.org/tomcat-10.0-doc/websocketapi/index.html?jakarta/websocket/Session.html
This question already has answers here:
Java socket API: How to tell if a connection has been closed?
(9 answers)
Closed 5 years ago.
When I'm using e.g. PuTTY and my connection gets lost (or when I do a manual ipconfig /release on Windows), it responds directly and notifies my connection was lost.
I want to create a Java program which monitors my Internet connection (to some reliable server), to log the date/times when my internet fails.
I tried use the Socket.isConnected() method but that will just forever return "true". How can I do this in Java?
Well, the best way to tell if your connection is interrupted is to try to read/write from the socket. If the operation fails, then you have lost your connection sometime.
So, all you need to do is to try reading at some interval, and if the read fails try reconnecting.
The important events for you will be when a read fails - you lost connection, and when a new socket is connected - you regained connection.
That way you can keep track of up time and down time.
Even though TCP/IP is "connection oriented" protocol, normally no data is sent over an idle connection. You can have a socket open for a year without a single bit sent over it by the IP stack. In order to notice that a connection is lost, you have to send some data on the application level.(*) You can try this out by unplugging the phone cable from your ADSL modem. All connections in your PC should stay up, unless the applications have some kind of application level keepalive mechanism.
So the only way to notice lost connection is to open TCP connection to some server and read some data from it. Maybe the most simple way could be to connect to some FTP server and fetch a small file - or directory listing - once in a while. I have never seen a generic server which was really meant to be used for this case, and owners of the FTP server may not like clients doing this.
(*) There is also a mechanism called TCP keepalive but in many OS's you have to activate it for all applications, and it is not really practical to use if you want to notice loss of connection quickly
If the client disconnects properly, a read() will return -1, readLine() returns null, readXXX() for any other X throws EOFException. The only reliable way to detect a lost TCP connection is to write to it. Eventually this will throw an IOException 'connection reset', but it takes at least two writes due to buffering.
Why not use the isReachable() method of the java.net.InetAddress class?
How this works is JVM implementation specific but:
A typical implementation will use ICMP ECHO REQUESTs if the privilege can be obtained, otherwise it will try to establish a TCP connection on port 7 (Echo) of the destination host.
If you want to keep a connection open continually so you can see when that fails you could connect to server running the ECHO protocol yourself rather than having isReachable() do it for you and read and write data and wait for it to fail.
You might want to try looking at the socket timeout interval. With a short timeout (I believe the default is 'infinite timeout') then you might be able to trap an exception or something when the host becomes unreachable.
Okay so I finally got it working with
try
{
Socket s = new Socket("stackoverflow.com",80);
DataOutputStream os = new DataOutputStream(s.getOutputStream());
DataInputStream is = new DataInputStream(s.getInputStream());
while (true)
{
os.writeBytes("GET /index.html HTTP/1.0\n\n");
is.available();
Thread.sleep(1000);
}
}
catch (IOException e)
{
System.out.println("connection probably lost");
e.printStackTrace();
}
Not as clean as I hoped but it's not working if I leave out the os.writeBytes().
You could ping a machine every number of seconds, and this would be pretty accurate. Be careful that you don't DOS it.
Another alternative would be run a small server on a remote machine and keep a connection to it.
Its probably simpler to connect to yahoo/google or somewhere like this.
URL yahoo = new URL("http://www.yahoo.com/");
URLConnection yc = yahoo.openConnection();
int dataLen = yc.getContentLength() ;
Neil
The isConnected()method inside Socket.java class is a little misleading. It does not tell you if the socket is currently connected to a remote host (like if it is unclosed). Instead, it tells you whether the socket has ever been connected to a remote host. If the socket was able to connect to the remote host at all, this method returns true, even after that socket has been closed. To tell if a socket is currently open, you need to check that isConnected() returns true and isClosed() returns false.
For example:
boolean connected = socket.isConnected() && !socket.isClosed();
I have implemented a small HTTP-server which allows clients to connect via HTTP and stream audio-data to them.
My problem is, that in case there's currently no audio-data available, the connection seems to break, either because the client is disconnecting, or due to another reason inside Android.
I'm acting like the following way:
serverSocket = new ServerSocket(0);
Socket socket = serverSocket.accept();
socket.setKeepAlive(true);
BufferedReader in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
BufferedWriter out = new BufferedWriter(new OutputStreamWriter(socket.getOutputStream()));
out.write("HTTP/1.1 200 OK\r\n");
out.write("Content-Type: audio/wav\r\n");
out.write("Accept-Ranges: none\r\n");
out.write("Connection: keep-alive\r\n"); // additionally added due to answer below
out.write("\r\n");
out.flush();
..
while(len=otherInput.read(audioBuffer)){
out.write(audioBuffer, 0, len);)
}
For sure this is just a snipped of the real code, but it shows what I'm doing.
Now, in case the "otherinput.read()" takes a long time because there's no data available at the moment, I get a
java.net.SocketException: sendto failed: EPIPE (Broken pipe)
at libcore.io.IoBridge.maybeThrowAfterSendto(IoBridge.java:499)
at libcore.io.IoBridge.sendto(IoBridge.java:468)
at java.net.PlainSocketImpl.write(PlainSocketImpl.java:508)
at java.net.PlainSocketImpl.access$100(PlainSocketImpl.java:46)
at java.net.PlainSocketImpl$PlainSocketOutputStream.write(PlainSocketImpl.java:270)
at java.io.BufferedOutputStream.flushInternal(BufferedOutputStream.java:185)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:139)
Who can tell me how I can prevent the connection from breaking/closing without a manual heartbeat? Do I miss some header or am I using something the wrong way?
Thanks for your help in advance, tried and searched myself crazy meanwhile.
There are at least two problems here.
Clients of HTTP servers are not well-behaved in the way you seem to expect. Consider a browser. The user can shut it down, go back, navigate away etc, any time he likes, even in the middle of a page load. If you get any error transmitting to the client there's nothing you can do except close the connection and forget about it. Same applies to any server really, but it applies unsolder to HTTP servers.
You're not reading the entire request sent by the client. You need to read all the headers until a blank line, then you need to read the body up to the length specified in the Content-length: header, or all the chunks, or until end of stream, as the case may be: see RFC 2616. The effect of this may be that you cause the behaviour at (1).
My server uses data from an internal web service to construct its response, on a per request basis. I'm using Apache HttpClient 4.1 to make the requests. Each initial request will result in about 30 requests to the web service. Of these, 4 - 8 will end up with sockets stuck in CLOSE_WAIT, which never get released. Eventually these stuck sockets exceed my ulimit and my process runs out of file descriptors.
I don't want to just raise my ulimit (1024), because that will just mask the problem.
The reason I've moved to HttpClient is that java.net.HttpUrlConnection was behaving the same way.
I have tried moving to a SingleClientConnManager per request, and calling client.getConnectionManager().shutdown() on it, but sockets still end up stuck.
Should I be trying to solve this so that I end up with 0 open sockets while there are no running requests, or should I be concentrating on request persistence and pooling?
For clarity I'm including some details which may be relevant:
OS: Ubuntu 10.10
JRE: 1.6.0_22
Language: Scala 2.8
Sample code:
val cleaner = Executors.newScheduledThreadPool(1)
private val client = {
val ssl_ctx = SSLContext.getInstance("TLS")
val managers = Array[TrustManager](TrustingTrustManager)
ssl_ctx.init(null, managers, new java.security.SecureRandom())
val sslSf = new org.apache.http.conn.ssl.SSLSocketFactory(ssl_ctx, SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER)
val schemeRegistry = new SchemeRegistry()
schemeRegistry.register(new Scheme("https", 443, sslSf))
val connection = new ThreadSafeClientConnManager(schemeRegistry)
object clean extends Runnable{
override def run = {
connection.closeExpiredConnections
connection.closeIdleConnections(30, SECONDS)
}
}
cleaner.scheduleAtFixedRate(clean,10,10,SECONDS)
val httpClient = new DefaultHttpClient(connection)
httpClient.getCredentialsProvider().setCredentials(new AuthScope(AuthScope.ANY), new UsernamePasswordCredentials(username,password))
httpClient
}
val get = new HttpGet(uri)
val entity = client.execute(get).getEntity
val stream = entity.getContent
val justForTheExample = IOUtils.toString(stream)
stream.close()
Test: netstat -a | grep {myInternalWebServiceName} | grep CLOSE_WAIT
(Lists sockets for my process that are in CLOSE_WAIT state)
Post comment discussion:
This code now demonstrates correct usage.
One needs to pro-actively evict expired / idle connections from the connection pool, as in the blocking I/O model connections cannot react to I/O events unless they are being read from / written to. For details see
http://hc.apache.org/httpcomponents-client-dev/tutorial/html/connmgmt.html#d4e631
I've marked oleg's answer as correct, as it highlights an important usage point about HttpClient's connection pooling.
To answer my specific original question, though, which was "Should I be trying to solve for 0 unused sockets or trying to maximize pooling?"
Now that the pooling solution is in place and working correctly the application throughput has increased by about 150%. I attribute this to not having to renegotiate SSL and multiple handshakes, instead reusing persistent connections in accordance with HTTP 1.1.
It is definitely worth working to utilize pooling as intended, rather than trying to hack around with calling ThreadSafeClientConnManager.shutdown() after each request etcetera. If, on the other hand, you were calling arbitrary hosts and not reusing routes the way I am you might easily find that it becomes necessary to do that sort of hackery, as the JVM might surprise you with the long life of CLOSE_WAIT designated sockets if you're not garbage collecting very often.
I had the same issue and solved it using the suggesting found here: here. The author touches on some TCP basics:
When a TCP connection is about to close, its finalization is negotiated by both parties. Think of it as breaking a contract in a civilized manner. Both parties sign the paper and it’s all good. In geek talk, this is done via the FIN/ACK messages. Party A sends a FIN message to indicate it wants to close the socket. Party B sends an ACK saying it received the message and is considering the demand. Party B then cleans up and sends a FIN to Party A. Party A responds with the ACK and everyone walks away.
The problem comes in
when B doesn’t send its FIN. A is kinda stuck waiting for it. It has
initiated its finalization sequence and is waiting for the other party
to do the same.
He then mentions RFC 2616, 14.10 to suggest setting up an http header to solve this issue:
postMethod.addHeader("Connection", "close");
Honestly, I don't really know the implications of setting this header. But it did stop CLOSE_WAIT from happening on my unit tests.
This question already has answers here:
Java socket API: How to tell if a connection has been closed?
(9 answers)
Closed 5 years ago.
When I'm using e.g. PuTTY and my connection gets lost (or when I do a manual ipconfig /release on Windows), it responds directly and notifies my connection was lost.
I want to create a Java program which monitors my Internet connection (to some reliable server), to log the date/times when my internet fails.
I tried use the Socket.isConnected() method but that will just forever return "true". How can I do this in Java?
Well, the best way to tell if your connection is interrupted is to try to read/write from the socket. If the operation fails, then you have lost your connection sometime.
So, all you need to do is to try reading at some interval, and if the read fails try reconnecting.
The important events for you will be when a read fails - you lost connection, and when a new socket is connected - you regained connection.
That way you can keep track of up time and down time.
Even though TCP/IP is "connection oriented" protocol, normally no data is sent over an idle connection. You can have a socket open for a year without a single bit sent over it by the IP stack. In order to notice that a connection is lost, you have to send some data on the application level.(*) You can try this out by unplugging the phone cable from your ADSL modem. All connections in your PC should stay up, unless the applications have some kind of application level keepalive mechanism.
So the only way to notice lost connection is to open TCP connection to some server and read some data from it. Maybe the most simple way could be to connect to some FTP server and fetch a small file - or directory listing - once in a while. I have never seen a generic server which was really meant to be used for this case, and owners of the FTP server may not like clients doing this.
(*) There is also a mechanism called TCP keepalive but in many OS's you have to activate it for all applications, and it is not really practical to use if you want to notice loss of connection quickly
If the client disconnects properly, a read() will return -1, readLine() returns null, readXXX() for any other X throws EOFException. The only reliable way to detect a lost TCP connection is to write to it. Eventually this will throw an IOException 'connection reset', but it takes at least two writes due to buffering.
Why not use the isReachable() method of the java.net.InetAddress class?
How this works is JVM implementation specific but:
A typical implementation will use ICMP ECHO REQUESTs if the privilege can be obtained, otherwise it will try to establish a TCP connection on port 7 (Echo) of the destination host.
If you want to keep a connection open continually so you can see when that fails you could connect to server running the ECHO protocol yourself rather than having isReachable() do it for you and read and write data and wait for it to fail.
You might want to try looking at the socket timeout interval. With a short timeout (I believe the default is 'infinite timeout') then you might be able to trap an exception or something when the host becomes unreachable.
Okay so I finally got it working with
try
{
Socket s = new Socket("stackoverflow.com",80);
DataOutputStream os = new DataOutputStream(s.getOutputStream());
DataInputStream is = new DataInputStream(s.getInputStream());
while (true)
{
os.writeBytes("GET /index.html HTTP/1.0\n\n");
is.available();
Thread.sleep(1000);
}
}
catch (IOException e)
{
System.out.println("connection probably lost");
e.printStackTrace();
}
Not as clean as I hoped but it's not working if I leave out the os.writeBytes().
You could ping a machine every number of seconds, and this would be pretty accurate. Be careful that you don't DOS it.
Another alternative would be run a small server on a remote machine and keep a connection to it.
Its probably simpler to connect to yahoo/google or somewhere like this.
URL yahoo = new URL("http://www.yahoo.com/");
URLConnection yc = yahoo.openConnection();
int dataLen = yc.getContentLength() ;
Neil
The isConnected()method inside Socket.java class is a little misleading. It does not tell you if the socket is currently connected to a remote host (like if it is unclosed). Instead, it tells you whether the socket has ever been connected to a remote host. If the socket was able to connect to the remote host at all, this method returns true, even after that socket has been closed. To tell if a socket is currently open, you need to check that isConnected() returns true and isClosed() returns false.
For example:
boolean connected = socket.isConnected() && !socket.isClosed();