We Keep Coding

Connection dropped by client when serving large files for download (Java, Jersey, HTTP, GET)

I have a HTTP server which serves files in download, and some of these files are pretty large (can be 7 GB or more). When these files are downloaded from some networks, the connection is dropped and we find the following error in the tomcat catalina log:
org.apache.catalina.connector.ClientAbortException: java.io.IOException: Connection reset by peer
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:393)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:426)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:339)
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:418)
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:406)
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:97)
at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:229)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2147)
...
SEVERE: org.glassfish.jersey.server.ServerRuntime$Responder: An I/O error has occurred while writing a response message entity to the container output stream.
org.glassfish.jersey.server.internal.process.MappableException: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
...
(let me know if you need more log lines)
We believe these errors are due to some firewall/proxy/NAT configuration: for some networks we have also determined a threshold over which the connection drops deterministically). We need to find a way to overcome these issues without asking the client to change their configurations (it can be done, as the same clients can download the same file via HTTP from Dropbox). Also, clients must be authenticated (i.e., have a valid session).
Here is my latest code:
#GET
#Path("download")
#Produces(MediaType.APPLICATION_OCTET_STREAM)
public Response download(#HeaderParam("x-session-token") String sSessionId, #QueryParam("filename") String sFileName)
{
User oUser = MyLib.GetUserFromSession(sSessionId);
if (oUser == null) {
return Response.status(Status.UNAUTHORIZED).build();
}
File oFile = getEntryFile(sFileName);
ResponseBuilder oResponseBuilder = null;
if(oFile == null) {
oResponseBuilder = Response.serverError();
} else {
FileStreamingOutput oStream = new FileStreamingOutput(oFile);
oResponseBuilder = Response.ok(oStream);
oResponseBuilder.header("Content-Disposition", "attachment; filename="+ oFile.getName());
}
return oResponseBuilder.build();
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
FileStreamingOutput is an extension of javax.ws.rs.core.StreamingOutput:
public class FileStreamingOutput implements StreamingOutput {
final File m_oFile;
public FileStreamingOutput(File oFile){
if(null==oFile) {
throw new NullPointerException("FileStreamingOutput.FileStreamingOutput: passed a null File");
}
m_oFile = oFile;
}
#Override
public void write(OutputStream oOutputStream) throws IOException, WebApplicationException {
if(null == oOutputStream) {
throw new NullPointerException("FileStreamingOutput.write: passed a null OutputStream");
}
InputStream oInputStream = null;
try {
oInputStream = new FileInputStream(m_oFile);
long lThreshold = 2L*1024*1024*1024;
long lSize = m_oFile.length();
if(lSize > lThreshold) {
IOUtils.copyLarge(oInputStream, oOutputStream);
} else {
IOUtils.copy(oInputStream, oOutputStream);
}
} catch (Exception e) {
e.printStackTrace();
} finally {
if( oOutputStream!=null ) {
oOutputStream.flush();
oOutputStream.close();
}
if( oInputStream !=null ) {
oInputStream.close();
}
}
}
}
Before using StreamingOutput I tried building the response with an OutputStream or with a File directly. Same result.
Additional requirements: we cannot introduce other frameworks (e.g. Spring...)
Any ideas on how to overcome this limitation?

I solved the problem simply (!) by specifying the size of the file to be transferred in the Content-Length header, i.e., by adding the following line:
oResponseBuilder.header("Content-Length", oFile.length());

Code not executed when trying to connecting to WSDL webservice

When trying to use a webservice, a NullPointerException is thrown, and expected log statements are missing in the application's log.
I get a NullPointerException in the following code:
public void useWebservice() {
initEndpoint();
try {
port.usefulFunctionWebserviceProvides(); // NullPointerException is thrown here!
} catch (javax.xml.ws.soap.SOAPFaultException ex) {
log.error("Something went wrong making a request to the webservice");
}
}
The initEndpoint method looks like this:
private volatile Webservice service = null; // instance variable
private WebservicePort port = null; // instance variable
private void initEndpoint() {
String username = "username"; // Loaded from a properties file
String password = "password"; // Loaded from a properties file
LoginResponse loginResponse;
Webservice theService = service;
if (theService == null || port == null) {
synchronized (this) {
theService = service;
if (theService == null) {
try {
log.info("Initializing Endpoint (service & port)"); // This line appears in the log
service = new Webservice();
port = service.getWebservicePort();
final String wsdlUrl = properties.getProperty(WSDL_URL, WSDL_DEFAULT_URL);
((BindingProvider)port).getRequestContext().put(BindingProvider.ENDPOINT_ADDRESS_PROPERTY, wsdlUrl);
log.info("EndpointAddress set");
LoginRequestType loginRequest = new LoginRequestType();
loginRequest.setUsername(username);
loginRequest.setPassword(password);
loginResponse = port.login(loginRequest, null, null);
} catch (Exception e) {
try {
log.info("re-Initializing Endpoint (service & port)");
// Try to connect to the webservice using a fallback URL
} catch (Exception e2) {
log.error("Couldn't connect to webservice");
service = null;
throw new CustomException();
}
}
if (loginResponse == null) {
service = null;
throw new CustomException();
}
}
}
}
}
None of the info logging is printed to the log file except for the log message "Initializing Endpoint (service & port)".
I don't understand how the port variable can be null in the useWebservice method. I also expect to see either "EndpointAddress set" or "re-Initializing Endpoint (service & port)" in the log file. But neither are printed to the file.
The code has worked fine before, but has started to give problems after being moved to a new project and used as a dependency.

The problem has been identified, it was a missing dependency. Because of that an error was thrown when the code tried to initialize the port. The catch block only catches exceptions so that code was never executed. If we caught throwable for example we would see the log statements from the catch block.
PS: Initializing the port object as it is done in the initEndpoint method in the question is NOT thread safe! The port needs to be initialized for each request.

Getting Error java.lang.NoClassDefFoundError: sun/net/www/protocol/https/HttpsURLConnectionImpl when deployed in Jboss EAP 6.4.x

I am writing a rest client using java.net which should do a PATCH request. But as PATCH is not a supported method in java.net, I used reflection to make it supported by changing the code like
private void updateConnectionToSupportPatchRequest(final HttpURLConnection conn)
throws ReflectiveOperationException {
try {
final Object targetConn;
if (conn instanceof HttpsURLConnectionImpl) {
final Field delegateField = HttpsURLConnectionImpl.class.getDeclaredField("delegate");
delegateField.setAccessible(true);
targetConn = delegateField.get(conn);
} else {
targetConn = conn;
}
final Field methodField = HttpURLConnection.class.getDeclaredField("method");
methodField.setAccessible(true);
methodField.set(targetConn, "PATCH");
} catch (final NoSuchFieldException ex) {
LOGGER.error("NoSuchFieldException: {} ", ex.getMessage());
}
}
but when I deploy my application which uses my rest client in JBoss, I get this error -
java.lang.NoClassDefFoundError: sun/net/www/protocol/https/HttpsURLConnectionImpl
I looked up on this error and came across this post http://planet.jboss.org/post/dealing_with_sun_jdk_related_noclassdeffounderror_under_jboss
I tried the suggested solution in the post still getting the same error. Any ideas on how to get passed this issue?
P.S. I cannot use the Apache HttpClient or RestEasy(Jboss) as there is another 3PP being used in the project which does not support Apache HttpClient

Have you tried using the workaround X-HTTP-Method-Override before trying to fiddle with internal classes of the JDK? If that's the case, you can use the instance's getClass-method to access fields and use isAssignableFrom as alternative to instanceof.
Another approach to get rid off specifying concrete classes is just trying to get the field in HttpsURLConnection and assuming a non-Https-URLConnection if the field can't be found. This might look like the following code:
private void updateConnectionToSupportPatchRequest(final HttpURLConnection conn)
throws ReflectiveOperationException {
try {
final Object targetConn = conn;
try {
final Field delegateField = findField(conn.getClass(), "delegate");
delegateField.setAccessible(true);
targetConn = delegateField.get(conn);
}
catch(NoSuchFieldException nsfe) {
// no HttpsURLConnection
}
final Field methodField = findField(conn.getClass(), "method");
methodField.setAccessible(true);
methodField.set(targetConn, "PATCH");
} catch (final NoSuchFieldException ex) {
LOGGER.error("NoSuchFieldException: {} ", ex.getMessage());
}
}
private Field findField(Class clazz, String name) throws NoSuchFieldException {
while (clazz != null) {
try {
return clazz.getDeclaredField(name);
}
catch(NoSuchFieldException nsfe) {
// ignore
}
clazz = clazz.getSuperclass();
}
throw new NoSuchFieldException(name);
}
But this might fail at another level because - obviously - the class that is used within JBoss is not the one you implemented the workaround, so fields and methods might be named differently.

JUnit test on URLConnection, use EasyMock?

Hey, have been trying to work this out for last day or so but hitting brick wall. Trying to unit test this bit of code. But not sure if need to use EasyMock or not?? Seem few examples online but seem to be using older techniques.
public boolean verifyConnection(final String url) {
boolean result;
final int timeout = getConnectionTimeout();
if (timeout < 0) {
log.info("No need to verify connection to client. Supplied timeout = {}", timeout);
result = true;
} else {
try {
log.debug("URL: {} Timeout: {} ", url, timeout);
final URL targetUrl = new URL(url);
final HttpURLConnection connection = (HttpURLConnection) targetUrl.openConnection();
connection.setConnectTimeout(timeout);
connection.connect();
result = true;
} catch (ConnectException e) {
log.warn("Could not connect to client supplied url: " + url, e);
result = false;
} catch (MalformedURLException e) {
log.error("Malformed client supplied url: " + url, e);
result = false;
} catch (IOException e) {
log.warn("Could not connect to client supplied url: " + url, e);
result = false;
}
}
return result;
}
It just take's in a url checks its valid and returns T or F.

I have always observed that Mocking Can be avoided as much as possible because it can lead to difficult to maintain JUnit tests and defeat the whole purpose.
My suggestion would be to create a temporary server on your local machine from a JUnit itself.
At the beginning of JUnit you can create a server(not more than 10-15 lines of coding required) using Java sockets and then in your code pass the URL for the local server. This way you are reducing mocking and ensuring maximum code coverage.
Something like this -
public class SimpleServer extends Thread {
public void run() {
try {
serverSocket = new ServerSocket(port);
while (true) {
Socket s = serverSocket.accept();
}
}
catch (IOException e) {
e.printStackTrace();
}
finally {
serverSocket = null;
}
}
}

If you want to mock this method, I'd recommend passing in the URL rather than the String. Don't have your method create the URL it needs; let the client create the URL for you and pass it in. That way your test can substitute a mock if it needs to.
It's almost a dependency injection idea - your method should be given its dependencies and not create them on its own. The call to "new" is the dead giveaway.
It's not a drastic change. You could overload the method and have two signatures: one that accepts a URL string and another that accepts the URL itself. Have the first method create the URL and call the second. That way you can test it and still have the method with the String signature in your API for convenience.

Trying to set up mock implementation of the HttpURLConnection. Like
public class MockHttpURLConnection extends HttpURLConnection {'
then added method to class to override
' protected HttpURLConnection createHttpURLConnection(URL url)
throws IOException {
return (HttpURLConnection) url.openConnection();
}
So test looking something like this:
#Test
public void testGetContentOk() throws Exception
{
String url = "http://localhost";
MockHttpURLConnection mockConnection = new MockHttpURLConnection();
TestableWebClient client = new TestableWebClient();
client.setHttpURLConnection(mockConnection);
boolean result = client.verify(url);
assertEquals(true, result);
}
#Test
public void testDoesNotGetContentOk() throws Exception
{
String url = "http://1.2.3.4";
MockHttpURLConnection mockConnection = new MockHttpURLConnection();
TestableWebClient client = new TestableWebClient();
client.setHttpURLConnection(mockConnection);
boolean result = client.verify(url);
assertEquals(false, result);
}
/**
* An inner, private class that extends WebClient and allows us
* to override the createHttpURLConnection method.
*/
private class TestableWebClient extends WebClient1 {
private HttpURLConnection connection;
/**
* Setter method for the HttpURLConnection.
*
* #param connection
*/
public void setHttpURLConnection(HttpURLConnection connection)
{
this.connection = connection;
}
/**
* A method that we overwrite to create the URL connection.
*/
#Override
public HttpURLConnection createHttpURLConnection(URL url) throws IOException
{
return this.connection;
}
}
First part passed but is getting true for false dummy test, thanks for feedback back so far best site I have found for help. So let me know if think on right track

problem with passing numeric parameter to web service from java client

i try to passing numeric parameter to a web service that receive the value and return it back.
this is the snippet of the web method :
#WebMethod(operationName = "getNumber")
public Integer getNumber(#WebParam(name = "i")
Integer i) {
//TODO write your implementation code here:
System.out.println("number : "+i);
return i;
}
an this is the snippet of my client code :
Map results = FastMap.newInstance();
results.put("result", "success");
String endPoint = "http://localhost:8084/ProvideWS/MathWS";
URL endpoint=null;
try{
endpoint = new URL(endPoint);
}
catch (MalformedURLException e) {
org.ofbiz.base.util.Debug.log("Location not a valid URL "+e);
// TODO: handle exception
}
Service service = null;
Call call = null;
try{
service = new Service();
call = (Call)service.createCall();
call.setTargetEndpointAddress(endpoint);
String nameSpace = "http://ws/";
String serviceName = "getNumber";
call.setOperationName(new QName(nameSpace, serviceName));
call.addParameter("i",org.apache.axis.Constants.XSD_INTEGER , ParameterMode.IN);
call.setReturnType(org.apache.axis.Constants.XSD_INTEGER);
Object msg[] = new Object[]{new Integer(5)};
for (Object o : msg) {
org.ofbiz.base.util.Debug.log("object to be sent===== "+o.toString());
}
Object ret = call.invoke(msg);
results.put("result", "result : "+ ret.toString());
}
catch (Exception e) {
// TODO: handle exception
e.printStackTrace();
org.ofbiz.base.util.Debug.log("exc when running soap client test : "+e);
results.put("result", "error : "+e);
}
return results;
the problem is the return value in the client always 0 (the server received the number as zero), the method i used to pass the parameter works fine when the paramater is String. I.ve tried to hard-coding the return value in server and the output in client is fine, so i thought it must be how the server retrieved the parameter is the problem.
do you have any idea why this is happen and how to solve this?
any help will be appreciated, thanks

I don't know what is causing your problem. But the first thing I would do would be to try to capture the actual request that is being sent to the server. That should give you some clues as to whether the root problem is on the client or server side.