We Keep Coding

Socket hangs when trying to read input stream from web browser Java

I am attempting to read an input stream from a socket provided by a web browser client. Every approach I have taken has got the same results thus far, it just hangs and I don't know why. I have tried mark() marking the read limit to what is available and still no go.
inputStream.mark(inputStream.available());
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
byte[] buffer = new byte[1024 * 9];
int read;
while((read = inputStream.read(buffer)) > 0) {
outputStream.write(buffer, 0, read);
}
byte[] bytes = outputStream.toByteArray();
I have also tried clientSocket.shutdownInput() to tried to fix this issue, still no good.
Here is my attempt below:
import java.io.*;
import java.net.InetAddress;
import java.net.ServerSocket;
import java.net.Socket;
import java.nio.charset.StandardCharsets;
public class Server {
public static void main(String[] args)
{
ServerSocket server = null;
try {
// Server is listening on port 3001
server = new ServerSocket(3001, 1, InetAddress.getByName("localhost"));
server.setReuseAddress(true);
// running infinite loop for getting
// client request
while (true) {
// socket object to receive incoming client
// requests
Socket client = server.accept();
// Displaying that new client is connected
// to Server
System.out.println("New client connected"
+ client.getInetAddress()
.getHostAddress());
// create a new thread object
ClientHandler clientSock
= new ClientHandler(client);
// This thread will handle the client
// separately
new Thread(clientSock).start();
}
}catch (IOException e) {
e.printStackTrace();
}
}
// ClientHandler class
private static class ClientHandler implements Runnable {
private final Socket clientSocket;
// Constructor
public ClientHandler(Socket clientSocket)
{
this.clientSocket = clientSocket;
}
public void run() {
InputStream inputStream = null;
OutputStream clientOutput = null;
try {
inputStream = clientSocket.getInputStream();
inputStream.mark(inputStream.available());
clientSocket.shutdownInput();
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
byte[] buffer = new byte[1024 * 9];
int numRead;
while((numRead = inputStream.read(buffer)) > 0) {
outputStream.write(buffer, 0, numRead);
}
byte[] bytes = outputStream.toByteArray();
String payloadString = new String(bytes, StandardCharsets.UTF_8);
System.out.println(payloadString);
clientOutput = clientSocket.getOutputStream();
clientOutput.write(("HTTP/1.1 \r\n" + "200 OK").getBytes());
clientOutput.write(("ContentType: " + "text/html" + "\r\n").getBytes());
clientOutput.write("\r\n".getBytes());
clientOutput.write("Hello World!".getBytes());
clientOutput.write("\r\n\r\n".getBytes());
clientOutput.flush();
inputStream.close();
clientOutput.close();
try{
clientSocket.close();
}catch(Exception ex){
ex.printStackTrace();
}
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
Any help would be most appreciated! Thank you.

browsers suggests you should toss this all in the garbage and use HTTP, because, well, browsers.
But, if you insist, there are two problems here.
You've made it crazy complicated.
You can take all of that code and replace it all with this simple little line:
byte[] bytes = in.readAllBytes();
That replaces the lines starting with in.mark(in.available()) (this line does nothing at all, I have no idea where you got this from. If you think it is supposed to do something specific, you might want to mention that. Because it doesn't do anything. mark is useful if you ever reset, which you aren't, and you don't need to here, hence, useless), all the way to `byte[] bytes =...;
sockets don't close unless sender goes out of its way to close it
Your read code (yours, or the much simpler one-liner above) reads everything until the stream closes. In your second snippet, you close it right away, which obviously doesn't work. You cannot know when to close it, the sender does this job. Evidently it's not doing it.
I advise you to adapt protocols that pre-roll sizes, so you know how much to read and aren't dependent on closing the socket just to signal that the data is sent.
For example:
byte[] sizeRaw = in.readNBytes(4);
int size = ByteBuffer.wrap(bytes).getInt();
byte[] bytes = in.readNBytes(size);
You will of course have to adjust the sending code to send the size first (as a 32-bit value, big endian). One way or another you have to look at the sending code here. Either fix it so that it closes once done, or, better yet, adjust it so it sends size first.

Download zip returns SocketException: Connection reset

There is a billion of topics about Connection, I've tried so many ways to download this file and always fail. When I disable cookies on my web browser I can't download it, for that reason I believe my problem is with cookies.
The function of my program is extract the zip, parse the html inside, with Jsoup, insert the content on mysql database and load it on JApplet. Everything is working except the auto-download part, which I have to do manual download in my web browser.
I'm using this class for the cookie, which returns error on
read.CookieManager.storeCookies(CookieManager.java:89)
which corresponds to this line from Cookie class
for (int i=1; (headerName = conn.getHeaderFieldKey(i)) != null; i++) {
and this one from download class
cm.storeCookies(urlConnection);
the download method
public static void main(String args[]) throws Exception {
downloadFromUrl("http://www1.caixa.gov.br/loterias/_arquivos/loterias/D_mgsasc.zip", "Mozilla", "C:/", "D_mgsasc.zip", true);
}
public static void downloadFromUrl(String srcAddress, String userAgent, String destDir, String destFileName, boolean overwrite) throws Exception
{
InputStream is = null;
FileOutputStream fos = null;
try {
CookieManager cm = new CookieManager();
URL url = new URL(srcAddress);
URLConnection urlConnection = url.openConnection();
urlConnection.setRequestProperty("User-Agent", userAgent);
urlConnection.setDoInput(true);
urlConnection.setDoOutput(true);
urlConnection.setConnectTimeout(30000);
urlConnection.setReadTimeout(30000);
urlConnection.setUseCaches(true);
urlConnection.connect();
cm.storeCookies(urlConnection);
cm.setCookies(url.openConnection());
is = urlConnection.getInputStream();
fos = new FileOutputStream(destFile);
byte[] buffer = new byte[1024];
int len, totBytes = 0;
while((len = is.read(buffer)) > 0)
{
totBytes += len;
fos.write(buffer, 0, len);
}
fos.flush();
fos.close();
}
}
*
updated, removed unnecessary code
*
which returns the following error
java.net.SocketException: Connection reset at
zip.DownloadFile.downloadFromUrl(DownloadFile.java:71)
related to this line in code
is = urlConnection.getInputStream();
When I remove cookies set code, the same error of Connection reset persists.

Resumable upload from Java client to Grails web application?

After almost 2 workdays of Googling and trying several different possibilities I found throughout the web, I'm asking this question here, hoping that I might finally get an answer.
First of all, here's what I want to do:
I'm developing a client and a server application with the purpose of exchanging a lot of large files between multiple clients on a single server. The client is developed in pure Java (JDK 1.6), while the web application is done in Grails (2.0.0).
As the purpose of the client is to allow users to exchange a lot of large files (usually about 2GB each), I have to implement it in a way, so that the uploads are resumable, i.e. the users are able to stop and resume uploads at any time.
Here's what I did so far:
I actually managed to do what I wanted to do and stream large files to the server while still being able to pause and resume uploads using raw sockets. I would send a regular request to the server (using Apache's HttpClient library) to get the server to send me a port that was free for me to use, then open a ServerSocket on the server and connect to that particular socket from the client.
Here's the problem with that:
Actually, there are at least two problems with that:
I open those ports myself, so I have to manage open and used ports myself. This is quite error-prone.
I actually circumvent Grails' ability to manage a huge amount of (concurrent) connections.
Finally, here's what I'm supposed to do now and the problem:
As the problems I mentioned above are unacceptable, I am now supposed to use Java's URLConnection/HttpURLConnection classes, while still sticking to Grails.
Connecting to the server and sending simple requests is no problem at all, everything worked fine. The problems started when I tried to use the streams (the connection's OutputStream in the client and the request's InputStream in the server). Opening the client's OutputStream and writing data to it is as easy as it gets. But reading from the request's InputStream seems impossible to me, as that stream is always empty, as it seems.
Example Code
Here's an example of the server side (Groovy controller):
def test() {
InputStream inStream = request.inputStream
if(inStream != null) {
int read = 0;
byte[] buffer = new byte[4096];
long total = 0;
println "Start reading"
while((read = inStream.read(buffer)) != -1) {
println "Read " + read + " bytes from input stream buffer" //<-- this is NEVER called
}
println "Reading finished"
println "Read a total of " + total + " bytes" // <-- 'total' will always be 0 (zero)
} else {
println "Input Stream is null" // <-- This is NEVER called
}
}
This is what I did on the client side (Java class):
public void connect() {
final URL url = new URL("myserveraddress");
final byte[] message = "someMessage".getBytes(); // Any byte[] - will be a file one day
HttpURLConnection connection = url.openConnection();
connection.setRequestMethod("GET"); // other methods - same result
// Write message
DataOutputStream out = new DataOutputStream(connection.getOutputStream());
out.writeBytes(message);
out.flush();
out.close();
// Actually connect
connection.connect(); // is this placed correctly?
// Get response
BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String line = null;
while((line = in.readLine()) != null) {
System.out.println(line); // Prints the whole server response as expected
}
in.close();
}
As I mentioned, the problem is that request.inputStream always yields an empty InputStream, so I am never able to read anything from it (of course). But as that is exactly what I'm trying to do (so I can stream the file to be uploaded to the server, read from the InputStream and save it to a file), this is rather disappointing.
I tried different HTTP methods, different data payloads, and also rearranged the code over and over again, but did not seem to be able to solve the problem.
What I hope to find
I hope to find a solution to my problem, of course. Anything is highly appreciated: hints, code snippets, library suggestions and so on. Maybe I'm even having it all wrong and need to go in a totally different direction.
So, how can I implement resumable file uploads for rather large (binary) files from a Java client to a Grails web application without manually opening ports on the server side?

HTTP GET method have special headers for range retrieval: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35 It's used by most downloaders to do resumable download from server.
As I understand, there are no standard practice for using this headers for POST/PUT request, but it's up to you, right? You can make pretty standard Grails controller, that will accept standard http upload, with header like Range: bytes=500-999. And controller should put this 500 uploaded bytes from client into file, starting at position 500
At this case you don't need to open any socket, and make own protocols, etc.
P.S. 500 bytes is just a example, probably you're using much bigger parts.

Client Side Java Programming:
public class NonFormFileUploader {
static final String UPLOAD_URL= "http://localhost:8080/v2/mobileApp/fileUploadForEOL";
static final int BUFFER_SIZE = 4096;
public static void main(String[] args) throws IOException {
// takes file path from first program's argument
String filePath = "G:/study/GettingStartedwithGrailsFinalInfoQ.pdf";
File uploadFile = new File(filePath);
System.out.println("File to upload: " + filePath);
// creates a HTTP connection
URL url = new URL(UPLOAD_URL);
HttpURLConnection httpConn = (HttpURLConnection) url.openConnection();
httpConn.setDoOutput(true);
httpConn.setRequestMethod("POST");
// sets file name as a HTTP header
httpConn.setRequestProperty("fileName", uploadFile.getName());
// opens output stream of the HTTP connection for writing data
OutputStream outputStream = httpConn.getOutputStream();
// Opens input stream of the file for reading data
FileInputStream inputStream = new FileInputStream(uploadFile);
byte[] buffer = new byte[BUFFER_SIZE];
int bytesRead = -1;
while ((bytesRead = inputStream.read(buffer)) != -1) {
System.out.println("bytesRead:"+bytesRead);
outputStream.write(buffer, 0, bytesRead);
outputStream.flush();
}
System.out.println("Data was written.");
outputStream.flush();
outputStream.close();
inputStream.close();
int responseCode = httpConn.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_OK) {
// reads server's response
BufferedReader reader = new BufferedReader(new InputStreamReader(
httpConn.getInputStream()));
String response = reader.readLine();
System.out.println("Server's response: " + response);
} else {
System.out.println("Server returned non-OK code: " + responseCode);
}
}
}
Server Side Grails Programme:
Inside the controller:
def fileUploadForEOL(){
def result
try{
result = mobileAppService.fileUploadForEOL(request);
}catch(Exception e){
log.error "Exception in fileUploadForEOL service",e
}
render result as JSON
}
Inside the Service Class:
def fileUploadForEOL(request){
def status = false;
int code = 500
def map = [:]
try{
String fileName = request.getHeader("fileName");
File saveFile = new File(SAVE_DIR + fileName);
System.out.println("===== Begin headers =====");
Enumeration<String> names = request.getHeaderNames();
while (names.hasMoreElements()) {
String headerName = names.nextElement();
System.out.println(headerName + " = " + request.getHeader(headerName));
}
System.out.println("===== End headers =====\n");
// opens input stream of the request for reading data
InputStream inputStream = request.getInputStream();
// opens an output stream for writing file
FileOutputStream outputStream = new FileOutputStream(saveFile);
byte[] buffer = new byte[BUFFER_SIZE];
int bytesRead = inputStream.read(buffer);
long count = bytesRead
while(bytesRead != -1) {
outputStream.write(buffer, 0, bytesRead);
bytesRead = inputStream.read(buffer);
count += bytesRead
}
println "count:"+count
System.out.println("Data received.");
outputStream.close();
inputStream.close();
System.out.println("File written to: " + saveFile.getAbsolutePath());
code = 200
}catch(Exception e){
mLogger.log(java.util.logging.Level.SEVERE,"Exception in fileUploadForEOL",e);
}finally{
map <<["code":code]
}
return map
}
I have tried with above code it is worked for me(only for file size 3 to 4MB, but for small size files some bytes of code missing or not even coming but in request header content-length is coming, not sure why it is happening.)

TCP connection is not reused for HTTP requests with HttpURLConnection

I'm have created an application which sends GET requests to a URL, and then downloads the full content of that page.
The client sends a GET to e.g. stackoverflow.com, and forwards the response to a parser, which has the resposibility to find all the sources from the page that needs to be downloaded with subsequent GET requests.
The method below is used to send those GET requests. It is called many times consecutively, with the URLs returned by the parser. Most of those URLs are located on the same host, and should be able to share the TCP connection.
public static void sendGetRequestToSubObject(String RecUrl)
{
URL url = new URL(recUrl.toString());
URLConnection connection = url.openConnection ();
InputStreamReader isr = new InputStreamReader(connection.getInputStream());
}
Each time this method is called, a new TCP connection is created (with a TCP 3-way handshake) and the GET is then sent on that connection. But I want to reuse the TCP connections, to improve performance.
I guess that since I create a new URL object each time the method is called, this is the way it going to work...
Maybe someone can help me do this in a better way?
Thanks!

HttpURLConnection will reuse connections if it can!
For this to work, several preconditions need to be fulfilled, mostly on the server side. Those preconditions are described in the article linked to above.

Found the problem! I was not reading the input stream properly. This caused the input stream objects to hang, and they could not be reused.
I only defined it, like this:
InputStreamReader isr = new InputStreamReader(connection.getInputStream());
but I never read from it :-)
I changed the read method as well. Instead of a buffered reader I stole this:
InputStream in = null;
String queryResult = "";
try {
URL url = new URL(archiveQuery);
HttpURLConnection urlConn = (HttpURLConnection) url.openConnection();
HttpURLConnection httpConn = (HttpURLConnection) urlConn;
httpConn.setAllowUserInteraction(false);
httpConn.connect();
in = httpConn.getInputStream();
BufferedInputStream bis = new BufferedInputStream(in);
ByteArrayBuffer baf = new ByteArrayBuffer(50);
int read = 0;
int bufSize = 512;
byte[] buffer = new byte[bufSize];
while(true){
read = bis.read(buffer);
if(read==-1){
break;
}
baf.append(buffer, 0, read);
}
queryResult = new String(baf.toByteArray());
} catch (MalformedURLException e) {
// DEBUG
Log.e("DEBUG: ", e.toString());
} catch (IOException e) {
// DEBUG
Log.e("DEBUG: ", e.toString());
}
}
From here: Reading HttpURLConnection InputStream - manual buffer or BufferedInputStream?

HttpURLConnection.getResponseCode() returns -1 on second invocation

I seem to be running into a peculiar problem on Android 1.5 when a library I'm using (signpost 1.1-SNAPSHOT), makes two consecutive connections to a remote server. The second connection always fails with a HttpURLConnection.getResponseCode() of -1
Here's a testcase that exposes the problem:
// BROKEN
public void testDefaultOAuthConsumerAndroidBug() throws Exception {
for (int i = 0; i < 2; ++i) {
final HttpURLConnection c = (HttpURLConnection) new URL("https://api.tripit.com/oauth/request_token").openConnection();
final DefaultOAuthConsumer consumer = new DefaultOAuthConsumer(api_key, api_secret, SignatureMethod.HMAC_SHA1);
consumer.sign(c); // This line...
final InputStream is = c.getInputStream();
while( is.read() >= 0 ) ; // ... in combination with this line causes responseCode -1 for i==1 when using api.tripit.com but not mail.google.com
assertTrue(c.getResponseCode() > 0);
}
}
Basically, if I sign the request and then consume the entire input stream, the next request will fail with a resultcode of -1. The failure doesn't seem to happen if I just read one character from the input stream.
Note that this doesn't happen for any url -- just specific urls such as the one above.
Also, if I switch to using HttpClient instead of HttpURLConnection, everything works fine:
// WORKS
public void testCommonsHttpOAuthConsumerAndroidBug() throws Exception {
for (int i = 0; i < 2; ++i) {
final HttpGet c = new HttpGet("https://api.tripit.com/oauth/request_token");
final CommonsHttpOAuthConsumer consumer = new CommonsHttpOAuthConsumer(api_key, api_secret, SignatureMethod.HMAC_SHA1);
consumer.sign(c);
final HttpResponse response = new DefaultHttpClient().execute(c);
final InputStream is = response.getEntity().getContent();
while( is.read() >= 0 ) ;
assertTrue( response.getStatusLine().getStatusCode() == 200);
}
}
I've found references to what seems to be a similar problem elsewhere, but so far no solutions. If they're truly the same problem, then the problem probably isn't with signpost since the other references make no reference to it.
Any ideas?

Try set this property to see if it helps,
http.keepAlive=false
I saw similar problems when server response is not understood by UrlConnection and client/server gets out of sync.
If this solves your problem, you have to get a HTTP trace to see exactly what's special about the response.
EDIT: This change just confirms my suspicion. It doesn't solve your problem. It just hides the symptom.
If the response from first request is 200, we need a trace. I normally use Ethereal/Wireshark to get the TCP trace.
If your first response is not 200, I do see a problem in your code. With OAuth, the error response (401) actually returns data, which includes ProblemAdvice, Signature Base String etc to help you debug. You need to read everything from error stream. Otherwise, it's going to confuse next connection and that's the cause of -1. Following example shows you how to handle errors correctly,
public static String get(String url) throws IOException {
ByteArrayOutputStream os = new ByteArrayOutputStream();
URLConnection conn=null;
byte[] buf = new byte[4096];
try {
URL a = new URL(url);
conn = a.openConnection();
InputStream is = conn.getInputStream();
int ret = 0;
while ((ret = is.read(buf)) > 0) {
os.write(buf, 0, ret);
}
// close the inputstream
is.close();
return new String(os.toByteArray());
} catch (IOException e) {
try {
int respCode = ((HttpURLConnection)conn).getResponseCode();
InputStream es = ((HttpURLConnection)conn).getErrorStream();
int ret = 0;
// read the response body
while ((ret = es.read(buf)) > 0) {
os.write(buf, 0, ret);
}
// close the errorstream
es.close();
return "Error response " + respCode + ": " +
new String(os.toByteArray());
} catch(IOException ex) {
throw ex;
}
}
}

I've encountered the same problem when I did not read in all the data from the InputStream before closing it and opening a second connection. It was also fixed either with System.setProperty("http.keepAlive", "false"); or simply just looping until I've read the rest of the InputStream.
Not completely related to your issue, but hope this helps anyone else with a similar problem.

Google provided an elegant workaround since it's only happening prior to Froyo:
private void disableConnectionReuseIfNecessary() {
// HTTP connection reuse which was buggy pre-froyo
if (Integer.parseInt(Build.VERSION.SDK) < Build.VERSION_CODES.FROYO) {
System.setProperty("http.keepAlive", "false");
}
}
Cf. http://android-developers.blogspot.ca/2011/09/androids-http-clients.html

Or, you can set HTTP header in the connection (HttpUrlConnection):
conn.setRequestProperty("Connection", "close");

Can you verify that the connection is not getting closed before you finish reading the response? Maybe HttpClient parses the response code right away, and saves it for future queries, however HttpURLConnection could be returning -1 once the connection is closed?