We Keep Coding

Stream over HTTP SSL is not flushed

I have a web application running behind nginx. Some pages are accessible via http, some others via https. I have some "pages", which are rather streams as the application does not close the connection and feeds data as they come. The feed then looks like this:
TIME1 MESSAGE1
TIME2 MESSAGE2
...
TIMEn MESSAGEn
After each line I write "\n" and then call flush(). Over http, it works correctly and my client can listen to new data. However, over https the client is not receiving any data until the connection is closed.
ServletOutputStream stream = applicationModel.getOutputStream();
OutputStreamWriter streamWriter = new OutputStreamWriter(stream);
BufferedWriter writer = new BufferedWriter(streamWriter);
while (true) {
wait();
writer.write(newMessage);
writer.flush();
}

Unless the application is tightly integrated with the web server a flush on the writer will only flush the buffers inside your application, so that the data gets send to the web server. Inside the web server there are more buffers, which are necessary to optimize the traffic by sending larger TCP packets and thus decrease the overhead for the data. And, if you use SSL there is yet another layer to watch, because your data will be encapsulated into an SSL frame which again adds overhead, so it is good to not only have a few bytes payload inside. Finally you have the buffering at the OS kernel, which might defer the sending of a small TCP packet for some time if there is hope that there will be more data.
Please be aware, that your wish to control the buffers is against a fundamental design concept of HTTP. HTTP is based on the idea that you have a request from the client to the server and then a response from the server, ideally with a known content-length up-front. There is no idea in the original design of a response which evolves slowly and where the browser will update the display once new data arrive. The real way to get updates would be instead to let the client send another request and then send the new response back. Another way would be to use WebSockets.

Keep TCP socket-connection alive if no data is currently available

I have implemented a small HTTP-server which allows clients to connect via HTTP and stream audio-data to them.
My problem is, that in case there's currently no audio-data available, the connection seems to break, either because the client is disconnecting, or due to another reason inside Android.
I'm acting like the following way:
serverSocket = new ServerSocket(0);
Socket socket = serverSocket.accept();
socket.setKeepAlive(true);
BufferedReader in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
BufferedWriter out = new BufferedWriter(new OutputStreamWriter(socket.getOutputStream()));
out.write("HTTP/1.1 200 OK\r\n");
out.write("Content-Type: audio/wav\r\n");
out.write("Accept-Ranges: none\r\n");
out.write("Connection: keep-alive\r\n"); // additionally added due to answer below
out.write("\r\n");
out.flush();
..
while(len=otherInput.read(audioBuffer)){
out.write(audioBuffer, 0, len);)
}
For sure this is just a snipped of the real code, but it shows what I'm doing.
Now, in case the "otherinput.read()" takes a long time because there's no data available at the moment, I get a
java.net.SocketException: sendto failed: EPIPE (Broken pipe)
at libcore.io.IoBridge.maybeThrowAfterSendto(IoBridge.java:499)
at libcore.io.IoBridge.sendto(IoBridge.java:468)
at java.net.PlainSocketImpl.write(PlainSocketImpl.java:508)
at java.net.PlainSocketImpl.access$100(PlainSocketImpl.java:46)
at java.net.PlainSocketImpl$PlainSocketOutputStream.write(PlainSocketImpl.java:270)
at java.io.BufferedOutputStream.flushInternal(BufferedOutputStream.java:185)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:139)
Who can tell me how I can prevent the connection from breaking/closing without a manual heartbeat? Do I miss some header or am I using something the wrong way?
Thanks for your help in advance, tried and searched myself crazy meanwhile.

There are at least two problems here.
Clients of HTTP servers are not well-behaved in the way you seem to expect. Consider a browser. The user can shut it down, go back, navigate away etc, any time he likes, even in the middle of a page load. If you get any error transmitting to the client there's nothing you can do except close the connection and forget about it. Same applies to any server really, but it applies unsolder to HTTP servers.
You're not reading the entire request sent by the client. You need to read all the headers until a blank line, then you need to read the body up to the length specified in the Content-length: header, or all the chunks, or until end of stream, as the case may be: see RFC 2616. The effect of this may be that you cause the behaviour at (1).

How to transfer data from one jsp to another jsp with sockets

I have 3 .jsp's. The first one asks the user for their username. Once the form is submitted it is taken to a 2nd jsp where a unique passcode is created for the user. How would I go about taking this passcode and passing it to a 3rd jsp using a socket?

You can use java.net.URL and java.net.URLConnection to fire and handle HTTP requests programmatically. They make use of sockets under the covers and this way you don't need to fiddle with low level details about the HTTP protocol. You can pass parameters as query string in the URL.
String url = "http://localhost:8080/context/3rd.jsp?passcode=" + URLEncoder.encode(passcode, "UTF-8");
InputStream input = new URL(url).openStream();
// ... (read it, it contains the response)
This way the passcode request parameter is available in the 3rd JSP by ${param.passcode} or request.getParameter("passcode") the usual way.
Better is however to just include that 3rd JSP in your 2nd JSP.
request.setAttribute("passcode", passcode);
request.getRequestDispatcher("3rd.jsp").include(request, response);
This way the passcode is available as request attribute in the 3rd JSP by ${passcode} or request.getAttribute("passcode") the usual way.
See also:
Using java.net.URLConnection to fire and handle HTTP requests
Unrelated to the concrete question, this is however a terribly nasty hack and the purpose of this is beyond me. There's somewhere a serious design flaw in your application. Most likely those JSPs are tight coupled with business logic which actually belongs in normal and reuseable Java classes like servlets and/or EJBs and/or JAX-WS/RS which you just import and call in your Java class the usual Java way. JSPs are meant to generate and send HTML, not to act as business services, let alone web services. See also How to avoid Java code in JSP files?

So, you want the username to be submitted from the first JSP to the second, by submitting a form to the second, right?
But, for interaction between the second and third, you want to avoid using the communication mechanisms behind the the JSP files and use your own, right?
Well, how you might implement doing this depends on where you're sending your communication from and to. For instance, are they on the same machine, or on different machines?
Generally speaking, you'll need a client-server type of relationship to be set up here. I imagine that you would want your third JSP to act as the server.
What the third JSP will do is will sit and wait for a client to try to communicate with it. But, before you can do that, you'll first need to bind a port to your application. Ports are allocated by the Operating System and are given to requesting processes.
When trying to implement this in Java, you might want to try something like the following:
int port_number = 1080;
ServerSocket server = new ServerSocket(port_number);
In the above example, the ServerSocket is already bound to the specified port 1080. It doesn't have to be 1080 - 1080 is just an example.
Next, you will want to listen and wait for a request to come in. You can implement this step in the following:
Socket request = null;
while((request = server.accept()) == null)
{}
This will cause the server socket to keep looping until it finally receives a request. When the request comes in, it will create a new Socket object to handle that request. So, you could come back to your loop later on and continue to wait and accept requests, while a child thread handles communication using your newly created request Socket.
But, for your project, I would guess that you don't need to communicate with more than one client at a time, so it's okay if we just simply stop listening once we receive a request, I suppose.
So, now onto the client application. Here, it's a little bit different from what we had with the server. First off, instead of listening in on the port and waiting for are request, the client's socket will actively try to connect to a remote host on their port. So, if there is no server listening in on that port, then the connection will fail.
So, two things will need to be know, those are:
What's the IP Address of the server?
What port is the server listening in on?
There are short-cuts to getting the connection using the Java Socket class, but I assume that you're going to test this out on the same machine, right? If so, then you will need two separate ports for both your client and server. That's because the OS won't allow two separate processes to share the same port. Once a process binds to the port, no other process is allowed to access it until that port releases it back to the OS.
So, to make the two separate JSP's communicate on the same physical machine, you'll need both a local port for your client, and you'll need the server's port number that it's listening in on.
So, let's try the following for the client application:
int local_port = 1079;
int remote_port = 1080;
InetSocketAddress localhost = new InetSocketAddress(local_port);
Socket client = new Socket(); //The client socket is not yet bound to any ports.
client.bind(localhost); //The client socket has just requested the specified port number from the OS and should be bound to it.
String remoteHostsName = "[put something here]";
InetSocketAddress remotehost = new InetSocketAddress(InetAddress.getByName(remoteHostsName), remote_port); //Performs a DSN lookup of the specified remote host and returns an IP address with the allocated port number
client.connect(remotehost); //Connection to the remote server is being made.
That should help you along your way.
A final note should be made here. You can't actually run these two applications using the same JVM. You'll need two separate processes for client and server applications to run.

using sockets to fetch a webpage with java

I'd like to fetch a webpage, just fetching the data (not parsing or rendering anything), just catch the data returned after a http request.
I'm trying to do this using the high-level Class Socket of the JavaRuntime Library.
I wonder if this is possible since I'm not at ease figuring out the beneath layer used for this two-point communication or I don't know if the trouble is coming from my own system.
.
Here's what my code is doing:
1) setting the socket.
this.socket = new Socket( "www.example.com", 80 );
2) setting the appropriate streams used for this communication.
this.out = new PrintWriter( socket.getOutputStream(), true);
this.in = new BufferedReader( new InputStreamReader( socket.getInputStream() ) );
3) requesting the page (and this is where I'm not sure it's alright to do like this).
String query = "";
query += "GET / HTTP/1.1\r\n";
query += "Host: www.example.com\r\n";
...
query += "\r\n";
this.out.print(query);
4) reading the result (nothing in my case).
System.out.print( this.in.readLine() );
5) closing socket and streams.

If you're on a *nix system, look into CURL, which allows you to retrieve information off the internet using the command line. More lightweight than a Java socket connection.
If you want to use Java, and are just retrieving information from a webpage, check out the Java URL library (java.net.URL). Some sample Java code:
URL ur = new URL("www.google.com");
URLConnection conn = ur.openConnection();
InputStream is = conn.getInputStream();
String foo = new Scanner(is).useDelimiter("\\A").next();
System.out.println(foo);
That'll grab the specified URL, grab the data (html in this case) and spit it out to the console. Might have to tweak the delimiter abit, but this will work with most network endpoints sending data.

Your code looks pretty close. Your GET request is probably malformed in some way. Try this: open up a telnet client and connect to a web server. Paste in the GET request as you believe it should work. See if that returns anything. If it doesn't it means there is a problem with the GET request. The easiest thing to do that point would be write a program that listens on a socket (more or less the inverse of what you're doing) and point a web browser to localhost:[correct port] and see what the web browser sends you. Use that as your template for the GET request.
Alternatively you could try and piece it together from the HTTP specification.

I had to add the full URL to the GET parameter. To make it work. Although I see you can specify HOST also if you want.
Socket socket = new Socket("youtube.com",80);
PrintWriter out = new PrintWriter(new BufferedWriter(new
OutputStreamWriter(socket.getOutputStream())));
out.println("GET http://www.youtube.com/yts/img/favicon_48-vflVjB_Qk.png
HTTP/1.0");
out.println();
out.flush();

Yes, it is possible. You just need to figure out the protocol. You are close.
I would create a simple server socket that prints out what it gets in. You can then use your browser to connect to the socket using a url like: http://localhost:8080. Then use your client socket to mimic the HTTP protocol from the browser.

Not sure why you're going lower down than URLConnection - its designed to do what you want to do: http://download.oracle.com/javase/tutorial/networking/urls/readingWriting.html.
The Java Tutorial on Sockets even says: "URLs and URLConnections provide a relatively high-level mechanism for accessing resources on the Internet. Sometimes your programs require lower-level network communication, for example, when you want to write a client-server application." Since you're not going lower than HTTP, I'm not sure what the point is of using a Socket.

java.net.SocketTimeoutException: Read timed out

I have an application with client server architecture. The client
use Java Web Start with Java Swing / AWT and the sert uses HTTP server / Servlet with
Tomcat.
The communication is made from the serialization of objects, create a
ObjectOutput serializes a byte array and send to the server
respectively called the ObjectInputStream and deserializes.
The application follows communicating correctly to a certain
time of concurrency where starting to show error
"SocketException read timeout". The erro happens when the server invoke the method
ObjectInputStream.getObject() in my servlet doPost method.
The tomcat will come slow and the errors start to decrease server response time until the crash time where i must restart the server and after everything works.
Someone went through this problem ?
Client Code
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
OutputStream os = conn.getOutputStream();
ObjectOutputStream oss = new ObjectOutputStream(os);
oss.writeUTF("protocol header sample");
oss.writeObject(_parameters);
oss.flush();
oss.close();
Server Code
ObjectInputStream input = new ObjectInputStream(_request.getInputStream());
String method = input.readUTF();
parameters = input.readObject();
input.readObject() is where the error is

You haven't given us much information to go on, especially about the client side. But my suspicion is that the client side is:
failing to setting the Content-length header (or setting it to the wrong value),
failing to flush the output stream, and/or
not closing the output side of the socket.
Mysterious.
Based on your updated question, it looks like none of the above. Here are a couple of other possibilities:
For some reason the client side is either locking up entirely during serialization or taking a VERY LONG TIME.
There is a proxy between the client and server that is causing problems.
You are experiencing load-related network problems, or network hardware problems.
Another possible explanation is that you have a memory leak, and that the slowdown is caused by the GC taking more and more time as you run out of memory. This will show up in the GC logs if you have them enabled.

I think During high Concurrency, the Socket Timeout set in Tomcat is Expired and the connection is closed. The next read by Tomcat for that connection is greater than the server socket timeout specified in the server.
If you want to avoid this problem you have to increase the timeout on the server-side which is expired in your case. But not advisable.
BTW you did not give enough information. Did you increase the no of threads for connection in Tomcat? If you did, this surely would happen.