We Keep Coding

How to send bytearray to specific receiver

I have a TCP Server and Client in Java. The Server can send commands to the Client, the Client will then execute the command, for example: send an image to the Server.
Im sending the data with a bytearray and thats working.
But lets imagine, I want to send an image and a file separately. How would the Server supposed to know which is the right bytearray? Or if I want to make a VoiceChat (which needs to be sending bytearrays continously) and separately sending an image?
Thats my code send bytes:
Client.java
public void writeBytes(byte[] bytes, Socket socket) throws IOException {
DataOutputStream out = new DataOutputStream(socket.getOutputStream());
out.write(bytes);
out.flush();
}
Thats my code to receive and convert them to an Image:
Server.java
public BufferedImage writeScreenshot(Socket socket, int length) throws IOException {
DataInputStream in = new DataInputStream(new BufferedInputStream(socket.getInputStream()));
byte[] buffer = new byte[length];
in.readFully(buffer);
return ImageIO.read(new ByteArrayInputStream(buffer));
}

You need to design a "protocol" for the communication. A protocol defines what are the messages that can be exchanged and how they are represented in the lower level data stream.
A quick and easy protocol is where you first send the length of the data you are going to send, and then the data:
DataOutputStream out = new DataOutputStream(socket.getOutputStream());
out.writeInt(bytes.length);
out.write(bytes);
out.flush();
The receiver now has to read the length field:
DataInputStream in = new DataInputStream(new BufferedInputStream(socket.getInputStream()));
int length = in.readInt()
byte[] buffer = new byte[length];
in.readFully(buffer);
When you get to applications like voice chat, the protocol has to get more complex. Each message has to have metadata, like what type of data it contains: image or voice or something else. Also you would likely not want to design this protocol from scratch, but use something that already exists - for example the real-time streaming protocol (RTSP).

c# consume http body without waiting end of stream/body

I am a Java developer
I'm trying to stream an unknown amount of binary data from client to server over http put request.
For some reason, my server is written in c# (.net core) and for convenience, my client (that i use only for test) is written in java.
As i'm waiting for unknown amount of data and in order to test my server rest api, my java client open a binary file and send binary packet over http put request (160 bytes by packet for example).
When i begin streaming data from client, i'm not able to consume the body (http stream) in server side until the client finished to send all binary packets.
http headers in use use: Connection-KeepAlive, Content-Type=> application/octetstream, transfer-encoding => chunked
So, I would like to consume the body without waiting the entire body.
Any hint of how to perform it ?
I'm thinking about removing await operator before async method that consume body and wait until client abort or close the connection in server side but i don't know if this is a good idea.
Below some piece of related code:
server side written in c#
[HttpPut]
public async Task<IActionResult> SendData(string id, string device, string token){
...
using (MemoryStream ms = new MemoryStream()){
await HttpContext.Request.Body.CopyToAsync(ms);
}
...
//do stuff
}
java client
HttpURLConnection conn = (HttpURLConnection) new URL(
"http://localhost:8080/test"
).openConnection();
byte[] data = Files.readAllBytes(Paths.get(FILE_TO_SEND_PATH));
ByteArrayInputStream is = new ByteArrayInputStream(data);
conn.setRequestMethod("PUT");
conn.setRequestProperty("Content-Type", "application/octet-stream");
conn.setRequestProperty("Connection", "Keep-Alive");
conn.setDoOutput(true);
conn.setDoInput(true);
conn.setChunkedStreamingMode(160);
//conn.setFixedLengthStreamingMode(is.available());
conn.setDefaultUseCaches(false);
conn.setUseCaches(false);
conn.connect();
OutputStream out = conn.getOutputStream();
//InputStream i = conn.getInputStream();
byte[] bytes = new byte[160];
int read = 0;
while ((read = is.read(bytes)) > 0) {
out.write(bytes, 0, read);
System.out.println(read);
TimeUnit.MICROSECONDS.sleep(125 * read);
out.flush();
}
out.close();

How can I forward HttpResponse to an open client socket? (Java Program acts as a proxy)

I am using httpclient lib from apache. I managed to get an HttpResponse by sending a GET request to the server. Now what I am trying to do is to send that response that I got to a clientSocket output stream.
So basically I want to send whatever I received from the server to the open client connection. Since I am using HttpClient I get the response in the form of an HttpResponse object. I tried the following:
private void forwardRequest(String header, String url){
try {
CloseableHttpClient httpclient = HttpClients.createDefault();
HttpGet httpget = new HttpGet(url);
CloseableHttpResponse response;
//Adding the request headers to httpget
String lines[] = header.split("\\n");
for (String str : lines) {
String parts[] = str.split(":", 2);
httpget.addHeader(parts[0], parts[1]);
}
HttpResponse respone;
response = httpclient.execute(httpget);
//It works till here I can read from the response and print out the html page
//But after this I don't know how to send it to client
OutputStream bos = clientSocket.getOutputStream();
PrintWriter pw = new PrintWriter(bos);
BufferedReader rd = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
while ((line = rd.readLine()) != null) {
pw.println(line);
//bos.write(line.getBytes()); //This also doesn't work
}
response.close();
}
Also clientSocket is a global variable which is associcated with a ServerSocket like:
clientSocket = serverSocket.accept();
I don't expect a full solution. Just point me in the right direction.. Thanks a ton!
EDIT:
I tried the following based on what EJP suggested.. It's still not working. I was wondering if it was correctly implemented?
int portNumber = 8012; // port on which the program listens
ServerSocket serverSocket =
new ServerSocket(portNumber); //the socket at which the program listens
Socket clientSocket = serverSocket.accept(); //clientSocket of the program
Socket toServer = new Socket("localhost", 8089); //proxy server to which program connects
PrintWriter out =
new PrintWriter(toServer.getOutputStream(), true);
PrintWriter outClient =
new PrintWriter(clientSocket.getOutputStream(), true);
BufferedReader in = new BufferedReader(
new InputStreamReader(clientSocket.getInputStream()));
BufferedReader inServer = new BufferedReader(
new InputStreamReader(toServer.getInputStream()));
) {
String inputLine;
while ((inputLine = in.readLine()) != null) {
out.println(inputLine); //Writing to proxy server
outClient.println(inServer.readLine()); //writing back to original request sender
System.out.println(inputLine);
}

The client made an HTTP request, so it will be expecting an HTTP response. If the global clientSocket is just a raw TCP socket and not an HttpClient, then you need to add the HTTP response protocol header yourself.
You have the content from the server, you'll want to first return an HTTP response 200 OK, then empty line with carriage return + linefeed (CR+LF), then Content-length: , then the document. If you are just proxying text documents, then you could convert to a string here, but otherwise, I would just pass the mime type, charset, and entity through as the raw bytes as the web server responded, that way you can proxy any document, including images or binary files.
It will look something like this:
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: length
<html> ...
</html>
To pass the http headers through from the server:
HttpEntity entity = response.getEntity();
// technically you should check the HTTP response rather than assume it is a 200
int statusCode = httpResp.getStatusLine().getStatusCode();
if(statusCode != 200)
... // do something with non 200 responses ?
clientSocket.write("HTTP/1.1 200 OK\r\n");
Header[] responseHeaders = response.getAllHeaders();
for(Header header : responseHeaders) {
clientSocket.write(header.toString() + "\r\n");
}
clientSocket.write("\r\n"); // empty line required
// Use BufferedInputStream to deal in bytes
BufferedInputStream input = new BufferedInputStream(entity.getContent());
byte[] buf = new byte[8192];
int bytesRead;
while ((bytesRead = input.read(buf, 8192)) > 0) {
clientSocket.write(buf, bytesRead);
}
I say "something like this", don't take this literal, I doubt it compiles. I don't have dev station in front of me, but this is the general idea.
NOTE: Since you are using the Apache client lib, you should be able to use the specific HTTP client instead of writing the raw protocol. This will abstract the HTTP protocol away somewhat. I'll update the answer later if nobody else provides a better one.

If you're just forwarding requests and responses you don't have any need to engage in the HTTP protocol at all beyond the first line of the request. If the client knows you're the proxy you will get either a GET request with the full URL or else a CONNECT request ditto. All you have to do is connect to the target and then just copy bytes in both directions simultaneously.

How to reuse inputstream from java.net.socket?

I want to make http request with socket because I want to test how many sockets I can create my server. So I write and read from my server using OutputStream and InputStream. But I can't read from the input stream again after the first response. Do you know how to read the second response without closing the socket?
Here is my code:
Socket socket = new Socket();
socket.connect(new InetSocketAddress(address, 80), 1000);
socket.setSoTimeout(25*1000);
OutputStream os = socket.getOutputStream();
os.write(getRequest(host)); // some request as bytearray, it has Connection: Keep-Alive in the header
os.flush();
InputStream is = socket.getInputStream();
BufferedInputStream bis = new BufferedInputStream(is);
String response = IOUtils.toString(bis);
System.out.println("RESPONSE = \n" + response); // this works fine
os.write(getRequestBodyBa()); // send another request, i can see it sent to server with wireshark
os.flush();
// try to read again but it always return empty string
response = IOUtils.toString(bis); // how to read the second response?????
System.out.println("RESPONSE = \n" + response);
os.close();
is.close();
socket.close();
Thanks.

I believe the HTTP standard are to close the connection after each response, unless the request has the Connection header set to keep-alive.

IOUtils.toString(InputStream) reads the stream to EOS, so there can't be anything left to read for next time. Don't use it. You need to parse the response headers, work out whether there is a Content-Length header, if so read the body for exactly that many bytes; if there is no Content-Length header (and no chunking) the connection is closed after the body so you can't send a second command; etc etc etc. It is endless. Don't use a Socket for this either: use an HTTP URL and URLConnection.

Resumable upload from Java client to Grails web application?

After almost 2 workdays of Googling and trying several different possibilities I found throughout the web, I'm asking this question here, hoping that I might finally get an answer.
First of all, here's what I want to do:
I'm developing a client and a server application with the purpose of exchanging a lot of large files between multiple clients on a single server. The client is developed in pure Java (JDK 1.6), while the web application is done in Grails (2.0.0).
As the purpose of the client is to allow users to exchange a lot of large files (usually about 2GB each), I have to implement it in a way, so that the uploads are resumable, i.e. the users are able to stop and resume uploads at any time.
Here's what I did so far:
I actually managed to do what I wanted to do and stream large files to the server while still being able to pause and resume uploads using raw sockets. I would send a regular request to the server (using Apache's HttpClient library) to get the server to send me a port that was free for me to use, then open a ServerSocket on the server and connect to that particular socket from the client.
Here's the problem with that:
Actually, there are at least two problems with that:
I open those ports myself, so I have to manage open and used ports myself. This is quite error-prone.
I actually circumvent Grails' ability to manage a huge amount of (concurrent) connections.
Finally, here's what I'm supposed to do now and the problem:
As the problems I mentioned above are unacceptable, I am now supposed to use Java's URLConnection/HttpURLConnection classes, while still sticking to Grails.
Connecting to the server and sending simple requests is no problem at all, everything worked fine. The problems started when I tried to use the streams (the connection's OutputStream in the client and the request's InputStream in the server). Opening the client's OutputStream and writing data to it is as easy as it gets. But reading from the request's InputStream seems impossible to me, as that stream is always empty, as it seems.
Example Code
Here's an example of the server side (Groovy controller):
def test() {
InputStream inStream = request.inputStream
if(inStream != null) {
int read = 0;
byte[] buffer = new byte[4096];
long total = 0;
println "Start reading"
while((read = inStream.read(buffer)) != -1) {
println "Read " + read + " bytes from input stream buffer" //<-- this is NEVER called
}
println "Reading finished"
println "Read a total of " + total + " bytes" // <-- 'total' will always be 0 (zero)
} else {
println "Input Stream is null" // <-- This is NEVER called
}
}
This is what I did on the client side (Java class):
public void connect() {
final URL url = new URL("myserveraddress");
final byte[] message = "someMessage".getBytes(); // Any byte[] - will be a file one day
HttpURLConnection connection = url.openConnection();
connection.setRequestMethod("GET"); // other methods - same result
// Write message
DataOutputStream out = new DataOutputStream(connection.getOutputStream());
out.writeBytes(message);
out.flush();
out.close();
// Actually connect
connection.connect(); // is this placed correctly?
// Get response
BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String line = null;
while((line = in.readLine()) != null) {
System.out.println(line); // Prints the whole server response as expected
}
in.close();
}
As I mentioned, the problem is that request.inputStream always yields an empty InputStream, so I am never able to read anything from it (of course). But as that is exactly what I'm trying to do (so I can stream the file to be uploaded to the server, read from the InputStream and save it to a file), this is rather disappointing.
I tried different HTTP methods, different data payloads, and also rearranged the code over and over again, but did not seem to be able to solve the problem.
What I hope to find
I hope to find a solution to my problem, of course. Anything is highly appreciated: hints, code snippets, library suggestions and so on. Maybe I'm even having it all wrong and need to go in a totally different direction.
So, how can I implement resumable file uploads for rather large (binary) files from a Java client to a Grails web application without manually opening ports on the server side?

HTTP GET method have special headers for range retrieval: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35 It's used by most downloaders to do resumable download from server.
As I understand, there are no standard practice for using this headers for POST/PUT request, but it's up to you, right? You can make pretty standard Grails controller, that will accept standard http upload, with header like Range: bytes=500-999. And controller should put this 500 uploaded bytes from client into file, starting at position 500
At this case you don't need to open any socket, and make own protocols, etc.
P.S. 500 bytes is just a example, probably you're using much bigger parts.

Client Side Java Programming:
public class NonFormFileUploader {
static final String UPLOAD_URL= "http://localhost:8080/v2/mobileApp/fileUploadForEOL";
static final int BUFFER_SIZE = 4096;
public static void main(String[] args) throws IOException {
// takes file path from first program's argument
String filePath = "G:/study/GettingStartedwithGrailsFinalInfoQ.pdf";
File uploadFile = new File(filePath);
System.out.println("File to upload: " + filePath);
// creates a HTTP connection
URL url = new URL(UPLOAD_URL);
HttpURLConnection httpConn = (HttpURLConnection) url.openConnection();
httpConn.setDoOutput(true);
httpConn.setRequestMethod("POST");
// sets file name as a HTTP header
httpConn.setRequestProperty("fileName", uploadFile.getName());
// opens output stream of the HTTP connection for writing data
OutputStream outputStream = httpConn.getOutputStream();
// Opens input stream of the file for reading data
FileInputStream inputStream = new FileInputStream(uploadFile);
byte[] buffer = new byte[BUFFER_SIZE];
int bytesRead = -1;
while ((bytesRead = inputStream.read(buffer)) != -1) {
System.out.println("bytesRead:"+bytesRead);
outputStream.write(buffer, 0, bytesRead);
outputStream.flush();
}
System.out.println("Data was written.");
outputStream.flush();
outputStream.close();
inputStream.close();
int responseCode = httpConn.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_OK) {
// reads server's response
BufferedReader reader = new BufferedReader(new InputStreamReader(
httpConn.getInputStream()));
String response = reader.readLine();
System.out.println("Server's response: " + response);
} else {
System.out.println("Server returned non-OK code: " + responseCode);
}
}
}
Server Side Grails Programme:
Inside the controller:
def fileUploadForEOL(){
def result
try{
result = mobileAppService.fileUploadForEOL(request);
}catch(Exception e){
log.error "Exception in fileUploadForEOL service",e
}
render result as JSON
}
Inside the Service Class:
def fileUploadForEOL(request){
def status = false;
int code = 500
def map = [:]
try{
String fileName = request.getHeader("fileName");
File saveFile = new File(SAVE_DIR + fileName);
System.out.println("===== Begin headers =====");
Enumeration<String> names = request.getHeaderNames();
while (names.hasMoreElements()) {
String headerName = names.nextElement();
System.out.println(headerName + " = " + request.getHeader(headerName));
}
System.out.println("===== End headers =====\n");
// opens input stream of the request for reading data
InputStream inputStream = request.getInputStream();
// opens an output stream for writing file
FileOutputStream outputStream = new FileOutputStream(saveFile);
byte[] buffer = new byte[BUFFER_SIZE];
int bytesRead = inputStream.read(buffer);
long count = bytesRead
while(bytesRead != -1) {
outputStream.write(buffer, 0, bytesRead);
bytesRead = inputStream.read(buffer);
count += bytesRead
}
println "count:"+count
System.out.println("Data received.");
outputStream.close();
inputStream.close();
System.out.println("File written to: " + saveFile.getAbsolutePath());
code = 200
}catch(Exception e){
mLogger.log(java.util.logging.Level.SEVERE,"Exception in fileUploadForEOL",e);
}finally{
map <<["code":code]
}
return map
}
I have tried with above code it is worked for me(only for file size 3 to 4MB, but for small size files some bytes of code missing or not even coming but in request header content-length is coming, not sure why it is happening.)