We Keep Coding

c# consume http body without waiting end of stream/body

I am a Java developer
I'm trying to stream an unknown amount of binary data from client to server over http put request.
For some reason, my server is written in c# (.net core) and for convenience, my client (that i use only for test) is written in java.
As i'm waiting for unknown amount of data and in order to test my server rest api, my java client open a binary file and send binary packet over http put request (160 bytes by packet for example).
When i begin streaming data from client, i'm not able to consume the body (http stream) in server side until the client finished to send all binary packets.
http headers in use use: Connection-KeepAlive, Content-Type=> application/octetstream, transfer-encoding => chunked
So, I would like to consume the body without waiting the entire body.
Any hint of how to perform it ?
I'm thinking about removing await operator before async method that consume body and wait until client abort or close the connection in server side but i don't know if this is a good idea.
Below some piece of related code:
server side written in c#
[HttpPut]
public async Task<IActionResult> SendData(string id, string device, string token){
...
using (MemoryStream ms = new MemoryStream()){
await HttpContext.Request.Body.CopyToAsync(ms);
}
...
//do stuff
}
java client
HttpURLConnection conn = (HttpURLConnection) new URL(
"http://localhost:8080/test"
).openConnection();
byte[] data = Files.readAllBytes(Paths.get(FILE_TO_SEND_PATH));
ByteArrayInputStream is = new ByteArrayInputStream(data);
conn.setRequestMethod("PUT");
conn.setRequestProperty("Content-Type", "application/octet-stream");
conn.setRequestProperty("Connection", "Keep-Alive");
conn.setDoOutput(true);
conn.setDoInput(true);
conn.setChunkedStreamingMode(160);
//conn.setFixedLengthStreamingMode(is.available());
conn.setDefaultUseCaches(false);
conn.setUseCaches(false);
conn.connect();
OutputStream out = conn.getOutputStream();
//InputStream i = conn.getInputStream();
byte[] bytes = new byte[160];
int read = 0;
while ((read = is.read(bytes)) > 0) {
out.write(bytes, 0, read);
System.out.println(read);
TimeUnit.MICROSECONDS.sleep(125 * read);
out.flush();
}
out.close();

How can I forward HttpResponse to an open client socket? (Java Program acts as a proxy)

I am using httpclient lib from apache. I managed to get an HttpResponse by sending a GET request to the server. Now what I am trying to do is to send that response that I got to a clientSocket output stream.
So basically I want to send whatever I received from the server to the open client connection. Since I am using HttpClient I get the response in the form of an HttpResponse object. I tried the following:
private void forwardRequest(String header, String url){
try {
CloseableHttpClient httpclient = HttpClients.createDefault();
HttpGet httpget = new HttpGet(url);
CloseableHttpResponse response;
//Adding the request headers to httpget
String lines[] = header.split("\\n");
for (String str : lines) {
String parts[] = str.split(":", 2);
httpget.addHeader(parts[0], parts[1]);
}
HttpResponse respone;
response = httpclient.execute(httpget);
//It works till here I can read from the response and print out the html page
//But after this I don't know how to send it to client
OutputStream bos = clientSocket.getOutputStream();
PrintWriter pw = new PrintWriter(bos);
BufferedReader rd = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
while ((line = rd.readLine()) != null) {
pw.println(line);
//bos.write(line.getBytes()); //This also doesn't work
}
response.close();
}
Also clientSocket is a global variable which is associcated with a ServerSocket like:
clientSocket = serverSocket.accept();
I don't expect a full solution. Just point me in the right direction.. Thanks a ton!
EDIT:
I tried the following based on what EJP suggested.. It's still not working. I was wondering if it was correctly implemented?
int portNumber = 8012; // port on which the program listens
ServerSocket serverSocket =
new ServerSocket(portNumber); //the socket at which the program listens
Socket clientSocket = serverSocket.accept(); //clientSocket of the program
Socket toServer = new Socket("localhost", 8089); //proxy server to which program connects
PrintWriter out =
new PrintWriter(toServer.getOutputStream(), true);
PrintWriter outClient =
new PrintWriter(clientSocket.getOutputStream(), true);
BufferedReader in = new BufferedReader(
new InputStreamReader(clientSocket.getInputStream()));
BufferedReader inServer = new BufferedReader(
new InputStreamReader(toServer.getInputStream()));
) {
String inputLine;
while ((inputLine = in.readLine()) != null) {
out.println(inputLine); //Writing to proxy server
outClient.println(inServer.readLine()); //writing back to original request sender
System.out.println(inputLine);
}

The client made an HTTP request, so it will be expecting an HTTP response. If the global clientSocket is just a raw TCP socket and not an HttpClient, then you need to add the HTTP response protocol header yourself.
You have the content from the server, you'll want to first return an HTTP response 200 OK, then empty line with carriage return + linefeed (CR+LF), then Content-length: , then the document. If you are just proxying text documents, then you could convert to a string here, but otherwise, I would just pass the mime type, charset, and entity through as the raw bytes as the web server responded, that way you can proxy any document, including images or binary files.
It will look something like this:
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: length
<html> ...
</html>
To pass the http headers through from the server:
HttpEntity entity = response.getEntity();
// technically you should check the HTTP response rather than assume it is a 200
int statusCode = httpResp.getStatusLine().getStatusCode();
if(statusCode != 200)
... // do something with non 200 responses ?
clientSocket.write("HTTP/1.1 200 OK\r\n");
Header[] responseHeaders = response.getAllHeaders();
for(Header header : responseHeaders) {
clientSocket.write(header.toString() + "\r\n");
}
clientSocket.write("\r\n"); // empty line required
// Use BufferedInputStream to deal in bytes
BufferedInputStream input = new BufferedInputStream(entity.getContent());
byte[] buf = new byte[8192];
int bytesRead;
while ((bytesRead = input.read(buf, 8192)) > 0) {
clientSocket.write(buf, bytesRead);
}
I say "something like this", don't take this literal, I doubt it compiles. I don't have dev station in front of me, but this is the general idea.
NOTE: Since you are using the Apache client lib, you should be able to use the specific HTTP client instead of writing the raw protocol. This will abstract the HTTP protocol away somewhat. I'll update the answer later if nobody else provides a better one.

If you're just forwarding requests and responses you don't have any need to engage in the HTTP protocol at all beyond the first line of the request. If the client knows you're the proxy you will get either a GET request with the full URL or else a CONNECT request ditto. All you have to do is connect to the target and then just copy bytes in both directions simultaneously.

readLine() never stops reading

inFromClientR.readLine() never stops. any ideas? Am I forgetting something?
Server:
/*{ some code:
send a file with a dataoutputstream to client using a new port(4000) and when transfer is done i want a responce message (e.g. OK) send back to server in the old port(6000)
}*/
ServerSocket listenTransferSocket = new ServerSocket(6000);
Socket connectionTransferSocket = listenTransferSocket.accept();
BufferedReader inFromClientR =
new BufferedReader(new InputStreamReader(connectionTransferSocket.getInputStream()));
System.out.println("Client's response to Transfer: " +inFromClientR.readLine());
Client:
/*{ some code:
receive the file on port (4000) and then the responce is sent to server using the following commands
}*/
Socket fileTransferSocket = new Socket("localhost", 6000);
DataOutputStream outToServerR =
new DataOutputStream(fileTransferSocket.getOutputStream());
outToServerR.writeBytes("Transfer completed " +'\n');

BufferedReader#readLine() tries to fill its buffer with 8192 bytes, regradless of any linefeeds it find meanwhile. Since you have the connection open, the receiving side will wait until 1) you have sent 8192 bytes, or 2) closes the connection.
You would be better off using some other framing mechanism, maybe an ObjectOutputStream/ObjectInputStream.

String line = null;
while ((line = inFromClientR.readLine()) != null) {
// do sth
}

Process raw HTTP headers by HttpClient in Java

My client receive raw HTTP headers (including GET, POST, Multipart POST, etc.) and I want to send them to a server and get output.
But I don't want to parse whole request manually, then set all that parsed stuff to HttpClient...
Does an elegant way to do this (even something like code below)?
AGoodHttpClient response = new AGoodHttpClient(host, port, myHeaders);
InputStream in = response.getInputStream();
// ...
Edited
Let's say I have this code. How do I recognize EOS (-1 isn't working for HTTP/1.1). Is there a guaranteed way how to cut the connection, when transfer is done? I want something what will care about cutting a connection (something like HttpClient), but with direct access to sending headers (like outToServer.write(myHeaders)).
Socket connectionToServer = new Socket(host, port);
OutputStream outToServer = connectionToServer.getOutputStream();
outToServer.write(myHeaders.getBytes());
InputStream inputFromServer = connectionToServer.getInputStream();
byte[] buff = new byte[1024];
int count;
while ((count = inputFromServer.read(buff)) != -1) {
System.out.write(buff, 0, count);
}
Thanks for help!

HttpsURLConnection Connection Problems

I'm a problem with a HttpsURLConnection that I can't seem to solve. Basically, I'm sending up some info to a server and if some of that data is wrong, the server sends me a 500 response code. However, it also sends a message in the response telling me which bit of data was wrong. The problem is that the message is always empty when I read it in. I think this is because a filenotfound exception always gets thrown before the stream can be read. Am I right? I tried reading the errorstream as well but this is always empty. Here's a snippet:
conn = (HttpsURLConnection) connectURL.openConnection();
conn.setDoOutput(true);
conn.setConnectTimeout(30000);
conn.setReadTimeout(30000);
conn.setRequestMethod("POST");
conn.setRequestProperty("Content-Length",
Integer.toString(outString.getBytes().length));
DataOutputStream wr = new DataOutputStream(conn
.getOutputStream());
wr.write(outString.getBytes());
wr.flush();
wr.close();
if(conn.getResponseCode>400{
String response = getErrorResponse(conn);
public String getErrorResponse(HttpsURLConnection conn) {
Log.i(TAG, "in getResponse");
InputStream is = null;
try {
//is = conn.getInputStream();
is = conn.getErrorStream();
// scoop up the reply from the server
int ch;
StringBuffer sb = new StringBuffer();
while ((ch = is.read()) != -1) {
sb.append((char) ch);
}
//System.out.println(sb.toString());
return sb.toString();
// return conferenceId;
}
catch (Exception e){
e.printStackTrace();
}
}

So just to follow up on this, here is how I solved it:
public static String getResponse(HttpsURLConnection conn) {
Log.i(TAG, "in getResponse");
InputStream is = null;
try {
if(conn.getResponseCode()>=400){
is = conn.getErrorStream();
}
else{
is=conn.getInputStream();
}
...read stream...
}
It seems that calling them like this produced an error stream with a message. Thanks for the suggestions!

Try setting content-type request property to "application/x-www-form-urlencoded"
The same is mentioned on this link:
http://developers.sun.com/mobility/midp/ttips/HTTPPost/
The Content-Length and Content-Type headers are critical because they tell the web server how many bytes of data to expect, and what kind, identified by a MIME type.
In MIDP clients the two most popular MIME types are application/octet-stream, to send raw binary data, and application/x-www-form-urlencoded, to send name-value pairs

Are you in control of the server? In other words, did you write the process that runs on the server and listens to the port you're trying to access?
If you did, then you should also be able to debug it and see why your process returns 404.
If you didn't, then describe your architecture (HTTP server, the component it invokes to respond to your HTTP(S) request, etc) and we'll take it from there.
In the very simplest case, of an HTTP server being an Apache server yielding control to some PHP script, it means that Apache couldn't assign your request to anything. Most likely a Web server misconfiguration. Provide some more details and we'll help you out.