We Keep Coding

POST file (as binary stream) using java.net.HttpURLConnection as a param file=<file binary stream>

I am trying to upload (POST) a file to an endpoint using java.net.HttpURLConnection but I keep getting http code 400 (bad request).
I refered to Send File And Parameters To Server With HttpURLConnection in android API 23
but problem is that I need to send this file as request body param (file=).
Note: The files will be of small size only (4-5mb) so I am reading it entirely in memory.
Corresponding curl request is:
curl -X POST "API" -H "Content-Type: multipart/form-data" -F "file="
Excerpts of Code that I am using:
Proxy webproxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("
<proxy host>", <proxy_port>));
HttpURLConnection http_conn = (HttpURLConnection)
url.openConnection(webproxy);
String authorization = getAuthorization(access_token);
http_conn.setRequestMethod("POST");
http_conn.setRequestProperty("Accept-Charset", "UTF-8");
http_conn.setRequestProperty("Authorization", authorization);
http_conn.setRequestProperty("Connection", "Keep-Alive");
http_conn.setRequestProperty("Content-Type", "multipart/form-data);
http_conn.setDoOutput(true);
http_conn.setDoInput(true);
DataOutputStream outputStream;
outputStream = new DataOutputStream(http_conn.getOutputStream());
File file_obj = new File(this.file);
byte[] allBytes = new byte[(int) file_obj.length()];
FileInputStream fileInputStream = new FileInputStream(file_obj);
outputStream.write("file=".getBytes("UTF-8")); <---Trying to add file param here
fileInputStream.read(allBytes);
outputStream.write(allBytes);
Post that I just read response using below piece of code (works fine for different GET requests):
InputStream inputStream = http_conn.getInputStream();
BufferedReader bufferedReader = new BufferedReader(new
InputStreamReader(inputStream));
String line = "";
while ((line = bufferedReader.readLine()) != null) {
data = data + line;
}
Note: I use java rarely an am not very familiar with it so please be descriptive in your response.

When looking at your curl command line, it shows that the file needs to be send as a multipart/form-data request. This is actually a complex way of formatting your data when it is requires.
An example of the format you need to send is:
Headers:
Content-Type: multipart/form-data; boundary=AaB03x
Body:
--AaB03x
Content-Disposition: form-data; name="files"; filename="file1.txt"
Content-Type: text/plain
... contents of file1.txt ...
--AaB03x--
At the moment, your code is sending the file as a POST/GET formatted request, and this doesn't work as the backend isn't expecting that.
To solve this problem, we need to format the source files into the format required by the backend, and once you know that the "boundary" header option is just a randomly generated value, it becomes more easy to send the request.
String boundary = "MY_AWESOME_BOUNDARY"
http_conn.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary);
try(DataOutputStream outputStream = new DataOutputStream(http_conn.getOutputStream())) {
File file_obj = new File(this.file);
// Write form-data header
outputStream.write(("--" + boundary + "\r\n").getBytes("UTF-8"));
outputStream.write(("Content-Disposition: form-data; name=\"file\"; filename=\"file1.txt\"\r\n").getBytes("UTF-8"));
outputStream.write(("Content-Type: text/plain\r\n").getBytes("UTF-8"));
outputStream.write(("\r\n").getBytes("UTF-8"));
// Write form-data body
Files.copy(file_obj.toPath(), outputStream)
// Write form-data "end"
outputStream.write(("--" + boundary + "--\r\n").getBytes("UTF-8"));
}
// Read backend response here
try(InputStream inputStream = http_conn.getInputStream()) {
BufferedReader bufferedReader = new BufferedReader(new
InputStreamReader(inputStream));
StringBuilder lines = new StringBuilder(); // StringBuilder is faster for concatination than appending strings
while ((line = bufferedReader.readLine()) != null) {
lines.append(line);
}
System.out.println(lines);
}
Note that I used "try-with-resource" blocks, these blocks make sure that any external resources are closed and disposed when you are done using them, generally the open resource limit of the OS is very low, compared to the amount of memory your program has, so what happens is that your program could give weird errors that only happens after some time of running or when the user executes certain actions inside your application

The above didnt worked for me so I switched to different package (okhttp3), here is what worked for me:
File file_obj = new File(this.file);
String authorization = "my authorization string";
Proxy webproxy = new Proxy(Proxy.Type.HTTP, new
InetSocketAddress("proxy", <port>));
OkHttpClient client = new OkHttpClient.Builder().proxy(webproxy).build();
RequestBody requestBody = new MultipartBody.Builder().setType(MultipartBody.FORM).addFormDataPart("file", "filename",
RequestBody.create(MediaType.parse("application/octet-stream"), file_obj)).build();
Request request = new Request.Builder().header("Authorization", authorization).url(this.url).post(requestBody).build();
try (Response response = client.newCall(request).execute()){
if(!response.isSuccessful()) return "NA";
return (response.body().string());
}

Premature end of Content-Length delimited message body (expected: 3233986; received: 0) [duplicate]

I am trying to get HTTP response with the help of apache httpclient. I get headers successfully but it throws exception when I try to get contents. Exception is:
org.apache.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 203856; received: 1070
at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:180)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:154)
at java.io.BufferedReader.readLine(BufferedReader.java:317)
at java.io.BufferedReader.readLine(BufferedReader.java:382)
and my code is:
InputStream is = entity.getContent();
BufferedReader br = new BufferedReader( new InputStreamReader(is, "UTF-8"));
String line;
String str = "";
while ((line = br.readLine()) != null) {
str = str + line + "\n";
}
log.debug(str);
any help will be appreciated. thanks

I might be replying on it late. But I also encounter the same problem. And I got the resolution of it.
In my case I was closing the client before utilizing the HttpEntity. And after closing the client I was trying to download the file. Below code is similar to what I was doing:
HttpEntity httpEntity = null;
try (final CloseableHttpClient client = createHttpClient()) {
httpEntity = getEntity(client);
}
return downloadFile(httpEntity, targetDirectory, fileName);
After adjusting my code to download the file before closing the client, Its working now for me. Below code is similar to what I did now:
try (final CloseableHttpClient client = createHttpClient()) {
HttpEntity httpEntity = getEntity(client);
return downloadFile(httpEntity, targetDirectory, fileName);
}

The problem appears to be on the server-side, not in the client code you've pasted.
The server claimed that the content contained 203856 bytes but only sent 1070.

I was closing the stream before returning it

I got the same issue while downloading the file from Jfrog Artifactory from jenkins and Changing the Nginx reverse proxy config value proxy_max_temp_file_size to 0solved my issue.

I encountered a similar error and solved it by persisting my HTTP connection for a few extra seconds. Something like
Basically add a keep-alive header on the HTTP request for something like 3-5 sec

How to output HTTP request message correctly

I use the following code to output HTTP request message. But the loop never ends after outputing HTTP request from browser. It seems that bufferReader just waiting for new bytes after outputing HTTP request.
InputStream inputStream = socket.getInputStream(); // socket is returned by
// serverSocket.accept()
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream));
// output HTTP request
String line;
while ((line = bufferedReader.readLine()) != null) { // the loop never ends
System.out.println(line);
}

I agree with mhasan. If you know ahead of time the expected length of the message, you can read that many bytes and then explicitly exit the loop.
See the code examples in this answer: https://stackoverflow.com/a/19863726/2751039

This is happening because the peer connection is never getting closed.
If it never closes, readLine() won't return null.

How to retrieve XML/RDF data from a link URL using in Java?

I have been trying to retrieve XML data from a URL and write to file on disk http://dbpedia.org/data/Berlin.rdf using the following code snippet.
URL urlObj = new URL("http://dbpedia.org/data/Berlin.rdf");
java.net.HttpURLConnection connection = (HttpURLConnection) urlObj.openConnection();
InputStream reader = new BufferedInputStream(connection.getInputStream());
BufferedReader breader = new BufferedReader(new InputStreamReader(reader));
String line;
BufferedWriter writer = new BufferedWriter(new eWriter("resource.xml"));
while ((line = breader.readLine()) != null) {
// writes the line to the output file
writer.write(line);
System.out.println(line);
}
writer.close();
connection.disconnect();
But I get this error: Exception in thread "main" java.io.IOException: Server returned HTTP response code: 502 for URL: http://dbpedia.org/data/Berlin.rdf
What is wrong ? How to fix this ? Thanks in advance.

A 502 HTTP Error is a Server Error.
If you go to the site (http://dbpedia.org/data/Berlin.rdf), you will see that dbpedia is currently undergoing maintenance. Go back in a couple of hours and try again and your code should work fine.
Update: It's working fine now.

How can I forward HttpResponse to an open client socket? (Java Program acts as a proxy)

I am using httpclient lib from apache. I managed to get an HttpResponse by sending a GET request to the server. Now what I am trying to do is to send that response that I got to a clientSocket output stream.
So basically I want to send whatever I received from the server to the open client connection. Since I am using HttpClient I get the response in the form of an HttpResponse object. I tried the following:
private void forwardRequest(String header, String url){
try {
CloseableHttpClient httpclient = HttpClients.createDefault();
HttpGet httpget = new HttpGet(url);
CloseableHttpResponse response;
//Adding the request headers to httpget
String lines[] = header.split("\\n");
for (String str : lines) {
String parts[] = str.split(":", 2);
httpget.addHeader(parts[0], parts[1]);
}
HttpResponse respone;
response = httpclient.execute(httpget);
//It works till here I can read from the response and print out the html page
//But after this I don't know how to send it to client
OutputStream bos = clientSocket.getOutputStream();
PrintWriter pw = new PrintWriter(bos);
BufferedReader rd = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
while ((line = rd.readLine()) != null) {
pw.println(line);
//bos.write(line.getBytes()); //This also doesn't work
}
response.close();
}
Also clientSocket is a global variable which is associcated with a ServerSocket like:
clientSocket = serverSocket.accept();
I don't expect a full solution. Just point me in the right direction.. Thanks a ton!
EDIT:
I tried the following based on what EJP suggested.. It's still not working. I was wondering if it was correctly implemented?
int portNumber = 8012; // port on which the program listens
ServerSocket serverSocket =
new ServerSocket(portNumber); //the socket at which the program listens
Socket clientSocket = serverSocket.accept(); //clientSocket of the program
Socket toServer = new Socket("localhost", 8089); //proxy server to which program connects
PrintWriter out =
new PrintWriter(toServer.getOutputStream(), true);
PrintWriter outClient =
new PrintWriter(clientSocket.getOutputStream(), true);
BufferedReader in = new BufferedReader(
new InputStreamReader(clientSocket.getInputStream()));
BufferedReader inServer = new BufferedReader(
new InputStreamReader(toServer.getInputStream()));
) {
String inputLine;
while ((inputLine = in.readLine()) != null) {
out.println(inputLine); //Writing to proxy server
outClient.println(inServer.readLine()); //writing back to original request sender
System.out.println(inputLine);
}

The client made an HTTP request, so it will be expecting an HTTP response. If the global clientSocket is just a raw TCP socket and not an HttpClient, then you need to add the HTTP response protocol header yourself.
You have the content from the server, you'll want to first return an HTTP response 200 OK, then empty line with carriage return + linefeed (CR+LF), then Content-length: , then the document. If you are just proxying text documents, then you could convert to a string here, but otherwise, I would just pass the mime type, charset, and entity through as the raw bytes as the web server responded, that way you can proxy any document, including images or binary files.
It will look something like this:
HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: length
<html> ...
</html>
To pass the http headers through from the server:
HttpEntity entity = response.getEntity();
// technically you should check the HTTP response rather than assume it is a 200
int statusCode = httpResp.getStatusLine().getStatusCode();
if(statusCode != 200)
... // do something with non 200 responses ?
clientSocket.write("HTTP/1.1 200 OK\r\n");
Header[] responseHeaders = response.getAllHeaders();
for(Header header : responseHeaders) {
clientSocket.write(header.toString() + "\r\n");
}
clientSocket.write("\r\n"); // empty line required
// Use BufferedInputStream to deal in bytes
BufferedInputStream input = new BufferedInputStream(entity.getContent());
byte[] buf = new byte[8192];
int bytesRead;
while ((bytesRead = input.read(buf, 8192)) > 0) {
clientSocket.write(buf, bytesRead);
}
I say "something like this", don't take this literal, I doubt it compiles. I don't have dev station in front of me, but this is the general idea.
NOTE: Since you are using the Apache client lib, you should be able to use the specific HTTP client instead of writing the raw protocol. This will abstract the HTTP protocol away somewhat. I'll update the answer later if nobody else provides a better one.

If you're just forwarding requests and responses you don't have any need to engage in the HTTP protocol at all beyond the first line of the request. If the client knows you're the proxy you will get either a GET request with the full URL or else a CONNECT request ditto. All you have to do is connect to the target and then just copy bytes in both directions simultaneously.