I am trying to call a REST API with a PUT Request but I am receiving a 400 Error Code (Bad Request). Can someone spot what I may be doing wrong?
I have successfully called this API with a REST Client, here are the headers and body used:
https://imgur.com/dZVyawn
https://imgur.com/lMtn2JB
String credentials = Base64.getEncoder().encodeToString(("wcadmin:wcadmin").getBytes());
URL url = new URL(getURL());
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("PUT");
connection.setDoOutput(true);
connection.setDoInput(true);
//Set Headers
String fileUrl = "c:\\0000000050.xml";
File fileToUpload = new File(fileUrl);
long length = fileToUpload.length();
String FORM_DATA_BOUNDARY = "------FormBoundary" + System.currentTimeMillis();
connection.setRequestProperty("csrf_nonce", getNonceValue());
connection.setRequestProperty("Accept", "application/xml");
connection.setRequestProperty("Authorization", "Basic " + credentials);
connection.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + FORM_DATA_BOUNDARY);
connection.setRequestProperty("Content-Length", Long.toString(length));
//Setup Request Body Writer
OutputStream requestBodyOutputStream = connection.getOutputStream();
BufferedWriter requestBodyWriter = new BufferedWriter(new OutputStreamWriter(requestBodyOutputStream));
//Write Body
requestBodyWriter.write("\r\n\r\n");
requestBodyWriter.write(FORM_DATA_BOUNDARY);
requestBodyWriter.write("\r\n");
requestBodyWriter.write("Content-Disposition: form-data; name=\"file\"; filename=\"" + fileUrl + "\"");
requestBodyWriter.write("\r\n");
requestBodyWriter.write("Content-Type: text/xml");
requestBodyWriter.write("\r\n\r\n");
requestBodyWriter.flush();
FileInputStream uploadFileStream = new FileInputStream(fileToUpload);
int bytesRead;
byte[] dataBuffer = new byte[1024];
while ((bytesRead = uploadFileStream.read(dataBuffer)) != -1) {
requestBodyOutputStream.write(dataBuffer, 0, bytesRead);
}
requestBodyOutputStream.flush();
requestBodyWriter.write("\r\n");
requestBodyWriter.write(FORM_DATA_BOUNDARY);
requestBodyWriter.flush();
//Close the streams
requestBodyOutputStream.close();
requestBodyWriter.close();
uploadFileStream.close();
//Read Response
String inputLine;
StringBuffer content = new StringBuffer();
InputStream inputStream = connection.getInputStream();
if (inputStream != null) {
BufferedReader responseReader = new BufferedReader(new InputStreamReader(inputStream));
if (responseReader != null) {
while ((inputLine = responseReader.readLine()) != null) {
content.append(inputLine);
}
responseReader.close();
}
}
connection.disconnect();
Error 400 Bad Request response received
First and most important: you cannot write to both an OutputStream, and a OutputStreamWriter which wraps that same OutputStream. They will conflict with each other.
Do not use OutputStreamWriter at all; instead, convert text to bytes yourself:
OutputStream requestBodyOutputStream = connection.getOutputStream();
requestBodyOutputStream.write("\r\n\r\n".getBytes(StandardCharsets.UTF_8));
requestBodyOutputStream.write(FORM_DATA_BOUNDARY.getBytes(StandardCharsets.UTF_8));
// etc.
Second, you are converting between bytes and strings using the system’s default charset, which means exactly what gets written depends on the system where the code is running. Don’t call String.getBytes without specifying an explicit Charset. Usually getBytes(StandardCharsets.UTF_8) is what you want.
Similarly, you need to pass a Charset to your InputStreamReader creation (although this isn’t the cause of your problem, since you aren’t getting a valid response at the moment). Don’t assume a charset; use the charset MIME type parameter from the response’s Content-Type header. If you’re using a version of Java older than 11, you can parse the Content-Type value with the javax.activation.MimeType class, but be aware that the javax.activation package has been removed from Java SE as of Java 11. For Java 11 and later, Java Activation can be downloaded as a stand-alone library. Another option is to use the JavaMail library, specifically its ContentType class, for parsing.
Third, the Content-Length header inside the body part (between the boundaries) should use the file’s length as a Content-Length. The Content-Length of the entire request body must be the length of everything you’ve written: the boundaries, the body part headers, and the file content.
The good news is, I think (though I’m not positive) that URLConnection will set the request’s overall Content-Length automatically, based on the bytes you write, so you probably don’t need to compute the length yourself; you can simply refrain from setting "Content-Length" at all.
When you do pass a correct request, you will find that you are dropping the newlines in the response. If the response is supposed to be human-readable text, those newlines are likely to matter. If you’re using Java 10 or later, you can use Reader.transferTo with a StringWriter:
StringWriter responseBody = new StringWriter();
responseReader.transferTo(responseBody);
String content = responseBody.toString();
If you’re using a version of Java older than 10:
new BufferedReader cannot return null, so checking for null is pointless. In Java, the new operator always, no matter what, returns a new object (unless an exception is thrown, in which case new doesn’t return at all).
You should use StringBuilder, not StringBuffer. They are identical except that StringBuffer is an older class that provides thread safety for every method, creating unnecessary overhead for nearly all use cases.
You are copying your file into the request without any buffering, which is going to be slow and inefficient. Consider using Files.copy(fileToUpload.toPath(), requestBodyOutputStream) instead.
Related
I am trying to upload (POST) a file to an endpoint using java.net.HttpURLConnection but I keep getting http code 400 (bad request).
I refered to Send File And Parameters To Server With HttpURLConnection in android API 23
but problem is that I need to send this file as request body param (file=).
Note: The files will be of small size only (4-5mb) so I am reading it entirely in memory.
Corresponding curl request is:
curl -X POST "API" -H "Content-Type: multipart/form-data" -F "file="
Excerpts of Code that I am using:
Proxy webproxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("
<proxy host>", <proxy_port>));
HttpURLConnection http_conn = (HttpURLConnection)
url.openConnection(webproxy);
String authorization = getAuthorization(access_token);
http_conn.setRequestMethod("POST");
http_conn.setRequestProperty("Accept-Charset", "UTF-8");
http_conn.setRequestProperty("Authorization", authorization);
http_conn.setRequestProperty("Connection", "Keep-Alive");
http_conn.setRequestProperty("Content-Type", "multipart/form-data);
http_conn.setDoOutput(true);
http_conn.setDoInput(true);
DataOutputStream outputStream;
outputStream = new DataOutputStream(http_conn.getOutputStream());
File file_obj = new File(this.file);
byte[] allBytes = new byte[(int) file_obj.length()];
FileInputStream fileInputStream = new FileInputStream(file_obj);
outputStream.write("file=".getBytes("UTF-8")); <---Trying to add file param here
fileInputStream.read(allBytes);
outputStream.write(allBytes);
Post that I just read response using below piece of code (works fine for different GET requests):
InputStream inputStream = http_conn.getInputStream();
BufferedReader bufferedReader = new BufferedReader(new
InputStreamReader(inputStream));
String line = "";
while ((line = bufferedReader.readLine()) != null) {
data = data + line;
}
Note: I use java rarely an am not very familiar with it so please be descriptive in your response.
When looking at your curl command line, it shows that the file needs to be send as a multipart/form-data request. This is actually a complex way of formatting your data when it is requires.
An example of the format you need to send is:
Headers:
Content-Type: multipart/form-data; boundary=AaB03x
Body:
--AaB03x
Content-Disposition: form-data; name="files"; filename="file1.txt"
Content-Type: text/plain
... contents of file1.txt ...
--AaB03x--
At the moment, your code is sending the file as a POST/GET formatted request, and this doesn't work as the backend isn't expecting that.
To solve this problem, we need to format the source files into the format required by the backend, and once you know that the "boundary" header option is just a randomly generated value, it becomes more easy to send the request.
String boundary = "MY_AWESOME_BOUNDARY"
http_conn.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary);
try(DataOutputStream outputStream = new DataOutputStream(http_conn.getOutputStream())) {
File file_obj = new File(this.file);
// Write form-data header
outputStream.write(("--" + boundary + "\r\n").getBytes("UTF-8"));
outputStream.write(("Content-Disposition: form-data; name=\"file\"; filename=\"file1.txt\"\r\n").getBytes("UTF-8"));
outputStream.write(("Content-Type: text/plain\r\n").getBytes("UTF-8"));
outputStream.write(("\r\n").getBytes("UTF-8"));
// Write form-data body
Files.copy(file_obj.toPath(), outputStream)
// Write form-data "end"
outputStream.write(("--" + boundary + "--\r\n").getBytes("UTF-8"));
}
// Read backend response here
try(InputStream inputStream = http_conn.getInputStream()) {
BufferedReader bufferedReader = new BufferedReader(new
InputStreamReader(inputStream));
StringBuilder lines = new StringBuilder(); // StringBuilder is faster for concatination than appending strings
while ((line = bufferedReader.readLine()) != null) {
lines.append(line);
}
System.out.println(lines);
}
Note that I used "try-with-resource" blocks, these blocks make sure that any external resources are closed and disposed when you are done using them, generally the open resource limit of the OS is very low, compared to the amount of memory your program has, so what happens is that your program could give weird errors that only happens after some time of running or when the user executes certain actions inside your application
The above didnt worked for me so I switched to different package (okhttp3), here is what worked for me:
File file_obj = new File(this.file);
String authorization = "my authorization string";
Proxy webproxy = new Proxy(Proxy.Type.HTTP, new
InetSocketAddress("proxy", <port>));
OkHttpClient client = new OkHttpClient.Builder().proxy(webproxy).build();
RequestBody requestBody = new MultipartBody.Builder().setType(MultipartBody.FORM).addFormDataPart("file", "filename",
RequestBody.create(MediaType.parse("application/octet-stream"), file_obj)).build();
Request request = new Request.Builder().header("Authorization", authorization).url(this.url).post(requestBody).build();
try (Response response = client.newCall(request).execute()){
if(!response.isSuccessful()) return "NA";
return (response.body().string());
}
I'm having some encoding problems in a Java application that makes HTTP requests to an IIS server.
Iterating over the headers of the URLConnection object I can see the following (relevant) headers:
Transfer-Encoding: [chunked]
Content-Encoding: [utf-8]
Content-Type: [text/html; charset=utf-8]
The URLConnection.getContentEncoding() method returns utf-8 as the document encoding.
This is how my HTTP request, and stream read is being made:
OutputStreamWriter sw = null;
BufferedReader br = null;
char[] buffer = null;
URL url;
url = new URL(this.URL);
URLConnection connection = url.openConnection();
connection.setDoOutput(true);
sw = new OutputStreamWriter(connection.getOutputStream());
sw.write(postData);
sw.flush();
br = new BufferedReader(new InputStreamReader(connection.getInputStream(), "UTF8"));
StringBuilder totalResponse = new StringBuilder();
String line;
while((line = br.readLine()) != null) {
totalResponse.append(line);
}
buffer = totalResponse.toString().toCharArray();
if (sw != null)
sw.close();
if (br != null)
br.close();
return buffer;
However the following string sent by the server "ÃÃÃção" is received by the client as "�����o".
What am I doing wrong ?
Based on your comments, you are trying to receive a FIX message from an IIS server and FIX uses ASCII. There are only a small subset of tags which support other encoding and they have to be treated in a special manner (non-ASCII tags in the standard FIX spec are 349,351,353,355,357,359,361,363,365). If such tags are present, you will get a tag 347 with a value specifying the encoding (for example UTF-8) and then each tag, will be preceded by a tag giving you the length of the coming encoded value (for tag 349, you will always get 348 first with an integer value)
In your case, it looks like the server is sending a custom tag 10411 (the 10xxx range) in some other encoding. By convention, the preceding tag 10410 should give you the length of the value in 10411, but it contains "0000" instead, which may have some other meaning.
Note that although FIX message are very readable, they should still be treated as binary data. Tags and values are mostly ASCII characters, but the delimiter (SOH) is 0x01 and as mentioned above, certain tags may be encoded with another encoding. The IIS service should really return the data as application/octet-stream so it can be received properly. Attempting to return it as text/html is asking for trouble :).
If the server really sends a Content-Encoding of "UTF-8" then it is very confused. See http://svn.tools.ietf.org/svn/wg/httpbis/specs/rfc7231.html#header.content-encoding
For good order a couple of corrections.
URLConnection connection = url.openConnection();
connection.setDoOutput(true);
connection.connect();
try (Writer sw = new OutputStreamWriter(connection.getOutputStream(),
StandardCharsets.UTF_8)) {
sw.write(postData);
sw.flush();
try (BufferedReader br = new BufferedReader(
new InputStreamReader(connection.getInputStream(),
StandardCharsets.UTF_8))) {
StringBuilder totalResponse = new StringBuilder();
String line;
while ((line = br.readLine()) != null) {
totalResponse.append(line).append("\r\n");
}
return totalResponse.toString().toCharArray();
} // Close br.
} // Close sw.
Maybe:
postData = ... + "Accept-Charset: utf-8\r\n" + ...;
Receiving the totalResponse.toString() you should have all read correctly.
But then when displaying again, the String/char is again converted to bytes, and there the encoding fails. For instance System.out.println will not do as probably the Windows encoding is used.
You can test the String by dumping its bytes:
String s = totalResponse.toString();
Logger.getLogger(getClass().getName()).log(Level.INFORMATION, "{0}",
Arrays.toString(s.getBytes(StandardCharsets.UTF_8)));
In some rare cases the font will not contain the special characters.
Can you try by putting the stream as part of request attribute and then printing it out on client side. a request attribute will be received as is withou any encoding issues
Trying to read a generated XML from a MS Webservice
URL page = new URL(address);
StringBuffer text = new StringBuffer();
HttpURLConnection conn = (HttpURLConnection) page.openConnection();
conn.connect();
InputStreamReader in = new InputStreamReader((InputStream) conn.getContent());
BufferedReader buff = new BufferedReader(in);
box.setText("Getting data ...");
String line;
do {
line = buff.readLine();
text.append(line + "\n");
} while (line != null);
box.setText(text.toString());
or
URL u = new URL(address);
URLConnection uc = u.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(uc.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null) {
inputLine = java.net.URLDecoder.decode(inputLine, "UTF-8");
System.out.println(inputLine);
}
in.close();
Any page reads fine except the web service output
it reads the greater and less than signs strangely
it read < to "& lt;" and > to "& gt;" without spaces, but if i type them here without spaces stackoverflow makes them < and >
Please help
thanks
First there seem to be a confusion on this row:
inputLine = java.net.URLDecoder.decode(inputLine, "UTF-8");
This effectively says that you expect every row in the document that your server is providing to be URL encoded. URL encoding is not the same as document encoding.
http://en.wikipedia.org/wiki/Percent-encoding
http://en.wikipedia.org/wiki/Character_encoding
Looking at your code snippet, I think URL encoding (percent encoding) is not what you're after.
In terms of document character encoding. You are making a conversion on this line:
InputStreamReader in = new InputStreamReader((InputStream) conn.getContent());
conn.getContent() returns an InputStream that operates on bytes, whilst the reader operates on chars - the character encoding conversion is done here. Checkout the other constructors of InputStreamReader which takes the encoding as second argument. Without the second argument you are falling back on whatever is your platform default in java.
InputStreamReader(InputStream in, String charsetName)
for instance lets you change your code to:
InputStreamReader in = new InputStreamReader((InputStream) conn.getContent(), "utf-8");
But the real question will be "what encoding is your server providing the content in?" If you own the server code too, you may just hard code it to something reasonable such as utf-8. But if it can vary, you need to look at the http header Content-Type to figure it out.
String contentType = conn.getHeaderField("Content-Type");
The contents of contentType will look like
text/plain; charset=utf-8
A short hand way of getting this field is:
String contentEncoding = conn.getContentEncoding();
Notice that it's entirely possible that no charset is provided, or no Content-Type header, in which case you must fall back on reasonable defaults.
Mark Rotteveel is correct, the webservice is the culprit here it's for some reason sending the greater than and less than sign with the & lt and & gt format
Thanks Martin Algesten but i have already stated i worked around it i was just looking for why it was this way.
Hey, I've tried researching how to POST data from java, and nothing seems to do what I want to do. Basically, theres a form for uploading an image to a server, and what I want to do is post an image to the same server - but from java. It also needs to have the right parameter name (whatever the form input's name is). I would also want to return the response from this method.
It baffles me as to why this is so difficult to find, since this seems like something so basic.
EDIT ---- Added code
Based on some of the stuff BalusC showed me, I created the following method. It still doesn't work, but its the most successful thing I've gotten yet (seems to post something to the other server, and returns some kind of response - I'm not sure I got the response correctly though):
EDIT2 ---- added to code based on BalusC's feedback
EDIT3 ---- posting code that pretty much works, but seems to have an issue:
....
FileItemFactory factory = new DiskFileItemFactory();
// Create a new file upload handler
ServletFileUpload upload = new ServletFileUpload(factory);
// Parse the request
List<FileItem> items = upload.parseRequest(req);
// Process the uploaded items
for(FileItem item : items) {
if( ! item.isFormField()) {
String fieldName = item.getFieldName();
String fileName = item.getName();
String itemContentType = item.getContentType();
boolean isInMemory = item.isInMemory();
long sizeInBytes = item.getSize();
// POST the file to the cdn uploader
postDataRequestToUrl("<the host im uploading too>", "uploadedfile", fileName, item.get());
} else {
throw new RuntimeException("Not expecting any form fields");
}
}
....
// Post a request to specified URL. Get response as a string.
public static void postDataRequestToUrl(String url, String paramName, String fileName, byte[] requestFileData) throws IOException {
URLConnection connection=null;
try{
String boundary = Long.toHexString(System.currentTimeMillis()); // Just generate some unique random value.
String charset = "utf-8";
connection = new URL(url).openConnection();
connection.setDoOutput(true);
connection.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary);
PrintWriter writer = null;
OutputStream output = null;
try {
output = connection.getOutputStream();
writer = new PrintWriter(new OutputStreamWriter(output, charset), true); // true = autoFlush, important!
// Send binary file.
writer.println("--" + boundary);
writer.println("Content-Disposition: form-data; name=\""+paramName+"\"; filename=\"" + fileName + "\"");
writer.println("Content-Type: " + URLConnection.guessContentTypeFromName(fileName));
writer.println("Content-Transfer-Encoding: binary");
writer.println();
output.write(requestFileData, 0, requestFileData.length);
output.flush(); // Important! Output cannot be closed. Close of writer will close output as well.
writer.println(); // Important! Indicates end of binary boundary.
// End of multipart/form-data.
writer.println("--" + boundary + "--");
} finally {
if (writer != null) writer.close();
if (output != null) output.close();
}
//* screw the response
int status = ((HttpURLConnection) connection).getResponseCode();
logger.info("Status: "+status);
for (Map.Entry<String, List<String>> header : connection.getHeaderFields().entrySet()) {
logger.info(header.getKey() + "=" + header.getValue());
}
} catch(Throwable e) {
logger.info("Problem",e);
}
}
I can see this code uploading the file, but only after I shutdown the tomcat. This leads me to believe that I'm leaving some sort of connection open.
This worked!
The core API you'd like to use is java.net.URLConnection. This is however pretty low level and verbose. You'd like to learn about the HTTP specifics in detail and take them into account (headers, etcetera). You can find here a related question with lot of examples.
A more convenient HTTP client API is the Apache Commons HttpComponents Client. You can find an example here.
Update: as per your update: you should read the response as a character stream, not as a binary stream and attempt to cast a byte to a char. This ain't going to work. Head to the Gathering HTTP response information part in the linked question with examples. Here's how it should look like:
BufferedReader reader = null;
StringBuilder builder = new StringBuilder();
try {
reader = new BufferedReader(new InputStreamReader(response, charset));
for (String line; (line = reader.readLine()) != null;) {
builder.append(line);
}
} finally {
if (reader != null) try { reader.close(); } catch (IOException logOrIgnore) {}
}
return builder.toString();
Update 2: as per your second update. Seeing the way how you continue to attampt reading/writing streams, I think it's high time to learn the basic Java IO :) Well, this part is also answered in the linked question. You would like to use Apache Commons FileUpload to parse a multipart/form-data request in a servlet. How to use it is also described/linked in the linked question. Look at the bottom of the Uploading files chapter. By the way, the content length header would return zero since you are not explicitly setting it (and also cannot do without buffering the entire request in memory).
Update 3:
I can see this code uploading the file, but only after I shutdown the tomcat. This leads me to believe that I'm leaving some sort of connection open.
You need to close the OutputStream with which you wrote the file to the disk. Once again, read the above linked basic Java IO tutorial.
What have you tried? If you google for Http Post Java, dozens of pages appear - what's wrong with them? This one, http://www.devx.com/Java/Article/17679/1954 for example, appears pretty decent.
I'm having problems with downloading binary file (zip file) in my app from te internet. I have to use basic access authentication to authorize acces to file, but server response is always HTTP/1.0 400 Bad request.
String authentication = this._login+":"+this._pass;
String encoding = Base64.encodeToString(authentication.getBytes(), 0);
String fileName = "data.zip";
URL url = new URL("http://10.0.2.2/androidapp/data.zip");
HttpURLConnection ucon = (HttpURLConnection) url.openConnection();
ucon.setRequestMethod("GET");
ucon.setDoOutput(true);
ucon.setRequestProperty ("Authorization", "Basic " + encoding);
ucon.connect();
/*
* Define InputStreams to read from the URLConnection.
*/
InputStream is = ucon.getInputStream();
BufferedInputStream bis = new BufferedInputStream(is);
/*
* Read bytes to the Buffer until there is nothing more to read(-1).
*/
ByteArrayBuffer bab = new ByteArrayBuffer(50);
int current = 0;
while ((current = bis.read()) != -1) {
bab.append((byte) current);
}
bis.close();
/* Convert the Bytes read to a String. */
FileOutputStream fos = this._context.openFileOutput(fileName, this._context.MODE_WORLD_READABLE);
fos.write(bab.toByteArray());
fos.close();
Could it be caused by whitespaces in password?
I might be a bit late but I just came across a similar problem.
The problem lies in the following line:
String encoding = Base64.encodeToString(authentication.getBytes(), 0);
If you change that line to look like this it should work:
String encoding = Base64.encodeToString(authentication.getBytes(), Base64.NO_WRAP);
By default the Android Base64 util adds a newline character to the end of the encoded string. This invalidates the HTTP headers and causes the "Bad request".
The Base64.NO_WRAP flag tells the util to create the encoded string without the newline character thus keeping the HTTP headers intact.