I try to talk to a server, by telneting to it, and send the following command through telnet terminal :
POST /%5bvUpJYKw4QvGRMBmhATUxRwv4JrU9aDnwNEuangVyy6OuHxi2YiY=%5dImage? HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Content-Length: 164
[SORT]=0,1,0,10,5,0,KL,0&[FIELD]=33,38,51,58,68,88,78,98,99,101,56,57,69,70,71,72,89,90,91,92,59,60,61,62,79,80,81,82&[LIST]=1155.KL,1295.KL,7191.KL,0097.KL,2267.KL
This works very fine. Now, I wish I can use HttpClient, to talk to the server, as I use telnet to talk to the server. The reason I wish to use HttpClient, instead of using raw TCP socket, is because HttpClient does support NTLM.
However, when I use POST method with NameValuePair :
new NameValuePair("[SORT]", "0,1,0,10,5,0,KL,0")
The request will become URL encoded. The server doesn't understand URL encoded request.
%5BSORT%5D: 0%2C1%2C0%2C10%2C5%2C0%2CKL%2C0
Is there any way I can avoid this? If not, what is the alternative library I can use? I wish to support NTLM as well.
As I mentioned in the other thread, this is not even valid HTTP POST. So you can't do it with default post mechanism in HttpClient. You need to make the invalid body yourself and post it.
Assuming you are using HttpClient 3, following code should work,
HttpClient httpClient = new HttpClient();
PostMethod method = new PostMethod(url);
String badFormPost = "[SORT]=0,1,0,10,5,0,KL,0&[FIELD]=33,38,51,58,68,88,78,98,99,101,56,57,69,70,71,72,89,90,91,92,59,60,61,62,79,80,81,82&[LIST]=1155.KL,1295.KL,7191.KL,0097.KL,2267.KL";
RequestEntity entity = new StringRequestEntity(badFormPost,
"application/x-www-form-urlencoded", "UTF-8");
method.setRequestEntity(entity);
method.setContentChunked(false);
httpClient.executeMethod(method);
...
It's getting URL encoded because your request formed with HTTPClient may be a GET request instead of a POST and is missing this header:
Content-Type: application/x-www-form-urlencoded
Look for the HTTPClient setting to set the Content-Type header correctly and make sure your request is a POST, not a get and you should be golden.
Related
I'm trying to access an Amazon Prometheus Server but am getting an authentication error when the request is done through a Java API, although it works on Postman when the request is the same.
Note: The Authentication setup currently is done through AWS Signature Version 4 Signing (https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html)
Here are the logs from Prometheus, showing all Request Headers used (hiding some potentially sensitive info)
GET https://{resource}-workspaces.us-east-1.amazonaws.com/XXXXXXXXXXX/api/v1/query
Host: {resource}-workspaces.us-east-1.amazonaws.com
X-Amz-Security-Token: FwoGZXIvYXdzEB4aDPxYxIKgd5JznR9gOyKkAkoDbjeTi79mRDgU6Hdd2AGlLwKnNGySAkNYwmKItTcSssS9zNZ+/s..........................................==
X-Amz-Date: 20221207T201649Z
Authorization: AWS4-HMAC-SHA256 Credential={AccessKey}/20221207/us-east-1/{Resource}/aws4_request, SignedHeaders=host;x-amz-date;x-amz-security-token, Signature=XXXXXXXXXXXXXXXXXXXXXXXXXX
User-Agent: PostmanRuntime/7.29.0
Accept: `*`/`*`
Cache-Control: no-cache
Postman-Token: ea3f71dd-db52-4869-a48a-2cf5f72d15b3`
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
If I copy almost this same Request using a Java API, it gives a 403 error.
Note: I'm making sure to include the X-Amz-Security-Token since this request needs a session token.
try {
final URL url = new URL("https://{resource}-workspaces.us-east-1.amazonaws.com/XXXXXXXXXXX/api/v1/query");
final HttpURLConnection httpConn = (HttpURLConnection) url.openConnection();
httpConn.setRequestMethod("GET");
httpConn.setRequestProperty("Host", "{resource}-workspaces.us-east-1.amazonaws.com");
httpConn.setRequestProperty("X-Amz-Security-Token", "FwoGZXIvYXdzEB4aDPxYxIKgd5JznR9gOyKkAkoDbjeTi79mRDgU6Hdd2AGlLwKnNGySAkNYwmKItTcSssS9zNZ+/s..........................................==");
httpConn.setRequestProperty("X-Amz-Date", "20221207T201649Z");
httpConn.setRequestProperty("Authorization", "AWS4-HMAC-SHA256 Credential={AccessKey}/20221207/us-east-1/{Resource}/aws4_request, SignedHeaders=host;x-amz-date;x-amz-security-token, Signature=XXXXXXXXXXXXXXXXXXXXXXXXXX");
httpConn.setRequestProperty("User-Agent", "Mozilla/5.0");
httpConn.setRequestProperty("Accept", "`*`/`*`");
httpConn.setRequestProperty("Cache-Control", "no-cache");
httpConn.setRequestProperty("Postman-Token", "ea3f71dd-db52-4869-a48a-2cf5f72d15b3");
httpConn.setRequestProperty("Accept-Encoding", "gzip, deflate, br");
httpConn.setRequestProperty("Connection", "keep-alive");
responseCode = httpConn.getResponseCode(); // 403 here
}
Things I've attempted:
Creating my own Signature instead of reusing the same made from the Postman request
Including/excluding some header properties set including: User-Agent, Accept, Cache-Control, Postman-Token, Accept-Encoding, Connection.
Any ideas on what could be leading to this 403 response?
Thanks
I tried to make a request to an AWS service using Signature Version 4 Signing - expecting a 200 response but getting a 403.
I just figured out the answer to this and wanted to share in case anyone else comes across this issue.
A couple of things.
First, if you copy the same request generated from Postman it will not work. This is because AWS Signature Version 4 generates a unique Signature after every request. So if you copy this "Authentication" header field, it's going to give you a 403. That leads me to...
Second, you need to generate your own signature through Amazon's four-step process. This can get very confusing if try to code it on your own. I would suggest trying two things. Either use AWS4Signer and utilize the sign method or follow the example AWS provides (this is what I ultimately chose).
Download that Zip and check out the GET example. Make sure you put the "x-amz-security-token" on the header if your request requires a session. And lastly, make sure your URL includes the encoded query parameter. This is what I spent most of my time on. My URL/query parameters looked something like this -
String notEncodedQueryParameter = "avg(process_runtime_jvm_classes_current_loaded{stack_name=\"dev\"})";
String encodedQueryParameter = URLEncoder.encode(notEncodedQueryParameter, "UTF-8");
new URL("https://aps-workspace-amazonaws.com/api/v1/query?query="+ encodedQueryParameter);
final Map<String, String> queryParameters = new HashMap<>();
queryParameters.put("query", notEncodedQueryParameter);
Then later I passed my query parameters to get encoded as part of the authorization signature being generated.
String authorization = signer.computeSignature(headers,
queryParameters,
AWS4SignerBase.EMPTY_BODY_SHA256,
awsAccessKey,
awsSecretKey);
I really recommend using that second link because if you don't get the correct response, it will give you a useful error message like
the request signature we calculated does not match the signature you
provided canonical request has different parameter encoding
Instead of just leaving you with a 403 error.
Best of luck.
I'm trying to read a JSON response from a RESTful webserver running on an IoT module (Advantech WISE-4012). According to the documentation, any GET request should be made in this form
GET /ai_value/slot_0/ch_0
Any Java implementation of GET requests (Java libraries, Apache etc.), anyway, append to the end of the request the protocol signature HTTP/1.1. E.g:
GET http://192.168.0.14/ai_value/slot_0/ch_0 HTTP/1.1
Because of this (probably) i'm getting Error 400 (Bad request) on every client i tried so far. The only working method i've discovered was sending a simple request through the address bar on Google Chrome browser (sometimes i get a response, sometimes a get a bad request error either). How can i write a java implementation of a GET request plain and simple as described by the documentation? How can i test a custom GET request without HTTP/1.1 at the end? Every chrome extension i tried (Advanced REST Client, Postman) add the protocol version at the end, so i haven't had the chance to verify if that's why i'm getting a bad request error.
EDIT:
This is the response header from Advanced REST client
Connection: close
Content-Type: application/json
Server: WISE-4000/8.1.0020
While the source message is the following one:
GET /ai_value/slot_0/ch_0 HTTP/1.1
HOST: 192.168.0.14
The only mismatch between the documentation is the HTTP/1.1 signature as mentioned before. Adding the "accept: application/json" makes no difference either
After a bit of digging into the documentation, it looks like the default timeout (i.e. 720 seconds) is the one causing an issue. There doesn't seem to be any way to work it around (ideally, the system should reset the time after a successful request and we should only get 400 - or 403 ideally after 720 seconds of inactivity).
A couple of points I would like to recommend to the API developers for WISE-4012 (if they are in touch with you):
Add brief documentation for authentication and timeout (probably, more response codes and error messages with each error response)
Enable OAuth for API Access
As far as current implentation is conerned, I guess you need to do a basic auth and pass username/password with every request, Or add Authentication header with every API request to get successful response without any 400s.
Check if this helps.
HttpClient httpClient = HttpClients.createDefault();
URI reqUri = new URI(<uri>);
RequestBuilder requestBuilder = RequestBuilder.create("GET");
requestBuilder.setUri(reqUri);
requestBuilder.setHeader(<headerKey>, <headerValue>);
requestBuilder.setEntity(<entity_data>);
HttpUriRequest httpRequest = requestBuilder.build();
httpResponse = httpClient.execute(httpRequest);
please read my post:
I need to post the image to the JSON WS with this parameters:
Content-Type: multipart/related; boundary="foo_bar_baz"
Content-Length: {number_of_bytes_in_entire_request_body} -- Check your rest client API's. Some would automatically determine content-length at runtime. Eg. jersey client api.
--foo_bar_baz
Content-Type: application/json;
{
"filename": "cloudx.jpg"
}
--foo_bar_baz
Content-Type: image/jpeg
{JPEG data}
--foo_bar_baz--
I'm building Android application and I need to write the request to send the image to the above WS. I was looking around for some time and I didn't found good resource to study this issue I have.
following may give you basic idea
HttpClient client = new DefaultHttpClient();
HttpPost post = new HttpPost(<URI>);
HttpEntity entity = new FileEntity(file,ContentType.MULTIPART_FORM_DATA);
//there are other types of entities and content types too check documentation
req.setEntity(entity);
HttpResponse response = httpClient.execute(req);
I am making an HttpGet to an url and I do not want the server to send the data gzipped. What header should I include in my HttpGet ?
With the default headers, the server sends gzipped data from time to time. I don't want this to happen. Thanks.
You want the Accept-Encoding HTTP request header.
Update: per #Selvin's comment, leave it empty or set it to "identity".
Update: The web application has to cooperate properly to be HTTP compliant, of course. If it's not honoring Accept-Encoding, look at its Content-Encoding HTTP response header. If it's "gzip", just read the response body with Java's GZIPInputStream.html. Then add "gzip" to your Accept-Encoding request header, since your client now handles GZIP. If the web application doesn't set the Content-Encoding header properly, that's another story altogether.
You should set the Accept-Encoding header to identity.
You could try to change the Accept-Encoding header, by removing the gzip|deflate value. If this doesn't work, you should also take into account that server doesn't care if the client supports the gzipped content (which is a bug and should be fixed).
I am curios about how one can set the request properties for a PostMethod in Apache Commons HttpClient?
I am refactoring some code written using HttpURLConnection class to post which looks like the following:
conn1.setRequestProperty(
"Content-Type", "multipart/related; type=\"application/xml\"; boundary="
+ boundary);
conn1.setRequestProperty("Authorization", auth);
... ...
To use:
PostMethod method = new PostMethod(_Server);
method.setRequestBody(...); or
method.setRequestHeader(...);
But i am not sure if / how this will map to what i want to do with the original URL class... can anyone help clarify how to set request properties with PostMethod class?
Thanks a lot!
-alex
Those are both request headers, so you would need to call setRequestHeader() to establish those values on the connection. HttpClient also supports handling basic authentication so the "Authorization" header can be refactored out, depending on how deep your changes go.