I am rewriting example code for access to the API of a web store from Java into classic ASP. The Java code works, it sends requests and the right results are returned. The ASP code does exactly the same but all it does is return an error that there is something wrong with the authorization. I have checked the hmac signature string (sha256, base64) but this gives the same results as the Java code.
So then I decided to check how the HTTP requests actually look, and compare them with each other. Fiddler seemed to be a good choice, but for some reason both HTTP request dont show up.
A lot has already been written on this subject and i tried several things, but so far no luck. They all describe routing the request through the Fiddler proxy, which is localhost:8888.
I use this object in the ASP code:
Set httpRequest = Server.CreateObject("MSXML2.ServerXMLHTTP.6.0")
For IIS 7 (on Windows 7) the proxy can be set in DOS like this:
netsh winhttp set proxy localhost:8888
And in the code like this (this may not work in other version of "MSXML2.ServerXMLHTTP" than v6):
httpRequest.setProxy 2, "http=localhost:8888", ""
The Java code uses Apache HttpClient, and according to the manual the proxy can be set like this:
DefaultHttpClient httpclient = new DefaultHttpClient();
HttpHost proxy = new HttpHost("localhost", 8888);
httpclient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy);
I added this code at the beginning of the main() method. The code is run inside Eclipse (Juno).
I also opened the proxy address (http://localhost:8888/) in a browser, and it shows me the 'Fiddler Echo Service' page. Fiddler is set to show 'All Processes'.
Still, none of two HTTP Request show up. The only thing that does show up is the ASP page I run (http://localhost:8082/test.asp) that does the HTTP request. When I use my browser other things show up on Fiddler, so I know it is working.
Anybody got any ideas?
I originally thought that because nothing showed up on Fiddler with both programs, something was wrong with Fiddler.
But things were wrong with both programs. For the ASP code, setProxy has to called like this:
xmlhttp.setProxy 2, "http://127.0.0.1:8888", ""
So "::/" instead of "="
Making Apache HttpClient requests in the Java code show up in Fiddler was very complicated because request were send using https protocol. Although I tried several proposed solutions, I could not get it to work.
In the end I found out that adding
-Djavax.net.debug=all
as an argument for the VM showed me the what I needed, the headers that were being send.
Related
What differentiate these 2 requests that cause them to have different results/responses from the server although they should be the same ?
Request initiated by Chrome after a simple
click/navigation(successful, response code is 302)
I simply copied
that request as a curl and imported it to Postman and then postman
hanged
I did the same with Java - HttpUrlConnection(mimicking all the request headers and cookies like Chrome sent), but it hanged and waited forever. Is this simply because of the server logic that doesn't accept non-browser client ?
Here are the steps that I tried:
1. Visited this link: https://www.tokopedia.com/p/handphone-tablet/handphone
2. I opened the inspector and opened the Network - All tab
3. I clicked one of the products
4. I clicked the top request from the Network - All tab
5. I copied it as cURL bash
6. I imported it to Postman
7. I ran that request
8. Postman hanged
Actually the problem might even go deeper than what the other answers say.
So neither the User-Agent request header nor telnet might solve that problem (unless you initialize the TLS handshake also with telnet MANUALLY, but that is near impossible to complete).
TLS fingerprinting
If the connection is an SSL/TLS connection, the server could detect which algorithm is used to generate keys, and most applications have their specific signature / cipher.
So only by the TLS handshake alone you can tell Chrome from Postman or FireFox or Java. Java usually - unless a JVM implementation REALLY wants to go off-road - has the same signature across all platforms, using the same cipher/algorithm across all implementations.
I am sorry I cannot properly recall the name of this technique. The first project I know that published this is called something like "A3" or "S3". Salesforce published an article about JA3 analysis. They describe the technique and show a list of signatures and applications so you can guesstimate what app you're talking to, without the need to even decrypt the data: https://engineering.salesforce.com/tls-fingerprinting-with-ja3-and-ja3s-247362855967
My Solution
I had that same problem too, wanted to scan the NVidia or AMD servers for graphics card availability. Did not work from Java, so after a lot of research, finding the project mentioned above, I simply used Selenium to control FireFox and that got the proper server responses and I achieved my goal this way.
The only way to be sure that the exact same data is sent is to manually send it yourself through something like telnet. I had a similar problem once- it turned out that the browser was sending the data in one big chunk, while my code was sending it line-by-line. No site should have this problem, but it's possible that it exists.
The server might be checking for User-Agent request header and will block traffic that does not originate from a browser. Try setting the header in curl or your Java Code to a value corresponding to (any) browser. I've encountered such behavior on some e-shops and commercial websites.
I have a Tomcat 7 , Spring 4.2 'RestController' implementation of REST API which seems to produce 'ERR_INVALID_CHUNKED_ENCODING' for few API calls on returning a JSON response.
It is the same code that creates a ResponseEntity. But for few API calls the "Content-Length" is set properly and other calls the "Transfer-Encoding" is set as Chunked.
private CacheControl cacheControl = CacheControl.noStore().mustRevalidate();
protected <T> ResponseEntity<TNRestResponse<T>> createEntity(TNRestResponse<T> res) {
return ResponseEntity.ok().cacheControl(cacheControl).body(res);
}
The weird part is the response for the same API call that creates ERR_INVALID_CHUNKED_ENCODING seems to work fine in another environment. The only difference is the client and service is running in the same server in the problematic scenario.
The solution already tried is to set the Content-Length manually which seems to result to premature end of file on the client.The JSON length is only around 468 characters but client receives only 409 characters , even though server logs shows that the full response has been sent and connection is closed.
We are so lost at the solution for this problem because it is the same code acting strangely in different environment.I tried to check the compression settings in server.xml on both the tomcat.But everything looks fine.
Also disabled the proxy setting in both IE and chrome.
Any helpful inputs or insights would be really good ? Thanks in advance.
Follow these steps:
1) Go to your OS's Control panel > internet options > Connections >
LAN Settings or to your browser settings.
2) Deselect "Use Proxy" for your LAN or for your browser.
ERR_INVALID_CHUNKED_ENCODING
Original answer
Another original answer
I tried stress testing with JMeter software to test a web site as it crashed after a sms campaign. Currently site has been moved to a physical server.
I tested multiple times by adding threads, it worked and gave few errors (for above 1000 threads), and worked for 400 threads with no error. So I tried distributed testing with 4 PCs including my one.
After I tried again with only my PC to send requests to the site by adding 400 threads(ramp up = 1 , loop = 1). But each and every requests gives error. Then I tried using 1 thread. Same error was given.
I checked my network connection, and there is no problem. Then I browsed the web site "http://www.myjobs.lk/", and it works fine.
These are the values I have given in testing.
Under this condition, I cannot perform the testing because it always gives errors. How can I overcome this problem?
You're using incorrect JMeter configuration, change it as follows:
Remove http:// from "Server Name or IP" input
Put http to "Protocol input
It is also possible to have the full URL in "Path" field like
But using "http://" in "Server Name or IP" won't work.
Also once you defined hostname, port, path, etc. in HTTP Request Defaults it will be automatically applied to all HTTP Request Samplers. You will be able to override an option for particular this or that sampler but if you don't - default value will be used. See Why It's SO Important To Use JMeter's HTTP Request Defaults for more detailed explanation and some use cases.
For me it was helpful to setup proxy server:
Looks like JMeter tries to connect to myjobs.lk, and you browse to www.myjobs.lk. Try changing so that JMeter also connects to www.myjobs.lk
I am getting FileNotFoundException when making GET call to REST API. Here HTTP status code I get is 403.
For POST call I get IOException : No authentication challenges found, whereas I pass Authorization header. Here I get HTTP status code 401.
Look at my already asked question to see the code and logcat screen shot for POST call.
Below I am attaching logcat screen shot for GET call :
Note :
1) I have tried using Authenticator.
2) Tried different base64 flags such as NO_WRAP, URL_SAFE, DEFAULT.
3) My simple call to www.google.com works.
4) When I log urlConnection.getErrorStream(), last line in image is printed. I don't understand what is that and what does it mean. I have specified Content-type to application/json in header.
UPDATE : I tried using Burp and found that headers "Accept", "Content-type" were different. I used the same as in iOS app. But still it does not work.
Things to note :
1) It always throws an Exception on the line in = urlConnection.getInputStream();.
2) I logged few things and according to it, content-length is 114, which is not null. content-type is application/json; charser=utf-8.
There's a difference between your HTTP traffic for iOS and for Android. This is guaranteed, otherwise you'd get identical behaviour from the server. The difference is probably in HTTP header(s) &/or parameter(s).
This is very difficult to debug remotely via SO Q&A - E.g. we don't know what headers & parameters your iOS client is successfully using nor how your server is configured & programmed.
How to diagnose the problem & correct yourself:
Trace your working HTTP traffic: iOS client <-> server
Trace your non-working HTTP traffic: Android client <-> server
Compare (2) and (3). For the most thorough comparison, save each HTTP request and response message as a separate file for (2) and (3), then diff the corresponding files.
Recommended HTTP tracing tools:
Fiddler2 (windows only) See also Documentation
Burp (JVM-based: windows, linux, OSX, etc) See also Getting Started
WebScarab (JVM-based: windows, linux, OSX, etc) See also Getting Started
UPDATE
Seems you have the same problem for both GET and POST: the server is configured for BASIC authentication, but the client is not following the authentication protocol correctly. I think it just shows as a slightly different sympton in the two cases: for GET it says 'resource not found' (because you're not authenticated) and for POST the resource is given by you, but the server says you're not authorized to change the resource on the server. I suggest you've done enough (good!) debugging of request contents and now you should stop and focus on getting authentication working.
Send you GET/POST request to the server without Authorization header
Allow the server to prompt you for authentication with a 401 response with an WWW-Authenticate header containing a challenge string (e.g. WWW-Authenticate: Basic realm="Protected" see RFC 2617 HTTP Basic Authentication and Digest Authentication)
Now send an additional GET request to server that (either without/with the original request contents), but includes the Authentication header, with Base 64 encoded username:password (Authorization: Basic ZnJhbms6ZmllZGxlcg==)
I solved my problem and it is something I never tried to focus on while solving the problem.
I need not pass Authorization header. The thing is that there are 2 credentials come into the picture. one is server's authorization and second is credentials for login API. In my app, user creates an account and login to it. To authenticate the user I pass credentials to server and server authenticate it.
So when user enters correct credentials then response received is correct. And, in case of wrong credentials, my server passes a error message You are not authorized person, which I want to display to user(as in my iOS app). So the problem is here that HTTP status code (in case of wrong credentals) is 401 and that is why I don't receive the message sent by the server (and receive No authentication challenges found message).
The reason why I don't receive server message is that HttpUrlConnection don't give server response when HTTP status code is >= 400.
The only option to get error details in case status code 400 and above is to use getErrorStream() method and using that I was receiving No authentication challenges found message.
Finally, either I had to handle each status code, that is equal and above 400, at client side or I can use HttpClient, instead of HttpUrlConnection. And now I am moving to HttpClient.
I had a similar problem, and solved by passing the authenticated session cookie. Not sure if that is possible in your situation.
AuthUser="foobar"
AuthPass="password"
URL targetUrl = new URL("http://www.google.com/");
HttpURLConnection connection = (HttpURLConnection) targetUrl.openConnection();
connection.setRequestMethod("GET");
connection.setDoInput(true);
String authStr = Base64Variants.MIME_NO_LINEFEEDS.encode((AuthUser+":"+AuthPass).getBytes());
connection.addRequestProperty("Authorization", "Basic "+authStr);
InputStream inputStream= connection.getInputStream();
I just run something like this and it worked perfectly for me. Just make sure you use Base64Variants.MIME_NO_LINEFEEDS and you should be able to create a proper authentication header. If that doesn't work, then you might have some problem on the server side.
How can I make a java-based application server reply with an empty-valued response header, like this?
content-length:\r\n
Unfortunately when I call
response.setHeader("Content-Length", len)
where len is either an empty string or null, the response will not include the header.
I've checked the HttpServletResponse and HttpServletResponseWrapper javadocs but couldn't figure out what could be overriden to provide my custom behaviour.
Background
I'm building a testing application that is supposed to emulate badly-behaved HTTP server scenarios. The application is supposed to reply to requests with preset pages and HTTP headers, including malformed ones like the above case.
The application is written in grails.
I'm building a testing application that is supposed to emulate badly-behaved HTTP server scenarios.
In such a case, attempting to get a well-behaving server to mimic such behavior is a bad idea. If you need to mimic a bad server, or a particular set of scenarios you wish to test, then you may do one of the following:
write a custom application that listens on a particular port (using the ServerSocket class) that will respond with malformed HTTP headers. Using HTTP libraries may not help, for libraries may have code to detect erroneous conditions and correct them automatically.
use a HTTP proxy that is capable of intercepting responses and allows for modifications of these responses. You will find several if you Google for "http debugging proxy", but if you haven't heard of any, I would suggest looking at Fiddler, WebScarab or Burp.
You can try a tool like SoapUI or Fiddler with it's Firefox extension. I havent tried setting a malformed header with them but I wouldn't be suprised if you could.
Something not clear for me: your application is written in Grails, but you are discussing of javadocs... Well, I suppose you try to create a bad server in JAVA...
As you said, answering with "Content-Length:\r\n" is not legal for HTTP. You must put an integer value or discard the header. I think setHeader() helps you to avoid to produce an illegal HTTP message.
You can workaround this way creating manually the headers (you can write directly to the socket without using the setHeader blocks).
Other solution is to create a filter (in addition of your servlet) with your own implementation of HttpServletResponse. You will pass this implementation to the servlet.