I would like to send an HTTP request and get the response body, but I have URL that is stored on multiple servers. Let's say I have this list:
www.mysite.com 192.168.1.31
www.mysite.com 192.168.1.32
and I want to make the request to all the the different servers (different IP's) but same URL
Is there any option to do that in Java?
Yes, you can do that. Define the URL as being what you want and using the IP address in place of a domain name. I.E. http://192.168.1.31/path/to/index.html Then add the "Host: www.mysite.com" header before issuing the request. Any HTTP/1.1 compliant server will use the value of that header as the domain with which it was accessed.
Exactly how you accomplish this depends on whatever library you're using to make the connection but they should all have the ability to set arbitrary headers -- just make sure it doesn't overwrite your custom "Host" header with one of its own from the URL. See this other StackOverflow question for examples of how to implement an HTTP request.
This works because on the wire it's all IP. You can try it yourself using nc, socket, or even telnet.
(open TCP connection to 192.168.1.31 port 80)
GET /path/to/index.html HTTP/1.1
Host: www.mysite.com
<--blank line signals end of headers
Related
Some third party is sending an Http Post request whenever something changes in their DB (e.g. when a contact has been updated, they send the contactID and 'contact_updated'). I have build a socket listener that catches those requests and is able to parse the information. However, I just can't get it to work to send back a response with the status '200 - OK'. Thus, the server on the client side keeps on trying (four times or so) to re-send the request.
Is there any, simple way to just send the response status without the need of adding external libs etc.?
It should be enough to send the string HTTP/1.1 200 OK back in your socket-listener.
If you have troubles, you can check out this answer, it shows how to use a HttpServer in Java just via plain JavaSE features.
Use
response.setStatus(HttpServletResponse.SC_OK);
to set the status code in your response header.
You may also set the content type.
response.setContentType("text/html;charset=UTF-8");
I need to force the client to retry its request (meaning to send the same request one more time). What I'm thinking of is a response with status-code 307 and header Location: <original-url> (that's good enough for now, unless there's a better way).
My question is, from HTTP specification point of view, what is the correct value for Location in this specific context. Or more specifically in Java having request of type HttpServletRequest, which one should I use: getRequestURI (Returns the part of this request's URL from the protocol name up to the query string in the first line of the HTTP request) or getRequestURL (Reconstructs the URL the client used to make the request containing protocol, server name, port number, and server path, but it does not include query string parameters).
Any other suggestion/comment is appreciated.
getRequestURL() returns complete URL used by the client where as getRequestURI() returns just the basic path resides in server.
i am using this technique to redirect with a response status this is my code this is useful:-
httpServletResponse.reset();
httpServletResponse.setStatus(HttpServletResponse.SC_UNAUTHORIZED);
httpServletResponse.setHeader("SERVER-RESPONSE", "bad request");
return;
and also you can set headers in response.
I believe a redirect is the wrong status code in the first place.
Isn't this what 503 is for? (https://www.greenbytes.de/tech/webdav/rfc7231.html#status.503)
I have been wondering if its possible to anonymize public URL. When user makes a request with this anonymized public URL, let Nginx decode, fetch and serve the URL.
Example
Public URL http://amazon.server.com/location/file.html
Anonymized URL https://amazon.server.com/09872340-932872389-390643289/983724.html
Nginx decodes 09872340-932872389-390643289/983724.html to location/file.html
Added image below for further clarification. Nginx has a reverse logic to decode, whereas Remote Server has the logic to Anonymize URL.
Question
All I need to know is how would Nginx decode anonymized URL? Nginx got anonymized URL request. There has to be a way to decode it.
This is an answer to the updated question:
Question All I need to know is how would Nginx decode anonymized URL? Nginx got anonymized URL request. There has to be a way to decode it.
Nginx would make a request to a script, e.g., either through proxy_pass or fastcgi_pass et al.
The script could decode the URL and provide the actual URL through a Location HTTP Response Header with a 302 Found HTTP Status.
Nginx would then have the decoded URL stored in the $upstream_http_location variable. It could subsequently be used in another proxy_pass et al within a named location #named, to which you could redirect the processing of the original request from the user through error_page 302 = #named.
In all, each user request would be processed twice within nginx, but it'll all be transparent to the user -- they simply receive the resource through the original URL, with all redirects being done internally within nginx.
Define Anonymize for a URL? You can use any of the same methods as URL shortners such as http://bitly.com. But that is not truely anonymous since there is a definite mapping between the shortened URL and the target public url. If you make this per user based there is still a mapping but it is user based.
Looks like what you are suggesting is a variation on the above scheme where instead of sending the user to the target URL via a redirect you want the your server to actually fetch the content and return to the user. You need to be aware of the linked content in the public URL such as style sheets and images and adjust them accordingly. Many of the standard proxies has this kind of functionality built in. Also take a look at
https://github.com/jenssegers/php-proxy
http://search.cpan.org/~book/HTTP-Proxy-0.304/lib/HTTP/Proxy.pm.
If you are planning to build your own these can serve as a base.
I think what you want to do here is somewhat similar to another question I've answered in the past, where for each request by the client, you effectively want to make two requests to two different upstreams under the hood (first one to an upstream capable of decoding the URL, second one to actually fetch said decoded URL), but, of course, only return one result.
https://serverfault.com/questions/202011/nginx-and-2-upstreams/485044#485044
As mentioned on serverfault, you could use error_page to process another request, after the first one is complete. You could then use $upstream_http_ to make the subsequent request based on the original one, for example, using $upstream_http_location.
You might also want to look into X-Accel-Redirect header, introduced in this context at proxy_ignore_headers.
I want to retrieve the server's response as is, with all headers. The first thing that comes to mind is to use raw sockets. As I have learned from the search, there are 3 ways to indicate the end of response:
(1) closing the connection;
(2) examining Content-Length;
(3) getting all chunks in the case of Transfer-Encoding: Chunked.
There is also
(4) the timeout method: assume that the timeout means end of data, but the latter is not really reliable.
I want a general-case solution and do not want to
add a Connection: close line to the request itself.
In addition, it is recommended to use an existing library rather than re-invent the wheel.
Question:
How do I use an existing package, preferably, something already present in Android, to detect the end of HTTP response while having access (without interference) to the raw data stream?
UPD: forgot to mention that the HTTP request is given to me as a sequence of bytes. Yes, it is for testing.
PS
relevant reading:
End of an HTTP Response
Detect the end of an HTTP Request in Java
Detect end of HTTP request body
How HTTP Server inform its clients that the response has ended
Proper handling of chuncked Http Response within Socket
Detect the end of a HTTP packet
Android socket & HTTP response headers
Java HTTP GET response waits until timeout
I suggest to use a the Apache HTTP client package (http://hc.apache.org/httpclient-3.x/ ) so you don't need to implement all the finicky details of the HTTP protocol.
The Apache Http Client will give you access to the headers and their content, which may be enough for you.
If you really need access to the actual character sequence sent by the server (e.g. for debugging purposes), you could then intercept the communication by replacing the connection socket factory with your own to create "intercepting" sockets which store all data transferred in a buffer where your code can access it later on. See http://hc.apache.org/httpcomponents-client-4.3.x/tutorial/html/connmgmt.html#d5e418
Suppose a browser makes a request
asking for the server to keep the connection alive
(connection: keep-alive).
And this request requires the invocation of a servlet.
Inside my servlet should I care to choose the best
way to send data (chunked or indicating
the lenght of the body ) ??
if the server does that for me , why , inside my servlet ,
I'm able to modyfy headers like:
content-length
and transfer encoding ?
thanks
If you know the body length up front, you should set Content-Length header before writing body.
Otherwise do nothing; the servlet container should be able to automatically add Transfer-Encoding, and chunk-ify your body. That is subject to client/request version and Connection header.