I've written a query tool which at it's heart uses this block to gather info from an external url:
AsyncHttpClientConfig proxiedCF = new DefaultAsyncHttpClientConfig.Builder().setUserAgent(pickUserAgent()).build();
AsyncHttpClient asyncHttpClient = new DefaultAsyncHttpClient(proxiedCF);
Future<Response> f = asyncHttpClient.prepareGet(url).setProxyServer(new ProxyServer.Builder(pickProxyServer(), 80)).execute();
It works fine. However it works even when an invalid proxy is provided, which is a bit suspicious and I feel my elaborate proxy configuration is not used at all.
I stumbled upon this by having pickProxyServer() return a String "1.1.1.1", which is obviously not a valid web proxy.
I use SLF4J for logging and it looks pretty normal:
20:31:47.454 [AsyncHttpClient-7-1] DEBUG o.a.n.channel.NettyConnectListener - Using new Channel '[id: 0x03359938, L:/10.0.0.101:59775 - R:/1.1.1.1:80]' for 'GET' to '[[url removed by me]]'
20:31:47.586 [AsyncHttpClient-7-1] DEBUG o.a.netty.handler.HttpHandler -
Request DefaultFullHttpRequest(decodeResult: success, version: HTTP/1.1, content: EmptyByteBufBE)
GET [[url removed by me]] HTTP/1.1
Host: [[url removed by me]]
Accept: */*
User-Agent: burning_dandelion
Response DefaultHttpResponse(decodeResult: success, version: HTTP/1.1)
HTTP/1.1 200 OK
Date: Fri, 13 Oct 2017 18:31:49 GMT
Expires: Fri, 13 Oct 2017 18:31:49 GMT
Cache-Control: private, max-age=3600
Content-Type: text/xml; charset=ISO-8859-1
P3P: CP="This is not a P3P policy!"
Server: gws
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Set-Cookie: 1P_JAR=2017-10-13-18; expires=Fri, 20-Oct-2017 18:31:49 GMT; path=/; domain=[[url removed by me]]
Set-Cookie: NID=114=qiVBv02cmXYHh2RfLQbhBfESWIoaGlf3d2jlSbAdQ8yWPDsCpOeK9aYbvfq0HWsER68W1oE53jiriM_fivTc1bJi1F2sfCi0wMptKI-9U3ueVKITtFvYYZx2T0rJf1kQ; expires=Sat, 14-Apr-2018 18:31:49 GMT; path=/; domain=[[url removed by me]]; HttpOnly
Accept-Ranges: none
Vary: Accept-Encoding
Transfer-Encoding: chunked
20:31:47.587 [AsyncHttpClient-7-1] DEBUG o.a.netty.channel.ChannelManager - Adding key: ProxyPartitionKey(proxyHost=1.1.1.1, proxyPort=80, secured=false, targetHostBaseUrl=[[url removed by me]]:80 for channel [id: 0x03359938, L:/10.0.0.101:59775 - R:/1.1.1.1:80]
Can someone point me towards my error? Obviously, I want an I/O exception or any kind of notice when an invalid proxy is called upon.
Ok it's the middle of the night but I found something. When the port is anything but :80, the proxy is correctly used. This does the trick for me.
Related
Using certbot fails to generate certificate with this error:
org.shredzone.acme4j.exception.AcmeException: Failed to pass the challenge for domain www.
mysampledomain123.com, ... Giving up.
I manually checked the challenge file and got
http://www.mysampledomain123.com/.well-known/acme-challenge/jU--PkDrn5tDZw2RN6NNJHbPD00ovHFkLFvN3mJdeQX
Inside the file:
jU--PkDrn5tDZw2RN6NNJHbPD00ovHFkLFvN3mJdeQX.tuMr-UijwpsJ1KVZkdWTYgodWZ2SxxKdB7_CMAAEfpg
And here's the complete HTTP response header:
Accept-Ranges: bytes
Connection: keep-alive
Content-Encoding: gzip
Content-Type: text/plain;charset=iso-8859-1
Date: Sun, 16 Feb 2020 14:15:22 GMT
Server: nginx/1.14.0 (Ubuntu)
Transfer-Encoding: chunked
Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept
X-Powered-By: MyServer
X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 999
X-RateLimit-Reset: 0
I'm wondering whether the problem is with the HTTP response headers or the content itself.
Any ideas would be appreciated.
I have a strange problem in WildFly 12. When I post a HTTP request sometimes body is replaced by HTTP response header. Sometimes it doesn't.
the error json data:
HTTP/1.1 200 OK^M Expires: 0^M Cache-Control: no-cache, no-store, max-age=0, must-revalidate^M X-Powered-By: Undertow/1^M Server: WildFly/11^M X-XSS-Protection: 1; mode=block^M Pragma: no-cache^M X-Frame-Options: SAMEORIGIN^M Date: Thu, 08 Nov 2018 13:00:25 GMT^M Connection: keep-alive^M X-Content-Type-Options: nosniff^M Content-Type: text/plain;charset=UTF-8^M Content-Length: 1569^M ^M ationId":"61e1e1b0867c11e8a9a81fbc702bb402","value":"MU5123","startStationName":"上海虹桥国际机场","endAreaName":"北京首都国际机场"},{"shiftId":"6c874935c59e11e8a4f557d52226d05c","startAreaId":"61e1e1b0867c11e8a9a81fbc702bb402","endStationName":"北京首都国际机场","lineId":"336fb55d-cc7c-11e8-9ed0-00163e02896a","label":"MU5159","dispatchTime":"14:30","endAreaId":"61e1e193867c11e8a9a81fbc702bb402","endStationId":"61e1e193867c11e8a9a81fbc702bb402","startAreaName":"上海虹桥国际机场","startStationId":"61e1e1b0867c11e8a9a81fbc702bb402","value":"MU5159","startStationName":"上海虹桥国际机场","endAreaName":"北京首都国际机场"},{"shiftId":"6c874930c59e11e8a4f557d52226d05c","startAreaId":"61e1e1b0867c11e8a9a81fbc702bb402","endStationName":"北京首都国际机场","lineId":"9db7b2a5c5a911e89ed000163e02896a","label":"MU5331","dispatchTime":"15:00","endAreaId":"61e1e193867c11e8a9a81fbc702bb402","endStationId":"61e1e193867c11e8a9a81fbc702bb402","startAreaName":"上海虹桥国际机场","startStationId":"61e1e1b0867c11e8a9a81fbc702bb402","value":"MU5331","startStationName":"上海虹桥国际机场","endAreaName":"北京首都国际机场"}]
Could someone let me know the reason would be for this behavior?
I've got this URL (http://vignette4.wikia.nocookie.net/fantendo/images/6/6e/Small-mario.png/revision/latest?cb=20120718024112)
how determinate the file extension if it isn't at the end of the url?
You'll need to read the response headers. The MIME Type if known is stored in the Content-Type header.
HTTP/1.1 200 OK
Content-Disposition: inline; filename="Small-mario.png"
X-Thumbnailer: Vignette
Content-Type: image/png
Cache-Control: public, max-age=31536000
X-Surrogate-Key: ad1f82ba0cbe38fa60f83c036993a71e05dae492
Server: Jetty(9.2.z-SNAPSHOT)
X-Cacheable: YES
Content-Length: 58457
Accept-Ranges: bytes
Date: Mon, 06 Jul 2015 16:12:31 GMT
Age: 65
Connection: keep-alive
X-Served-By: thumbnailer-s1, cache-wk-sjc3160-WIKIA, cache-lhr6322-LHR
X-Cache: ORIGIN, MISS, HIT
X-Cache-Hits: ORIGIN, 0, 5
X-Timer: S1436199151.564330,VS0,VE0
Vary: Accept-Encoding
Timing-Allow-Origin: *
You're looking for the Content-Type header, which the server ought to send in the HTTP response to tell you this.
Note that it is not guaranteed to be accurate, or present at all.
I'm using java sockets to communicate with my server. I'm getting http response from the server in a byte[] format. Now I need to parse the headers and content of the byte[]. The sample response I'm getting from server is :
HTTP/1.1 200 OK
alternate-protocol: 443:quic,p=0.01
cache-control: no-cache, must-revalidate
content-disposition: attachment; filename="f.txt"
content-encoding: gzip
content-length: 341
content-type: text/javascript; charset=UTF-8
date: Sat, 15 Nov 2014 18:32:55 GMT
expires: -1
pragma: no-cache
server: gws
status: 200 OK
version: HTTP/1.1
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
1. Either I need an easiest way to parse this in Java (May be a library)
2. Or a way to convert it into HttpResponse object.
Apache HttpComponents is your friend in this case.
Please take a look at http://hc.apache.org/httpclient-3.x/apidocs/org/apache/commons/httpclient/HttpParser.html
when I get the following url with curl
curl -D headers.http "http://www.springerlink.com/index/10.1007/s00453-007-9157-8"
the file headers.http contains a "Location" header:
HTTP/1.1 302 Found
Date: Tue, 27 Oct 2009 17:00:20 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
X-AspNet-Version: 2.0.50727
Location: http://www.springerlink.com/link.asp?id=c104731297q64224
Set-Cookie: CookiesSupported=True; expires=Wed, 27-Oct-2010 17:00:20 GMT; path=/
Cache-Control: private
Content-Type: text/html; charset=utf-8
Content-Length: 173
but when I used the apache httpclient library this "Location:" header was missing (?).
int status = httpClient.executeMethod(method);
if(status!=HttpStatus.SC_OK &&
status!=HttpStatus.SC_MOVED_TEMPORARILY &&
status!=HttpStatus.SC_MOVED_PERMANENTLY
)
{
throw new IOException("connection failure for "+url+" status:"+status);
}
Header header=method.getResponseHeader("Location");
if(header==null )
{
for(Header h:method.getResponseHeaders())
{
LOG.info(h.toString());
}
throw new IOException(
"Expected a redirect for "+url
);
}
I've listed the headers below:
INFO: Date: Tue, 27 Oct 2009 17:05:13 GMT
INFO: Server: Microsoft-IIS/6.0
INFO: X-Powered-By: ASP.NET
INFO: X-AspNet-Version: 2.0.50727
INFO: Set-Cookie: ASP.NET_SessionId=js1o5wqnuhuh24islnvkyr45; path=/; HttpOnly
INFO: Cache-Control: private
INFO: Content-Type: text/html; charset=utf-8
INFO: Content-Length: 17245
uhh ???
What's going on is that with curl , you are getting a 302 which is actually a redirect, to the URL in the location header.
With the Apache httpclient it is doing the redirect for you, and returning the headers from the request to the redirected-to location.
To demonstrate this try
curl -D headers.http "http://www.springerlink.com/link.asp?id=c104731297q64224"
and compare the response.
edit: There are actually about 4 redirects in there if you follow each location header through with curl.
http://www.springerlink.com/index/10.1007/s00453-007-9157-8 is actually a redirect. Since the -D option means "headers only", the first one is not redirecting to the specified Location: ..., while the second one is. Take a look at the Content-Length, it's much different.
What happens when you leave out the -D?
Add this
method.setFollowRedirects(false);
Before you execute the method.
HttpClient follows the redirect automatically by default but Curl doesn't.