Use of java.net.URLConnection is asked about pretty often here, and the Oracle tutorial is too concise about it.
That tutorial basically only shows how to fire a GET request and read the response. It doesn't explain anywhere how to use it to, among others, perform a POST request, set request headers, read response headers, deal with cookies, submit a HTML form, upload a file, etc.
So, how can I use java.net.URLConnection to fire and handle "advanced" HTTP requests?
First a disclaimer beforehand: the posted code snippets are all basic examples. You'll need to handle trivial IOExceptions and RuntimeExceptions like NullPointerException, ArrayIndexOutOfBoundsException and consorts yourself.
In case you're developing for Android instead of Java, note also that since introduction of API level 28, cleartext HTTP requests are disabled by default. You are encouraged to use HttpsURLConnection, but if it is really necessary, cleartext can be enabled in the Application Manifest.
Preparing
We first need to know at least the URL and the charset. The parameters are optional and depend on the functional requirements.
String url = "http://example.com";
String charset = "UTF-8"; // Or in Java 7 and later, use the constant: java.nio.charset.StandardCharsets.UTF_8.name()
String param1 = "value1";
String param2 = "value2";
// ...
String query = String.format("param1=%s¶m2=%s",
URLEncoder.encode(param1, charset),
URLEncoder.encode(param2, charset));
The query parameters must be in name=value format and be concatenated by &. You would normally also URL-encode the query parameters with the specified charset using URLEncoder#encode().
The String#format() is just for convenience. I prefer it when I would need the String concatenation operator + more than twice.
Firing an HTTP GET request with (optionally) query parameters
It's a trivial task. It's the default request method.
URLConnection connection = new URL(url + "?" + query).openConnection();
connection.setRequestProperty("Accept-Charset", charset);
InputStream response = connection.getInputStream();
// ...
Any query string should be concatenated to the URL using ?. The Accept-Charset header may hint the server what encoding the parameters are in. If you don't send any query string, then you can leave the Accept-Charset header away. If you don't need to set any headers, then you can even use the URL#openStream() shortcut method.
InputStream response = new URL(url).openStream();
// ...
Either way, if the other side is an HttpServlet, then its doGet() method will be called and the parameters will be available by HttpServletRequest#getParameter().
For testing purposes, you can print the response body to standard output as below:
try (Scanner scanner = new Scanner(response)) {
String responseBody = scanner.useDelimiter("\\A").next();
System.out.println(responseBody);
}
Firing an HTTP POST request with query parameters
Setting the URLConnection#setDoOutput() to true implicitly sets the request method to POST. The standard HTTP POST as web forms do is of type application/x-www-form-urlencoded wherein the query string is written to the request body.
URLConnection connection = new URL(url).openConnection();
connection.setDoOutput(true); // Triggers POST.
connection.setRequestProperty("Accept-Charset", charset);
connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded;charset=" + charset);
try (OutputStream output = connection.getOutputStream()) {
output.write(query.getBytes(charset));
}
InputStream response = connection.getInputStream();
// ...
Note: whenever you'd like to submit a HTML form programmatically, don't forget to take the name=value pairs of any <input type="hidden"> elements into the query string and of course also the name=value pair of the <input type="submit"> element which you'd like to "press" programmatically (because that's usually been used in the server side to distinguish if a button was pressed and if so, which one).
You can also cast the obtained URLConnection to HttpURLConnection and use its HttpURLConnection#setRequestMethod() instead. But if you're trying to use the connection for output you still need to set URLConnection#setDoOutput() to true.
HttpURLConnection httpConnection = (HttpURLConnection) new URL(url).openConnection();
httpConnection.setRequestMethod("POST");
// ...
Either way, if the other side is an HttpServlet, then its doPost() method will be called and the parameters will be available by HttpServletRequest#getParameter().
Actually firing the HTTP request
You can fire the HTTP request explicitly with URLConnection#connect(), but the request will automatically be fired on demand when you want to get any information about the HTTP response, such as the response body using URLConnection#getInputStream() and so on. The above examples does exactly that, so the connect() call is in fact superfluous.
Gathering HTTP response information
HTTP response status:
You need an HttpURLConnection here. Cast it first if necessary.
int status = httpConnection.getResponseCode();
HTTP response headers:
for (Entry<String, List<String>> header : connection.getHeaderFields().entrySet()) {
System.out.println(header.getKey() + "=" + header.getValue());
}
HTTP response encoding:
When the Content-Type contains a charset parameter, then the response body is likely text based and we'd like to process the response body with the server-side specified character encoding then.
String contentType = connection.getHeaderField("Content-Type");
String charset = null;
for (String param : contentType.replace(" ", "").split(";")) {
if (param.startsWith("charset=")) {
charset = param.split("=", 2)[1];
break;
}
}
if (charset != null) {
try (BufferedReader reader = new BufferedReader(new InputStreamReader(response, charset))) {
for (String line; (line = reader.readLine()) != null;) {
// ... System.out.println(line)?
}
}
} else {
// It's likely binary content, use InputStream/OutputStream.
}
Maintaining the session
The server side session is usually backed by a cookie. Some web forms require that you're logged in and/or are tracked by a session. You can use the CookieHandler API to maintain cookies. You need to prepare a CookieManager with a CookiePolicy of ACCEPT_ALL before sending all HTTP requests.
// First set the default cookie manager.
CookieHandler.setDefault(new CookieManager(null, CookiePolicy.ACCEPT_ALL));
// All the following subsequent URLConnections will use the same cookie manager.
URLConnection connection = new URL(url).openConnection();
// ...
connection = new URL(url).openConnection();
// ...
connection = new URL(url).openConnection();
// ...
Note that this is known to not always work properly in all circumstances. If it fails for you, then best is to manually gather and set the cookie headers. You basically need to grab all Set-Cookie headers from the response of the login or the first GET request and then pass this through the subsequent requests.
// Gather all cookies on the first request.
URLConnection connection = new URL(url).openConnection();
List<String> cookies = connection.getHeaderFields().get("Set-Cookie");
// ...
// Then use the same cookies on all subsequent requests.
connection = new URL(url).openConnection();
for (String cookie : cookies) {
connection.addRequestProperty("Cookie", cookie.split(";", 2)[0]);
}
// ...
The split(";", 2)[0] is there to get rid of cookie attributes which are irrelevant for the server side like expires, path, etc. Alternatively, you could also use cookie.substring(0, cookie.indexOf(';')) instead of split().
Streaming mode
The HttpURLConnection will by default buffer the entire request body before actually sending it, regardless of whether you've set a fixed content length yourself using connection.setRequestProperty("Content-Length", contentLength);. This may cause OutOfMemoryExceptions whenever you concurrently send large POST requests (e.g. uploading files). To avoid this, you would like to set the HttpURLConnection#setFixedLengthStreamingMode().
httpConnection.setFixedLengthStreamingMode(contentLength);
But if the content length is really not known beforehand, then you can make use of chunked streaming mode by setting the HttpURLConnection#setChunkedStreamingMode() accordingly. This will set the HTTP Transfer-Encoding header to chunked which will force the request body being sent in chunks. The below example will send the body in chunks of 1 KB.
httpConnection.setChunkedStreamingMode(1024);
User-Agent
It can happen that a request returns an unexpected response, while it works fine with a real web browser. The server side is probably blocking requests based on the User-Agent request header. The URLConnection will by default set it to Java/1.6.0_19 where the last part is obviously the JRE version. You can override this as follows:
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36"); // Do as if you're using Chrome 41 on Windows 7.
Use the User-Agent string from a recent browser.
Error handling
If the HTTP response code is 4nn (Client Error) or 5nn (Server Error), then you may want to read the HttpURLConnection#getErrorStream() to see if the server has sent any useful error information.
InputStream error = ((HttpURLConnection) connection).getErrorStream();
If the HTTP response code is -1, then something went wrong with connection and response handling. The HttpURLConnection implementation is in older JREs somewhat buggy with keeping connections alive. You may want to turn it off by setting the http.keepAlive system property to false. You can do this programmatically in the beginning of your application by:
System.setProperty("http.keepAlive", "false");
Uploading files
You'd normally use multipart/form-data encoding for mixed POST content (binary and character data). The encoding is in more detail described in RFC2388.
String param = "value";
File textFile = new File("/path/to/file.txt");
File binaryFile = new File("/path/to/file.bin");
String boundary = Long.toHexString(System.currentTimeMillis()); // Just generate some unique random value.
String CRLF = "\r\n"; // Line separator required by multipart/form-data.
URLConnection connection = new URL(url).openConnection();
connection.setDoOutput(true);
connection.setRequestProperty("Content-Type", "multipart/form-data; boundary=" + boundary);
try (
OutputStream output = connection.getOutputStream();
PrintWriter writer = new PrintWriter(new OutputStreamWriter(output, charset), true);
) {
// Send normal param.
writer.append("--" + boundary).append(CRLF);
writer.append("Content-Disposition: form-data; name=\"param\"").append(CRLF);
writer.append("Content-Type: text/plain; charset=" + charset).append(CRLF);
writer.append(CRLF).append(param).append(CRLF).flush();
// Send text file.
writer.append("--" + boundary).append(CRLF);
writer.append("Content-Disposition: form-data; name=\"textFile\"; filename=\"" + textFile.getName() + "\"").append(CRLF);
writer.append("Content-Type: text/plain; charset=" + charset).append(CRLF); // Text file itself must be saved in this charset!
writer.append(CRLF).flush();
Files.copy(textFile.toPath(), output);
output.flush(); // Important before continuing with writer!
writer.append(CRLF).flush(); // CRLF is important! It indicates end of boundary.
// Send binary file.
writer.append("--" + boundary).append(CRLF);
writer.append("Content-Disposition: form-data; name=\"binaryFile\"; filename=\"" + binaryFile.getName() + "\"").append(CRLF);
writer.append("Content-Type: " + URLConnection.guessContentTypeFromName(binaryFile.getName())).append(CRLF);
writer.append("Content-Transfer-Encoding: binary").append(CRLF);
writer.append(CRLF).flush();
Files.copy(binaryFile.toPath(), output);
output.flush(); // Important before continuing with writer!
writer.append(CRLF).flush(); // CRLF is important! It indicates end of boundary.
// End of multipart/form-data.
writer.append("--" + boundary + "--").append(CRLF).flush();
}
If the other side is an HttpServlet, then its doPost() method will be called and the parts will be available by HttpServletRequest#getPart() (note, thus not getParameter() and so on!). The getPart() method is however relatively new, it's introduced in Servlet 3.0 (Glassfish 3, Tomcat 7, etc.). Prior to Servlet 3.0, your best choice is using Apache Commons FileUpload to parse a multipart/form-data request. Also see this answer for examples of both the FileUpload and the Servelt 3.0 approaches.
Dealing with untrusted or misconfigured HTTPS sites
In case you're developing for Android instead of Java, be careful: the workaround below may save your day if you don't have correct certificates deployed during development. But you should not use it for production. These days (April 2021) Google will not allow your app be distributed on Play Store if they detect insecure hostname verifier, see https://support.google.com/faqs/answer/7188426.
Sometimes you need to connect an HTTPS URL, perhaps because you're writing a web scraper. In that case, you may likely face a javax.net.ssl.SSLException: Not trusted server certificate on some HTTPS sites who doesn't keep their SSL certificates up to date, or a java.security.cert.CertificateException: No subject alternative DNS name matching [hostname] found or javax.net.ssl.SSLProtocolException: handshake alert: unrecognized_name on some misconfigured HTTPS sites.
The following one-time-run static initializer in your web scraper class should make HttpsURLConnection more lenient as to those HTTPS sites and thus not throw those exceptions anymore.
static {
TrustManager[] trustAllCertificates = new TrustManager[] {
new X509TrustManager() {
#Override
public X509Certificate[] getAcceptedIssuers() {
return null; // Not relevant.
}
#Override
public void checkClientTrusted(X509Certificate[] certs, String authType) {
// Do nothing. Just allow them all.
}
#Override
public void checkServerTrusted(X509Certificate[] certs, String authType) {
// Do nothing. Just allow them all.
}
}
};
HostnameVerifier trustAllHostnames = new HostnameVerifier() {
#Override
public boolean verify(String hostname, SSLSession session) {
return true; // Just allow them all.
}
};
try {
System.setProperty("jsse.enableSNIExtension", "false");
SSLContext sc = SSLContext.getInstance("SSL");
sc.init(null, trustAllCertificates, new SecureRandom());
HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
HttpsURLConnection.setDefaultHostnameVerifier(trustAllHostnames);
}
catch (GeneralSecurityException e) {
throw new ExceptionInInitializerError(e);
}
}
Last words
The Apache HttpComponents HttpClient is much more convenient in this all :)
HttpClient Tutorial
HttpClient Examples
Parsing and extracting HTML
If all you want is parsing and extracting data from HTML, then better use a HTML parser like Jsoup.
What are the pros/cons of leading HTML parsers in Java
How to scan and extract a webpage in Java
When working with HTTP it's almost always more useful to refer to HttpURLConnection rather than the base class URLConnection (since URLConnection is an abstract class when you ask for URLConnection.openConnection() on a HTTP URL that's what you'll get back anyway).
Then you can instead of relying on URLConnection#setDoOutput(true) to implicitly set the request method to POST instead do httpURLConnection.setRequestMethod("POST") which some might find more natural (and which also allows you to specify other request methods such as PUT, DELETE, ...).
It also provides useful HTTP constants so you can do:
int responseCode = httpURLConnection.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_OK) {
Inspired by this and other questions on Stack Overflow, I've created a minimal open source basic-http-client that embodies most of the techniques found here.
google-http-java-client is also a great open source resource.
I suggest you take a look at the code on kevinsawicki/http-request, its basically a wrapper on top of HttpUrlConnection it provides a much simpler API in case you just want to make the requests right now or you can take a look at the sources (it's not too big) to take a look at how connections are handled.
Example: Make a GET request with content type application/json and some query parameters:
// GET http://google.com?q=baseball%20gloves&size=100
String response = HttpRequest.get("http://google.com", true, "q", "baseball gloves", "size", 100)
.accept("application/json")
.body();
System.out.println("Response was: " + response);
Update
The new HTTP Client shipped with Java 9 but as part of an
Incubator module named jdk.incubator.httpclient. Incubator modules are
a means of putting non-final APIs in the hands of developers while the
APIs progress towards either finalization or removal in a future
release.
In Java 9, you can send a GET request like:
// GET
HttpResponse response = HttpRequest
.create(new URI("http://www.stackoverflow.com"))
.headers("Foo", "foovalue", "Bar", "barvalue")
.GET()
.response();
Then you can examine the returned HttpResponse:
int statusCode = response.statusCode();
String responseBody = response.body(HttpResponse.asString());
Since this new HTTP Client is in java.httpclient jdk.incubator.httpclient module, you should declare this dependency in your module-info.java file:
module com.foo.bar {
requires jdk.incubator.httpclient;
}
There are two options you can go with HTTP URL Hits : GET / POST
GET Request:
HttpURLConnection.setFollowRedirects(true); // Defaults to true
String url = "https://name_of_the_url";
URL request_url = new URL(url);
HttpURLConnection http_conn = (HttpURLConnection)request_url.openConnection();
http_conn.setConnectTimeout(100000);
http_conn.setReadTimeout(100000);
http_conn.setInstanceFollowRedirects(true);
System.out.println(String.valueOf(http_conn.getResponseCode()));
POST request:
HttpURLConnection.setFollowRedirects(true); // Defaults to true
String url = "https://name_of_the_url"
URL request_url = new URL(url);
HttpURLConnection http_conn = (HttpURLConnection)request_url.openConnection();
http_conn.setConnectTimeout(100000);
http_conn.setReadTimeout(100000);
http_conn.setInstanceFollowRedirects(true);
http_conn.setDoOutput(true);
PrintWriter out = new PrintWriter(http_conn.getOutputStream());
if (urlparameter != null) {
out.println(urlparameter);
}
out.close();
out = null;
System.out.println(String.valueOf(http_conn.getResponseCode()));
I was also very inspired by this response.
I am often on projects where I need to do some HTTP, and I may not want to bring in a lot of third-party dependencies (which bring in others and so on and so on, etc.)
I started to write my own utilities based on some of this conversation (not any where done):
package org.boon.utils;
import java.io.IOException;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLConnection;
import java.util.Map;
import static org.boon.utils.IO.read;
public class HTTP {
Then there are just a bunch or static methods.
public static String get(
final String url) {
Exceptions.tryIt(() -> {
URLConnection connection;
connection = doGet(url, null, null, null);
return extractResponseString(connection);
});
return null;
}
public static String getWithHeaders(
final String url,
final Map<String, ? extends Object> headers) {
URLConnection connection;
try {
connection = doGet(url, headers, null, null);
return extractResponseString(connection);
} catch (Exception ex) {
Exceptions.handle(ex);
return null;
}
}
public static String getWithContentType(
final String url,
final Map<String, ? extends Object> headers,
String contentType) {
URLConnection connection;
try {
connection = doGet(url, headers, contentType, null);
return extractResponseString(connection);
} catch (Exception ex) {
Exceptions.handle(ex);
return null;
}
}
public static String getWithCharSet(
final String url,
final Map<String, ? extends Object> headers,
String contentType,
String charSet) {
URLConnection connection;
try {
connection = doGet(url, headers, contentType, charSet);
return extractResponseString(connection);
} catch (Exception ex) {
Exceptions.handle(ex);
return null;
}
}
Then post...
public static String postBody(
final String url,
final String body) {
URLConnection connection;
try {
connection = doPost(url, null, "text/plain", null, body);
return extractResponseString(connection);
} catch (Exception ex) {
Exceptions.handle(ex);
return null;
}
}
public static String postBodyWithHeaders(
final String url,
final Map<String, ? extends Object> headers,
final String body) {
URLConnection connection;
try {
connection = doPost(url, headers, "text/plain", null, body);
return extractResponseString(connection);
} catch (Exception ex) {
Exceptions.handle(ex);
return null;
}
}
public static String postBodyWithContentType(
final String url,
final Map<String, ? extends Object> headers,
final String contentType,
final String body) {
URLConnection connection;
try {
connection = doPost(url, headers, contentType, null, body);
return extractResponseString(connection);
} catch (Exception ex) {
Exceptions.handle(ex);
return null;
}
}
public static String postBodyWithCharset(
final String url,
final Map<String, ? extends Object> headers,
final String contentType,
final String charSet,
final String body) {
URLConnection connection;
try {
connection = doPost(url, headers, contentType, charSet, body);
return extractResponseString(connection);
} catch (Exception ex) {
Exceptions.handle(ex);
return null;
}
}
private static URLConnection doPost(String url, Map<String, ? extends Object> headers,
String contentType, String charset, String body
) throws IOException {
URLConnection connection;/* Handle output. */
connection = new URL(url).openConnection();
connection.setDoOutput(true);
manageContentTypeHeaders(contentType, charset, connection);
manageHeaders(headers, connection);
IO.write(connection.getOutputStream(), body, IO.CHARSET);
return connection;
}
private static void manageHeaders(Map<String, ? extends Object> headers, URLConnection connection) {
if (headers != null) {
for (Map.Entry<String, ? extends Object> entry : headers.entrySet()) {
connection.setRequestProperty(entry.getKey(), entry.getValue().toString());
}
}
}
private static void manageContentTypeHeaders(String contentType, String charset, URLConnection connection) {
connection.setRequestProperty("Accept-Charset", charset == null ? IO.CHARSET : charset);
if (contentType!=null && !contentType.isEmpty()) {
connection.setRequestProperty("Content-Type", contentType);
}
}
private static URLConnection doGet(String url, Map<String, ? extends Object> headers,
String contentType, String charset) throws IOException {
URLConnection connection;/* Handle output. */
connection = new URL(url).openConnection();
manageContentTypeHeaders(contentType, charset, connection);
manageHeaders(headers, connection);
return connection;
}
private static String extractResponseString(URLConnection connection) throws IOException {
/* Handle input. */
HttpURLConnection http = (HttpURLConnection)connection;
int status = http.getResponseCode();
String charset = getCharset(connection.getHeaderField("Content-Type"));
if (status==200) {
return readResponseBody(http, charset);
} else {
return readErrorResponseBody(http, status, charset);
}
}
private static String readErrorResponseBody(HttpURLConnection http, int status, String charset) {
InputStream errorStream = http.getErrorStream();
if ( errorStream!=null ) {
String error = charset== null ? read( errorStream ) :
read( errorStream, charset );
throw new RuntimeException("STATUS CODE =" + status + "\n\n" + error);
} else {
throw new RuntimeException("STATUS CODE =" + status);
}
}
private static String readResponseBody(HttpURLConnection http, String charset) throws IOException {
if (charset != null) {
return read(http.getInputStream(), charset);
} else {
return read(http.getInputStream());
}
}
private static String getCharset(String contentType) {
if (contentType==null) {
return null;
}
String charset = null;
for (String param : contentType.replace(" ", "").split(";")) {
if (param.startsWith("charset=")) {
charset = param.split("=", 2)[1];
break;
}
}
charset = charset == null ? IO.CHARSET : charset;
return charset;
}
Well, you get the idea....
Here are the tests:
static class MyHandler implements HttpHandler {
public void handle(HttpExchange t) throws IOException {
InputStream requestBody = t.getRequestBody();
String body = IO.read(requestBody);
Headers requestHeaders = t.getRequestHeaders();
body = body + "\n" + copy(requestHeaders).toString();
t.sendResponseHeaders(200, body.length());
OutputStream os = t.getResponseBody();
os.write(body.getBytes());
os.close();
}
}
#Test
public void testHappy() throws Exception {
HttpServer server = HttpServer.create(new InetSocketAddress(9212), 0);
server.createContext("/test", new MyHandler());
server.setExecutor(null); // creates a default executor
server.start();
Thread.sleep(10);
Map<String,String> headers = map("foo", "bar", "fun", "sun");
String response = HTTP.postBodyWithContentType("http://localhost:9212/test", headers, "text/plain", "hi mom");
System.out.println(response);
assertTrue(response.contains("hi mom"));
assertTrue(response.contains("Fun=[sun], Foo=[bar]"));
response = HTTP.postBodyWithCharset("http://localhost:9212/test", headers, "text/plain", "UTF-8", "hi mom");
System.out.println(response);
assertTrue(response.contains("hi mom"));
assertTrue(response.contains("Fun=[sun], Foo=[bar]"));
response = HTTP.postBodyWithHeaders("http://localhost:9212/test", headers, "hi mom");
System.out.println(response);
assertTrue(response.contains("hi mom"));
assertTrue(response.contains("Fun=[sun], Foo=[bar]"));
response = HTTP.get("http://localhost:9212/test");
System.out.println(response);
response = HTTP.getWithHeaders("http://localhost:9212/test", headers);
System.out.println(response);
assertTrue(response.contains("Fun=[sun], Foo=[bar]"));
response = HTTP.getWithContentType("http://localhost:9212/test", headers, "text/plain");
System.out.println(response);
assertTrue(response.contains("Fun=[sun], Foo=[bar]"));
response = HTTP.getWithCharSet("http://localhost:9212/test", headers, "text/plain", "UTF-8");
System.out.println(response);
assertTrue(response.contains("Fun=[sun], Foo=[bar]"));
Thread.sleep(10);
server.stop(0);
}
#Test
public void testPostBody() throws Exception {
HttpServer server = HttpServer.create(new InetSocketAddress(9220), 0);
server.createContext("/test", new MyHandler());
server.setExecutor(null); // creates a default executor
server.start();
Thread.sleep(10);
Map<String,String> headers = map("foo", "bar", "fun", "sun");
String response = HTTP.postBody("http://localhost:9220/test", "hi mom");
assertTrue(response.contains("hi mom"));
Thread.sleep(10);
server.stop(0);
}
#Test(expected = RuntimeException.class)
public void testSad() throws Exception {
HttpServer server = HttpServer.create(new InetSocketAddress(9213), 0);
server.createContext("/test", new MyHandler());
server.setExecutor(null); // creates a default executor
server.start();
Thread.sleep(10);
Map<String,String> headers = map("foo", "bar", "fun", "sun");
String response = HTTP.postBodyWithContentType("http://localhost:9213/foo", headers, "text/plain", "hi mom");
System.out.println(response);
assertTrue(response.contains("hi mom"));
assertTrue(response.contains("Fun=[sun], Foo=[bar]"));
Thread.sleep(10);
server.stop(0);
}
You can find the rest here:
https://github.com/RichardHightower/boon
My goal is to provide the common things one would want to do in a bit more easier way then....
Initially I was misled by this article which favours HttpClient.
Later I have realized that HttpURLConnection is going to stay from this article.
As per the Google blog:
Apache HTTP client has fewer bugs on Eclair and Froyo. It is the best choice for these releases. For Gingerbread , HttpURLConnection is the best choice. Its simple API and small size makes it great fit for Android.
Transparent compression and response caching reduce network use, improve speed and save battery. New applications should use HttpURLConnection; it is where we will be spending our energy going forward.
After reading this article and some other stack over flow questions, I am convinced that HttpURLConnection is going to stay for longer durations.
Some of the SE questions favouring HttpURLConnections:
On Android, make a POST request with URL Encoded Form data without using UrlEncodedFormEntity
HttpPost works in Java project, but not on Android
There is also OkHttp, which is an HTTP client that’s efficient by default:
HTTP/2 support allows all requests to the same host to share a socket.
Connection pooling reduces request latency (if HTTP/2 isn’t available).
Transparent GZIP shrinks download sizes.
Response caching avoids the network completely for repeat requests.
First create an instance of OkHttpClient:
OkHttpClient client = new OkHttpClient();
Then, prepare your GET request:
Request request = new Request.Builder()
.url(url)
.build();
finally, use OkHttpClient to send prepared Request:
Response response = client.newCall(request).execute();
For more details, you can consult the OkHttp's documentation
If you are using HTTP GET, please remove this line:
urlConnection.setDoOutput(true);
You can also use JdkRequest from jcabi-http (I'm a developer), which does all this work for you, decorating HttpURLConnection, firing HTTP requests and parsing responses, for example:
String html = new JdkRequest("http://www.google.com").fetch().body();
Check this blog post for more info: http://www.yegor256.com/2014/04/11/jcabi-http-intro.html
If you are using Java 11 (except on Android), instead of the legacy HttpUrlConnection class, you can use Java 11 new HTTP Client API.
An example GET request:
var uri = URI.create("https://httpbin.org/get?age=26&isHappy=true");
var client = HttpClient.newHttpClient();
var request = HttpRequest
.newBuilder()
.uri(uri)
.header("accept", "application/json")
.GET()
.build();
var response = client.send(request, HttpResponse.BodyHandlers.ofString());
System.out.println(response.statusCode());
System.out.println(response.body());
The same request executed asynchronously:
var responseAsync = client
.sendAsync(request, HttpResponse.BodyHandlers.ofString())
.thenApply(HttpResponse::body)
.thenAccept(System.out::println);
// responseAsync.join(); // Wait for completion
An example POST request:
var request = HttpRequest
.newBuilder()
.uri(uri)
.version(HttpClient.Version.HTTP_2)
.timeout(Duration.ofMinutes(1))
.header("Content-Type", "application/json")
.header("Authorization", "Bearer fake")
.POST(BodyPublishers.ofString("{ title: 'This is cool' }"))
.build();
var response = client.send(request, HttpResponse.BodyHandlers.ofString());
For sending form data as multipart (multipart/form-data) or url-encoded (application/x-www-form-urlencoded) format, see this solution.
See this article for examples and more information about HTTP Client API.
Sidenote
For Java standard library HTTP server, see this post.
I try to send a SOAP message in an XML file to a webservice and than grab the binary output and decode it. Endpoint uses HTTPS protocol, so I used TrustManager in my code to avoid PKIX problems. You can see my code here:
import javax.net.ssl.*;
import java.io.*;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLConnection;
import java.security.cert.X509Certificate;
public class Main{
public static void sendSoapRequest() throws Exception {
String SOAPUrl = "URL HERE";
String xmlFile2Send = ".\\src\\request.xml";
String responseFileName = ".\\src\\response.xml";
String inputLine;
TrustManager[] trustAllCerts = new TrustManager[] { new X509TrustManager() {
public java.security.cert.X509Certificate[] getAcceptedIssuers() { return null; }
public void checkClientTrusted(X509Certificate[] certs, String authType) { }
public void checkServerTrusted(X509Certificate[] certs, String authType) { }
} };
SSLContext sc = SSLContext.getInstance("SSL");
sc.init(null, trustAllCerts, new java.security.SecureRandom());
HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
// Create all-trusting host name verifier
HostnameVerifier allHostsValid = new HostnameVerifier() {
public boolean verify(String hostname, SSLSession session) { return true; }
};
// Install the all-trusting host verifier
HttpsURLConnection.setDefaultHostnameVerifier(allHostsValid);
// Create the connection with http
URL url = new URL(SOAPUrl);
URLConnection connection = url.openConnection();
HttpURLConnection httpConn = (HttpURLConnection) connection;
FileInputStream fin = new FileInputStream(xmlFile2Send);
ByteArrayOutputStream bout = new ByteArrayOutputStream();
copy(fin, bout);
fin.close();
byte[] b = bout.toByteArray();
StringBuffer buf=new StringBuffer();
String s=new String(b);
b=s.getBytes();
// Set the appropriate HTTP parameters.
httpConn.setRequestProperty("Content-Length", String.valueOf(b.length));
httpConn.setRequestProperty("Content-Type", "text/xml; charset=utf-8");
httpConn.setRequestProperty("SOAPAction", "");
httpConn.setRequestMethod("POST");
httpConn.setDoOutput(true);
OutputStream out = httpConn.getOutputStream();
out.write(b);
out.close();
// Read the response.
httpConn.connect();
System.out.println("http connection status :"+ httpConn.getResponseMessage());
InputStreamReader isr = new InputStreamReader(httpConn.getInputStream());
BufferedReader in = new BufferedReader(isr);
while ((inputLine = in.readLine()) != null)
System.out.println(inputLine);
FileOutputStream fos=new FileOutputStream(responseFileName);
copy(httpConn.getInputStream(),fos);
in.close();
}
public static void copy(InputStream in, OutputStream out) throws IOException {
synchronized (in) {
synchronized (out) {
byte[] buffer = new byte[256];
while (true) {
int bytesRead = in.read(buffer);
if (bytesRead == -1)
break;
out.write(buffer, 0, bytesRead);
}
}
}
}
public static void main(String args[]) throws Exception {
sendSoapRequest();
}
}
I get following error code, when I execute this.
Exception in thread "main" java.io.IOException: Server returned HTTP
response code: 403 for URL
Your implementation is alright, the problem is related to your Content-Type header, in fact.
The value text/xml; charset=utf-8 is the default Content-Type of SOAP 1.1, which is probably not the version of yours. SOAP 1.2 expects a header of type application/soap+xml; charset=utf-8, so changing your line of code to this one below is gonna make it working:
httpConn.setRequestProperty("Content-Type", "application/soap+xml; charset=utf-8");
In SoapUI, it's possible to check the headers calling the request and going to the Headers tab on the bottom of the window:
Then, you can compare the differences between your application configs and the SoapUI ones.
403 error might be related to your soap request headers being sent to the server.
All Hosts valid will allow your Java App to trust the SSL Cert for the URL.
Check if your server is expecting soap header with username/password. If you have access to this server, you can check through the web server logs on where your request is failing. Error code points to to missing Soap Header particularly Soap Headers with username and password
Wonder if your SOAP request contains any kind of authentication information in headers like SAML. One option is, in your above code where you read the file and send the data to server, instead of sending it to server you dump it to another file. Dump that byteoutputstream. Then copy text from that file and put it in SOAP UI and try running that. Does that work?
In a similar situation we have been some time before, and as long as trying TrustManager didn't work as expected, we managed to overcome this problem by installing the certificate from server to JVM's keystore (JVM used to run the application). More information about how to do it you can find in several posts, like
How to import a .cer certificate into a java keystore?
I am aware that it is a try to force JVM to accept SSL certificates, and this functionality would be better to live in application context, but as long as we were building a web application which ran in specific application servers, the implemented solution was an accepted one.
In a script I have, I've created a small and simple REST client. The script itself is a prototype, and therefore the code is not 'production worthy' - so ignore lazy catch expressions and alike.
There are two types of servers that contain the REST service that I fetch data from; either a WildFly 8.2.0 or a GlassFish 3.1.2.2. And the catch here is: My REST client works fine for fetching data from the Wildfly server, but the GlassFish server returns an HTTP 400 Bad Request, for any request.
I can access the REST service for both servers through a web browser, so I know that they are both working properly. I can even do a raw connection though a socket to both servers and they response with the correct data.
So, what could be the reason for GlassFish to not accept the requests?
Socket connection (for testing)
import java.net.Socket;
Socket s = new Socket("localhost", 8080);
String t = "GET /rest/appointment/appointments/search/?fromDate=2016-11-21&branchId=3 HTTP/1.1\nhost: localhost:8080\nAuthorization: Basic base64encodedUsername:PasswordHere\n\n"
OutputStream out = s.getOutputStream();
out.write(t.getBytes());
InputStream inn = s.getInputStream();
Scanner scan = new Scanner(inn);
String line;
while ((line = scan.nextLine()) != null) {
println line;
}
s.close();
REST client code:
import groovy.json.JsonSlurper;
import javax.xml.bind.DatatypeConverter;
/*
REST-client (a very simple one)
*/
public class RESTclient {
public static Object get(URL url, Map<String, String> headers) {
return http(url, "GET", null, headers);
}
public static Object post(URL url, String data, Map<String, String> headers) {
return http(url, "POST", data, headers);
}
public static Object put(URL url, String data, Map<String, String> headers) {
return http(url, "PUT", data, headers);
}
public static Object delete(URL url, String data, Map<String, String> headers) {
return http(url, "DELETE", data, headers);
}
private static Object http(URL url, String method, String data, Map<String, String> headers) {
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
Authenticator.setDefault(new Authenticator() {
protected PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication("username", "password".toCharArray());
}
});
connection.setRequestMethod(method);
for (String header : headers.keySet()) {
connection.setRequestProperty(header, headers.get(header));
}
if (data != null) {
connection.setRequestProperty("Content-Type", "application/json");
connection.setDoOutput(true);
OutputStream outputStream =connection.getOutputStream();
outputStream.write(data.getBytes());
}
int responseCode = connection.getResponseCode();
switch (responseCode) {
case HttpURLConnection.HTTP_NO_CONTENT:
// This happens when the server doesn't give back content, but all was ok.
return (new HashMap());
case HttpURLConnection.HTTP_OK:
InputStream inputStream = connection.getInputStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream));
String response = reader.readLine();
JsonSlurper parser = new JsonSlurper();
Object jsonResponse = parser.parseText(response); // This can be either a List or a Map
// Close the connection
try { connection.close(); } catch (Exception e) { /* Already closed */ }
return jsonResponse;
default:
println "response code: " + responseCode;
println connection.getResponseMessage();
println connection.getHeaderFields();
// Close the connection
try { connection.close(); } catch (Exception e) { /* Already closed */ }
return null;
}
}
}
Usage:
URL appointmentSearchURL = new URL("http://localhost:8080/rest/appointment/appointments/search/?fromDate=2016-11-21&branchId=3");
Object response = RESTclient.get(appointmentSearchURL, new HashMap<String, String>());
println response;
All that is printed out:
response code: 400
Bad Request
[null:[HTTP/1.1 400 Bad Request], Server:[GlassFish Server Open Source Edition 3.1.2.2], Connection:[close], Set-Cookie:[rememberMe=deleteMe; Path=/; Max-Age=0; Expires=Tue, 22-Nov-2016 08:43:29 GMT, SSOcookie=2a86cf4b-a772-435a-b92e-f12845dc20a2; Path=/; HttpOnly], Content-Length:[1090], Date:[Wed, 23 Nov 2016 08:43:28 GMT], Content-Type:[text/html], X-Powered-By:[Servlet/3.0 JSP/2.2 (GlassFish Server Open Source Edition 3.1.2.2 Java/Oracle Corporation/1.7)]]
null
I found my answer! So, I will leave this here if any other stumble across the same issue in the future:
There was a missing Accept header, I guess the server-side only accept json content. I have not researched further on why the WildFly server does not response with a 400 bad request, but I suppose WildFly tries to guess/deduce the incoming data.
So the whole issue was resolved by adding the following:
connection.setRequestProperty("Accept", "application/json");
I have been using the following code to connect to one of google's service. This code worked fine on my local machine :
HttpClient client=new DefaultHttpClient();
HttpPost post = new HttpPost("https://www.google.com/accounts/ClientLogin");
post.setEntity(new UrlEncodedFormEntity(myData));
HttpResponse response = client.execute(post);
I put this code in a production environment, which had blocked Google.com. On request, they allowed communication with Google server by allowing me to accessing an IP : 74.125.236.52 - which is one of Google's IPs. I edited my hosts file to add this entry too.
Still I could not access the URL, which I wonder why. So I replaced the above code with :
HttpPost post = new HttpPost("https://74.125.236.52/accounts/ClientLogin");
Now I get an error like this :
javax.net.ssl.SSLException: hostname in certificate didn't match:
<74.125.236.52> != <www.google.com>
I guess this is because Google has multiple IPs. I cant ask the network admin to allow me access to all those IPs - I may not even get this entire list.
What should I do now ? Is there a workaround at Java level ? Or is it totally in hands of the network guy ?
You can also try to set a HostnameVerifier as described here. This worked for me to avoid this error.
// Do not do this in production!!!
HostnameVerifier hostnameVerifier = org.apache.http.conn.ssl.SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER;
DefaultHttpClient client = new DefaultHttpClient();
SchemeRegistry registry = new SchemeRegistry();
SSLSocketFactory socketFactory = SSLSocketFactory.getSocketFactory();
socketFactory.setHostnameVerifier((X509HostnameVerifier) hostnameVerifier);
registry.register(new Scheme("https", socketFactory, 443));
SingleClientConnManager mgr = new SingleClientConnManager(client.getParams(), registry);
DefaultHttpClient httpClient = new DefaultHttpClient(mgr, client.getParams());
// Set verifier
HttpsURLConnection.setDefaultHostnameVerifier(hostnameVerifier);
// Example send http request
final String url = "https://encrypted.google.com/";
HttpPost httpPost = new HttpPost(url);
HttpResponse response = httpClient.execute(httpPost);
The certificate verification process will always verify the DNS name of the certificate presented by the server, with the hostname of the server in the URL used by the client.
The following code
HttpPost post = new HttpPost("https://74.125.236.52/accounts/ClientLogin");
will result in the certificate verification process verifying whether the common name of the certificate issued by the server, i.e. www.google.com matches the hostname i.e. 74.125.236.52. Obviously, this is bound to result in failure (you could have verified this by browsing to the URL https://74.125.236.52/accounts/ClientLogin with a browser, and seen the resulting error yourself).
Supposedly, for the sake of security, you are hesitant to write your own TrustManager (and you musn't unless you understand how to write a secure one), you ought to look at establishing DNS records in your datacenter to ensure that all lookups to www.google.com will resolve to 74.125.236.52; this ought to be done either in your local DNS servers or in the hosts file of your OS; you might need to add entries to other domains as well. Needless to say, you will need to ensure that this is consistent with the records returned by your ISP.
I had similar problem. I was using Android's DefaultHttpClient. I have read that HttpsURLConnection can handle this kind of exception. So I created custom HostnameVerifier which uses the verifier from HttpsURLConnection. I also wrapped the implementation to custom HttpClient.
public class CustomHttpClient extends DefaultHttpClient {
public CustomHttpClient() {
super();
SSLSocketFactory socketFactory = SSLSocketFactory.getSocketFactory();
socketFactory.setHostnameVerifier(new CustomHostnameVerifier());
Scheme scheme = (new Scheme("https", socketFactory, 443));
getConnectionManager().getSchemeRegistry().register(scheme);
}
Here is the CustomHostnameVerifier class:
public class CustomHostnameVerifier implements org.apache.http.conn.ssl.X509HostnameVerifier {
#Override
public boolean verify(String host, SSLSession session) {
HostnameVerifier hv = HttpsURLConnection.getDefaultHostnameVerifier();
return hv.verify(host, session);
}
#Override
public void verify(String host, SSLSocket ssl) throws IOException {
}
#Override
public void verify(String host, X509Certificate cert) throws SSLException {
}
#Override
public void verify(String host, String[] cns, String[] subjectAlts) throws SSLException {
}
}
A cleaner approach ( only for test environment) in httpcliet4.3.3 is as follows.
SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(sslContext,SSLConnectionSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);
CloseableHttpClient httpclient = HttpClients.custom().setSSLSocketFactory(sslsf).build();
In httpclient-4.3.3.jar, there is another HttpClient to use:
public static void main (String[] args) throws Exception {
// org.apache.http.client.HttpClient client = new DefaultHttpClient();
org.apache.http.client.HttpClient client = HttpClientBuilder.create().build();
System.out.println("HttpClient = " + client.getClass().toString());
org.apache.http.client.methods.HttpPost post = new HttpPost("https://www.rideforrainbows.org/");
org.apache.http.HttpResponse response = client.execute(post);
java.io.InputStream is = response.getEntity().getContent();
java.io.BufferedReader rd = new java.io.BufferedReader(new java.io.InputStreamReader(is));
String line;
while ((line = rd.readLine()) != null) {
System.out.println(line);
}
}
This HttpClientBuilder.create().build() will return org.apache.http.impl.client.InternalHttpClient. It can handle the this hostname in certificate didn't match issue.
Thanks Vineet Reynolds. The link you provided held a lot of user comments - one of which I tried in desperation and it helped. I added this method :
// Do not do this in production!!!
HttpsURLConnection.setDefaultHostnameVerifier( new HostnameVerifier(){
public boolean verify(String string,SSLSession ssls) {
return true;
}
});
This seems fine for me now, though I know this solution is temporary. I am working with the network people to identify why my hosts file is being ignored.
The concern is we should not use ALLOW_ALL_HOSTNAME_VERIFIER.
How about I implement my own hostname verifier?
class MyHostnameVerifier implements org.apache.http.conn.ssl.X509HostnameVerifier
{
#Override
public boolean verify(String host, SSLSession session) {
String sslHost = session.getPeerHost();
System.out.println("Host=" + host);
System.out.println("SSL Host=" + sslHost);
if (host.equals(sslHost)) {
return true;
} else {
return false;
}
}
#Override
public void verify(String host, SSLSocket ssl) throws IOException {
String sslHost = ssl.getInetAddress().getHostName();
System.out.println("Host=" + host);
System.out.println("SSL Host=" + sslHost);
if (host.equals(sslHost)) {
return;
} else {
throw new IOException("hostname in certificate didn't match: " + host + " != " + sslHost);
}
}
#Override
public void verify(String host, X509Certificate cert) throws SSLException {
throw new SSLException("Hostname verification 1 not implemented");
}
#Override
public void verify(String host, String[] cns, String[] subjectAlts) throws SSLException {
throw new SSLException("Hostname verification 2 not implemented");
}
}
Let's test against https://www.rideforrainbows.org/ which is hosted on a shared server.
public static void main (String[] args) throws Exception {
//org.apache.http.conn.ssl.SSLSocketFactory sf = org.apache.http.conn.ssl.SSLSocketFactory.getSocketFactory();
//sf.setHostnameVerifier(new MyHostnameVerifier());
//org.apache.http.conn.scheme.Scheme sch = new Scheme("https", 443, sf);
org.apache.http.client.HttpClient client = new DefaultHttpClient();
//client.getConnectionManager().getSchemeRegistry().register(sch);
org.apache.http.client.methods.HttpPost post = new HttpPost("https://www.rideforrainbows.org/");
org.apache.http.HttpResponse response = client.execute(post);
java.io.InputStream is = response.getEntity().getContent();
java.io.BufferedReader rd = new java.io.BufferedReader(new java.io.InputStreamReader(is));
String line;
while ((line = rd.readLine()) != null) {
System.out.println(line);
}
}
SSLException:
Exception in thread "main" javax.net.ssl.SSLException: hostname in certificate didn't match: www.rideforrainbows.org != stac.rt.sg OR stac.rt.sg OR www.stac.rt.sg
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:231)
...
Do with MyHostnameVerifier:
public static void main (String[] args) throws Exception {
org.apache.http.conn.ssl.SSLSocketFactory sf = org.apache.http.conn.ssl.SSLSocketFactory.getSocketFactory();
sf.setHostnameVerifier(new MyHostnameVerifier());
org.apache.http.conn.scheme.Scheme sch = new Scheme("https", 443, sf);
org.apache.http.client.HttpClient client = new DefaultHttpClient();
client.getConnectionManager().getSchemeRegistry().register(sch);
org.apache.http.client.methods.HttpPost post = new HttpPost("https://www.rideforrainbows.org/");
org.apache.http.HttpResponse response = client.execute(post);
java.io.InputStream is = response.getEntity().getContent();
java.io.BufferedReader rd = new java.io.BufferedReader(new java.io.InputStreamReader(is));
String line;
while ((line = rd.readLine()) != null) {
System.out.println(line);
}
}
Shows:
Host=www.rideforrainbows.org
SSL Host=www.rideforrainbows.org
At least I have the logic to compare (Host == SSL Host) and return true.
The above source code is working for httpclient-4.2.3.jar and httpclient-4.3.3.jar.
Updating the java version from 1.8.0_40 to 1.8.0_181 resolved the issue.
SSLConnectionSocketFactory sslConnectionSocketFactory = new SSLConnectionSocketFactory(
SSLContexts.custom().loadTrustMaterial(null, new TrustSelfSignedStrategy()).build(),
SSLConnectionSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);
CloseableHttpClient httpClient = HttpClients.custom().setSSLSocketFactory(sslConnectionSocketFactory).build();
I am trying to send a request to a server using the HttpsUrlConnection class. The server has certificate issues, so I set up a TrustManager that trusts everything, as well as a hostname verifier that is equally lenient. This manager works just fine when I make my request directly, but it doesn't seem to be used at all when I send the request through a proxy.
I set my proxy settings like this:
Properties systemProperties = System.getProperties();
systemProperties.setProperty( "http.proxyHost", "proxyserver" );
systemProperties.setProperty( "http.proxyPort", "8080" );
systemProperties.setProperty( "https.proxyHost", "proxyserver" );
systemProperties.setProperty( "https.proxyPort", "8080" );
The TrustManager for the default SSLSocketFactory is set up like this:
SSLContext sslContext = SSLContext.getInstance( "SSL" );
// set up a TrustManager that trusts everything
sslContext.init( null, new TrustManager[]
{
new X509TrustManager()
{
public X509Certificate[] getAcceptedIssuers()
{
return null;
}
public void checkClientTrusted( X509Certificate[] certs, String authType )
{
// everything is trusted
}
public void checkServerTrusted( X509Certificate[] certs, String authType )
{
// everything is trusted
}
}
}, new SecureRandom() );
// this doesn't seem to apply to connections through a proxy
HttpsURLConnection.setDefaultSSLSocketFactory( sslContext.getSocketFactory() );
// setup a hostname verifier that verifies everything
HttpsURLConnection.setDefaultHostnameVerifier( new HostnameVerifier()
{
public boolean verify( String arg0, SSLSession arg1 )
{
return true;
}
} );
If I run the following code, I end up with an SSLHandshakException ("Remote host closed connection during handshake"):
URL url = new URL( "https://someurl" );
HttpsURLConnection connection = (HttpsURLConnection)url.openConnection();
connection.setDoOutput( true );
connection.setRequestMethod( "POST" );
connection.setRequestProperty( "Content-Type", "application/x-www-form-urlencoded" );
connection.setRequestProperty( "Content-Length", "0" );
connection.connect();
I assume I am missing some kind of setting having to do with using a proxy when dealing with SSL. If I don't use a proxy, my checkServerTrusted method gets called; this is what I need to happen when I am going through the proxy as well.
I don't usually deal with Java and I don't have much experience with HTTP/web stuff. I believe I have provided all the detail necessary to understand what I am trying to do. If this isn't the case, let me know.
Update:
After reading the article suggested by ZZ Coder, I made the following changes to the connection code:
HttpsURLConnection connection = (HttpsURLConnection)url.openConnection();
connection.setSSLSocketFactory( new SSLTunnelSocketFactory( proxyHost, proxyPort ) );
connection.setDoOutput( true );
connection.setRequestMethod( "POST" );
connection.setRequestProperty( "Content-Type", "application/x-www-form-urlencoded" );
connection.setRequestProperty( "Content-Length", "0" );
connection.connect();
The result (SSLHandshakeException) is the same. When I set the SLLSocketFactory here to the SSLTunnelSocketFactory (the class explained in the article), the stuff I did with the TrustManager and the SSLContext is overridden. Don't I still need that?
Another Update:
I modified the SSLTunnelSocketFactory class to use the SSLSocketFactory that uses my TrustManager that trusts everything. It doesn't appear that this has made any difference. This is the createSocket method of SSLTunnelSocketFactory:
public Socket createSocket( Socket s, String host, int port, boolean autoClose )
throws IOException, UnknownHostException
{
Socket tunnel = new Socket( tunnelHost, tunnelPort );
doTunnelHandshake( tunnel, host, port );
SSLSocket result = (SSLSocket)dfactory.createSocket(
tunnel, host, port, autoClose );
result.addHandshakeCompletedListener(
new HandshakeCompletedListener()
{
public void handshakeCompleted( HandshakeCompletedEvent event )
{
System.out.println( "Handshake finished!" );
System.out.println(
"\t CipherSuite:" + event.getCipherSuite() );
System.out.println(
"\t SessionId " + event.getSession() );
System.out.println(
"\t PeerHost " + event.getSession().getPeerHost() );
}
} );
result.startHandshake();
return result;
}
When my code calls connection.connect, this method is called, and the call to doTunnelHandshake is successful. The next line of code uses my SSLSocketFactory to create an SSLSocket; the toString value of result after this call is:
"1d49247[SSL_NULL_WITH_NULL_NULL: Socket[addr=/proxyHost,port=proxyPort,localport=24372]]".
This is meaningless to me, but it might be the reason things break down after this.
When result.startHandshake() is called, the same createSocket method is called again from, according to the call stack, HttpsClient.afterConnect, with the same arguments, except Socket s is null, and when it comes around to result.startHandshake() again, the result is the same SSLHandshakeException.
Am I still missing an important piece to this increasingly complicated puzzle?
This is the stack trace:
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:808)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1112)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1139)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1123)
at gsauthentication.SSLTunnelSocketFactory.createSocket(SSLTunnelSocketFactory.java:106)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:391)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:166)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:133)
at gsauthentication.GSAuthentication.main(GSAuthentication.java:52)
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:333)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:789)
... 8 more
HTTPS proxy doesn't make sense because you can't terminate your HTTP connection at the proxy for security reasons. With your trust policy, it might work if the proxy server has a HTTPS port. Your error is caused by connecting to HTTP proxy port with HTTPS.
You can connect through a proxy using SSL tunneling (many people call that proxy) using proxy CONNECT command. However, Java doesn't support newer version of proxy tunneling. In that case, you need to handle the tunneling yourself. You can find sample code here,
http://www.javaworld.com/javaworld/javatips/jw-javatip111.html
EDIT: If you want defeat all the security measures in JSSE, you still need your own TrustManager. Something like this,
public SSLTunnelSocketFactory(String proxyhost, String proxyport){
tunnelHost = proxyhost;
tunnelPort = Integer.parseInt(proxyport);
dfactory = (SSLSocketFactory)sslContext.getSocketFactory();
}
...
connection.setSSLSocketFactory( new SSLTunnelSocketFactory( proxyHost, proxyPort ) );
connection.setDefaultHostnameVerifier( new HostnameVerifier()
{
public boolean verify( String arg0, SSLSession arg1 )
{
return true;
}
} );
EDIT 2: I just tried my program I wrote a few years ago using SSLTunnelSocketFactory and it doesn't work either. Apparently, Sun introduced a new bug sometime in Java 5. See this bug report,
http://bugs.sun.com/view_bug.do?bug_id=6614957
The good news is that the SSL tunneling bug is fixed so you can just use the default factory. I just tried with a proxy and everything works as expected. See my code,
public class SSLContextTest {
public static void main(String[] args) {
System.setProperty("https.proxyHost", "proxy.xxx.com");
System.setProperty("https.proxyPort", "8888");
try {
SSLContext sslContext = SSLContext.getInstance("SSL");
// set up a TrustManager that trusts everything
sslContext.init(null, new TrustManager[] { new X509TrustManager() {
public X509Certificate[] getAcceptedIssuers() {
System.out.println("getAcceptedIssuers =============");
return null;
}
public void checkClientTrusted(X509Certificate[] certs,
String authType) {
System.out.println("checkClientTrusted =============");
}
public void checkServerTrusted(X509Certificate[] certs,
String authType) {
System.out.println("checkServerTrusted =============");
}
} }, new SecureRandom());
HttpsURLConnection.setDefaultSSLSocketFactory(
sslContext.getSocketFactory());
HttpsURLConnection
.setDefaultHostnameVerifier(new HostnameVerifier() {
public boolean verify(String arg0, SSLSession arg1) {
System.out.println("hostnameVerifier =============");
return true;
}
});
URL url = new URL("https://www.verisign.net");
URLConnection conn = url.openConnection();
BufferedReader reader =
new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line;
while ((line = reader.readLine()) != null) {
System.out.println(line);
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
This is what I get when I run the program,
checkServerTrusted =============
hostnameVerifier =============
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
......
As you can see, both SSLContext and hostnameVerifier are getting called. HostnameVerifier is only involved when the hostname doesn't match the cert. I used "www.verisign.net" to trigger this.
Try the Apache Commons HttpClient library instead of trying to roll your own:
http://hc.apache.org/httpclient-3.x/index.html
From their sample code:
HttpClient httpclient = new HttpClient();
httpclient.getHostConfiguration().setProxy("myproxyhost", 8080);
/* Optional if authentication is required.
httpclient.getState().setProxyCredentials("my-proxy-realm", " myproxyhost",
new UsernamePasswordCredentials("my-proxy-username", "my-proxy-password"));
*/
PostMethod post = new PostMethod("https://someurl");
NameValuePair[] data = {
new NameValuePair("user", "joe"),
new NameValuePair("password", "bloggs")
};
post.setRequestBody(data);
// execute method and handle any error responses.
// ...
InputStream in = post.getResponseBodyAsStream();
// handle response.
/* Example for a GET reqeust
GetMethod httpget = new GetMethod("https://someurl");
try {
httpclient.executeMethod(httpget);
System.out.println(httpget.getStatusLine());
} finally {
httpget.releaseConnection();
}
*/