We Keep Coding

How can I set https.proxyHost and https.proxyPort for individual HttpsURLConnections?

I'm trying to make HTTPS requests through a proxy. Here's what I've got so far, based on code from this question:
try {
HttpsURLConnection connection = (HttpsURLConnection) new URL("https://proxylist.geonode.com/api/proxy-list?limit=1&page=1&sort_by=speed&sort_type=asc&protocols=https").openConnection();
connection.setRequestMethod("GET");
connection.setDoInput(true);
connection.setDoOutput(true);
connection.setRequestProperty("user-agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36");
connection.setConnectTimeout(30000);
connection.connect();
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String rawJSON = reader.readLine();
if(rawJSON == null) throw new IOException("No data");
JSONObject data = new JSONObject(rawJSON).getJSONArray("data").getJSONObject(0);
String ipAddress = data.getString("ip"), port = data.getString("port");
System.setProperty("https.proxyHost", ipAddress);
System.setProperty("https.proxyPort", port);
SSLContext sslContext = SSLContext.getInstance("SSL");
// set up a TrustManager that trusts everything
sslContext.init(null, new TrustManager[] { new X509TrustManager() {
public X509Certificate[] getAcceptedIssuers() { return null; }
public void checkClientTrusted(X509Certificate[] certs, String authType) {}
public void checkServerTrusted(X509Certificate[] certs, String authType) {}
} }, new SecureRandom());
HttpsURLConnection.setDefaultSSLSocketFactory(sslContext.getSocketFactory());
HttpsURLConnection.setDefaultHostnameVerifier((arg0, arg1) -> true);
HttpsURLConnection testConnection = (HttpsURLConnection) new URL("https://example.com").openConnection();
testConnection.connect();
StringBuilder result = new StringBuilder();
String line;
try(BufferedReader reader2 = new BufferedReader(new InputStreamReader(testConnection.getInputStream()))) {
while ((line = reader2.readLine()) != null) result.append(line);
}
System.out.println(result);
} catch(Exception e) {
e.printStackTrace();
}
The code works, but there's a problem. My application (https://encyclosearch.org) is multithreaded, and I need to make some requests through a proxy, and some directly. Since system properties are global, if I set https.proxyHost and https.proxyPort using System.setProperty, some requests that aren't supposed to go through the proxy will go through the proxy.
I can use java.net.Proxy like this:
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(ipAddress, Integer.parseInt(port)));
HttpsURLConnection testConnection = (HttpsURLConnection) new URL("http://example.com").openConnection(proxy);
But that only works for HTTP proxies, not HTTPS ones, so I can't make HTTPS requests. There's no Proxy.Type.HTTPS.
Any help would be greatly appreciated. Thanks in advance.

If you choose which connections go through a proxy and which do not by the destination url then you can use the property http.nonProxyHosts. This property is used for http and https as the documentation states:
for the "non proxy hosts" list, the HTTPS protocol handler will use the same as the http handler (i.e. http.nonProxyHosts).
You set the property value by adding patterns of urls separated by | For example:
System.setProperty("http.nonProxyHosts", ”localhost|host.example.com”)
You could also use ProxySelector class for choosing which connections go through a proxy. for more info (a bit old):
https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html
As for the absence of Proxy.Type.HTTPS, it is because a proxy in general is not the final destination so the secure connection will be with the final destination not the proxy itself. There is such thing as SSL tunneling through a proxy, but i am not well informed about it.

With #Bashi's help, I figured it out. For direct connections, I used:
url.openConnection(Proxy.NO_PROXY);
This works for Jsoup, too:
Document document = Jsoup.connect("https://example.com").proxy(Proxy.NO_PROXY).get();
Explanation from https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html:
Now, this guarantees you that this specific URL will be retrieved though a direct connection bypassing any other proxy settings, which can be convenient.

Open webpage with Socks5 proxy in JAVA

I am trying to open a page URL using Proxy via Socks5 (Already created on server-side)
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("10.0.0.1", 8080));
conn = new URL(urlString).openConnection(proxy);
//Connection
Authenticator authenticator = new Authenticator() {
public PasswordAuthentication getPasswordAuthentication() {
return (new PasswordAuthentication("user",
"password".toCharArray()));
}
};
Authenticator.setDefault(authenticator);
Now I need to open (and closing option?) an URL using this Proxy connection...

Now I need to open (and closing option?) an URL using this Proxy connection...
You have already done so. Just get the input stream and proceed as usual.

pinnaclesports api basic auth with java

When trying to get account balance used api I got an exception:
IOException: Server returned HTTP response code: 401 for URL:
https://api.pinnaclesports.com/v1/client/balance
Here's the code:
public void getBalance() throws Exception{
byte[] bytes = "username:password".getBytes("UTF-8");
String encoded = Base64.encodeBase64String(bytes);
System.out.println(encoded);
URL api = new URL("https://api.pinnaclesports.com/v1/client/balance");
URLConnection apiConnection = api.openConnection();
apiConnection.addRequestProperty("Authorization", "Basic "+encoded);
BufferedReader in = new BufferedReader(new InputStreamReader(apiConnection.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null)
System.out.println(inputLine);
in.close();
}

I think your authentication is broken or not set properly. The API uses Basic Auth and a Base64 encoded username/password pair. You should read http://www.pinnaclesports.com/en/api/manual#authentication and make sure that your authorization is correct.
Here's an explanation for HTTP status code 401:
Similar to 403 Forbidden, but specifically for use when authentication
is possible but has failed or not yet been provided. The response
must include a WWW-Authenticate header field containing a challenge
applicable to the requested resource.

Try using an Authenticator, like this:
Authenticator.setDefault(new Authenticator() {
#Override
protected PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication("username", "password".toCharArray());
}
});
URL api = new URL("https://api.pinnaclesports.com/v1/client/balance");
BufferedReader in = new BufferedReader(new InputStreamReader(api.openStream()));
Or if that doesn't work, then try using an HttpURLConnection instead of URLConnection,
like this:
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("POST");
connection.setDoOutput(true);
connection.setRequestProperty ("Authorization", "Basic " + encoded);
You might also find this related post helpful, using Apache Commons.

download file from url with authentication

i have a secured url , if i open in browser, a pop up comes and i authenitcate with userid/password, then a pdf file is downloaded .
I want this functionality in java. It should authenticate using proxy server/port and user details then download
URL server = new URL("http://some.site.url/download.aspx?client=xyz&docid=1001");
System.setProperty("https.proxyUser", userName);
System.setProperty("https.proxyPassword", password);
System.setProperty("https.proxyHost",proxy);
System.setProperty("https.proxyPort",port);
URLConnection connection = (URLConnection)server.openConnection();
connection.connect();
InputStream is = connection.getInputStream();
//then i read this input stream and write to a pdf file in temporary folder
It gives me connection timeout error.
Then i thought adding authentication
String authentication = "Basic " + new
sun.misc.BASE64Encoder().encode("myuserid:mypassword".getBytes());
connection.setRequestProperty("Proxy-Authorization", authentication);
Still doesnt work,
Please let me know .

I solved this issue.
I used a customized authenticator before connecting the URL, and it authenticates and downloads the document. FYI - once connected, till next server restart, it doesn't need authentication.
URL server = new URL(url); //works for https and not for http, i needed https in my case.
Authenticator.setDefault((new MyAuthenticator()));
URLConnection connection = (URLConnection)server.openConnection();
connection.connect();
InputStream is = connection.getInputStream();
.... //write code to fetch inputstream
Define your own authenticator as given below
public class MyAuthenticator extends Authenticator {
final PasswordAuthentication authentication;
public MyAuthenticator(String userName, String password) {
authentication = new PasswordAuthentication(userName, password.toCharArray());
}
}

Java HTTP PUT with Digest authentication in Java

I am trying to upload a file with Java using PUT, server does Digest authentication. I want to keep it lean, so I try to use HttpURLConnection.
public void putData(String path, byte [] data) throws IOException, MalformedURLException {
Authenticator.setDefault(new Authenticator() {
protected PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication(user,password.toCharArray());
}});
debug("Default authenticator set");
//Safeguard against double slashes
if (path.startsWith("/")) {
path = path.replaceFirst("/","");
}
debug(hostname + path);
URL url = new URL(hostname + path);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
debug("HttpURLConnection acquired");
conn.setDoOutput(true);
conn.setDoInput(true);
conn.setRequestMethod("PUT");
conn.setRequestProperty("Content-Length",String.valueOf(data.length));
conn.setRequestProperty("Content-type","application/binary");
conn.setFixedLengthStreamingMode(data.length);
conn.connect();
debug("Properties set");
OutputStream out = conn.getOutputStream();
debug("Outputstrem acquired");
out.write(data,0,data.length);
out.flush();
out.close();
debug("Data written, stream closed.");
}
For some reason, this fails hopelessly: I see a 401 coming back, and then it's done. If I disable authorization, same code works. Downloading a file with similar code using digest authentication "just works". Any ideas? I'd really rather not start using the next so many libraries like htclient from Apache or so (...it's 2010... you'd expect http requests with digest authN to work in any standard library).

You should at least try conn.getInputStream() at the end of your method, to force evaluation of the server response. Otherwise, potential error messages from the server will not be detected properly.