We Keep Coding

Get Proxy Automatically

I have the following HttpConnection call to get and parse json object.
As you see, I am passing the proxy inside the code as follows.
However, I need to know how could I able to get proxy without passing it manually.
proxy= new Proxy(Proxy.Type.HTTP, new InetSocketAddress(proxyString, 80));
HttpURLConnection dataURLConnection = endURL.openConnection(proxy);
dataURLConnection .setRequestProperty ("Authorization", token);
dataURLConnection .setRequestMethod("GET");
InputStream response = dataURLConnection .getInputStream();
BufferedReader reader = new BufferedReader (new InputStreamReader(response));

I believe you are looking for the java.net.ProxySelector class, since 1.5.
Here's an example of it's functionality;
URI targetURI = new URI("http://stackoverflow.com/");
ProxySelector proxySelector = ProxySelector.getDefault();
List<Proxy> proxies = proxySelector.select(targetURI);
for(Proxy proxy : proxies) {
Proxy.Type proxyType = proxy.type(); //Will return a Proxy.Type (SOCKS, for example)
SocketAddress address = proxy.address(); //Returns null if no proxy is available
}
Edit: Just realized you're already using the Proxy class, so you can just use one of the resulting proxies directly in the HttpURLConnection, of course.

You could set proxy host and port on jvm system property, that way you don't have to create and pass the proxy while creating a new connection.
System.setProperty("https.proxyHost", "myproxy.com");
System.setProperty("https.proxyPort", "8080");
System.setProperty("http.proxyHost", "myproxy.com");
System.setProperty("http.proxyPort", "8080");
But, keep in mind, this will affect all new connections across the JVM once the proxy is set. If you were using Spring or some other framework, they may offer you option to contextually set the proxy.

Incredibly long wait times when trying to read content of secure https website in Java using basic approaches

After reading and trying various approaches to reading the content of websites I realised that I am unable to get the input stream of secure web pages (or have to wait for minutes for a single response). These are pages that I can easily access via a browser (no proxies involved).
The different fixes that I tried are the following:
Setting user agent
Following redirects
Using JSoup
Catering for encoding
Using Scanner to parse the stream
Using cookie manager
The two basic approaches that seem most popular, one using Jsoup:
Document doc = Jsoup.connect(url)
.userAgent(userAgent)
.timeout(5000).followRedirects(true).execute().parse();
Elements body = doc.select("body");
System.out.println(body.html());
The other with vanilla Java:
URL obj = new URL(url);
CookieHandler.setDefault(new CookieManager(null, CookiePolicy.ACCEPT_ALL));
HttpsURLConnection con = (HttpsURLConnection) obj.openConnection();
con.setRequestMethod("GET");
con.setRequestProperty("User-Agent", userAgent);
con.setConnectTimeout(5000);
con.setReadTimeout(5000);
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuilder response = new StringBuilder();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
The execution goes into a painfully slow loop in case of https addresses such as https://en.wikipedia.org/wiki/Java in the execution phase (so no status code can be retrieved from the response either), or when the input stream is being fetched in case of the second attempt. Both approaches work perfectly fine for insecure addresses (e.g. http://www.codingpedia.org/ ) - with a switch to HttpURLConnection.
Weirdly, there is also very high disc usage while waiting ( > 20 MB/sec).
All help greatly appreciated!

Connecting to an HTTPS connection from Java

I want to connect to an https url https://www.ovh.com/cgi-bin/sms/http2sms.cgi, I used the following code:
URL ovhUrl = new URL("https://www.ovh.com/cgi-bin/sms/http2sms.cgi");
HttpsURLConnection connection = (HttpsURLConnection)ovhUrl.openConnection(); // error here
String s_getBody = String.format("account=%s&login=%s&password=%s&from=%s&to=%s&message=%s",
URLEncoder.encode(account, "UTF-8"),
URLEncoder.encode(login, "UTF-8"),
URLEncoder.encode(password, "UTF-8"),
URLEncoder.encode(from, "UTF-8"),
URLEncoder.encode(to, "UTF-8"),
URLEncoder.encode(msg, "UTF-8"));
However, everytime I execute it, I get an exception at the second line:
weblogic.net.http.SOAPHttpsURLConnection cannot be cast to javax.net.ssl.HttpsURLConnection
I am using Weblogic Server, I found a solution here which is to add to the classpath, but I'd rather not change the classpath of my application or fiddling with Weblogic Server. Is there a another way to solve it, maybe by using another class or another method to be able to execute the https url?

Simply don't cast to HttpsURLConnection. Use
URLConnection connection = ovhUrl.openConnection();

I had same issue and setting handler to URL constructor helped me.
URL url = new URL(null, "https://www.ovh.com/cgi-bin/sms/http2sms.cgi", new sun.net.www.protocol.https.Handler());

Java set accept on http get

I am trying to send a request to a server using GET that will respond with XML. I am told that I need to set the "Accept" property, code follows:
StringBuffer url = new StringBuffer(BASE_URL);
url.append(DRS_SERVICE_RELATIVE_URL);
url.append("?").append(DOC_PARAM_NAME).append("=").append(docId);
url.append("&").append(DOB_PARAM_NAME).append("=").append(dob);
try
{
this.server = new URL(url.toString());
URLConnection urlCon = this.server.openConnection();
HttpURLConnection con = (HttpURLConnection)urlCon;
con.addRequestProperty("Accept", "text/xml, application/*+xml, application/xml, text/xml, application/*+xml");
con.connect();
input = new BufferedReader(new InputStreamReader(con.getInputStream()));
String line = null;
while((line = input.readLine()) != null)
System.out.println(line);
I get response code 500. When I talk to the developers of the URL I am trying to access they say I am not setting the "Accept" property to XML? What am I doing wrong? How are you supposed to set that property?
EDIT:
OK this is embarassing. The problem had to do with my development enviroment, specifically the way I set up a TCP/IP monitoring tool. When I stopped monitoring the network messages it worked as expected.

The problem had to do with my development enviroment, specifically the way I set up a TCP/IP monitoring tool. When I stopped monitoring the network messages it worked as expected.

HttpURLConnection returns 503 error when accessed through proxy

I am working on creating a Video sitemap for a site that has hosted videos on Brightcove video cloud. In order to get all the video information from the site, Brightcove suggests to read the response from their url of following form
http://api.brightcove.com/services/library?token="+accountToken+"&page_size=1&command=find_all_videos&output=JSON&get_item_count=true
the output of the url is in JSON, where accountToken is just an identifier of the account.
When I hit the above url with Token in the browser, it gives me the correct response.
I wrote below program snippet to read from that url
URL jsonURL = new URL("http://api.brightcove.com/services/library?token="+accountToken+"&page_size=1&command=find_all_videos&output=JSON&get_item_count=true");
HttpURLConnection connection = (HttpURLConnection) jsonURL.openConnection();
connection.setRequestMethod("GET");
connection.connect();
BufferedReader reader = new BufferedReader(
new InputStreamReader(connection.getInputStream()));
String lineRead = "";
while (reader.ready()) {
lineRead = lineRead + reader.readLine();
}
As my browser uses proxy, I added below code to include proxy settings
System.setProperty("http.proxyHost", "my.proxyurl.com");
System.setProperty("http.proxyPort", "80");
Without using proxy settings, it returns java.net.ConnectException: Connection refused: connect and with proxy it gives me java.io.IOException: Server returned HTTP response code: 503
So my question is , why is it giving me a 503(Service Unavailable) error ? From the browser its working fine.
Update 1:
It seems like an issue with the Network. I pinged the domain and it said "Request Timed out". Working via HTTP though. Looks like an issue with the Firewall.

I think, it may due to your internet connection, I have tried your code I didn't get any 503(Service Unavailable). Check out with different connection connection(without proxy) and it should work. Or you can try it with slightly different approach:
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("host", "port));
conn = new URL(jsonURL).openConnection(proxy);
If you have SOCKS type proxy, change Proxy's constructor parameter to Proxy.Type.SOCKS.

Minor correction to Jamas code
String host="myproxy.com";
int port=8080;
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(host, port));