We Keep Coding

How to make sure URLConnection is valid and will successfully connect in Java?

how can you ensure URL you receive from users is a valid url and not just a http://nothing.com
my code looks like this:
String urlFromUser = getUrlFromUser(); // might return: http://www.notARealSite.com
URL url = new URL(urlFromUser);
// this might fail
URLConnection urlConnection = url.openConnection();
and try catch is not enough, i want to make sure the site is real

you can also use:
if(new InetSocketAddress(urlFromUser, 80).isUnresolved()) {
// URL is a not a valid server address
}
else {
// URL is valid
}

Http POST in android app without data

I'm trying to send a video url in a http POST request but it's not working for me, I think I've (almost?) the necessary code to make it work, or else I'm missing something very simple?
public void postVideoURL() throws IOException {
String encodedUrl = URLEncoder.encode("http://video.ted.com/talks/podcast/DavidBrooks_2011.mp4", "UTF-8");
URL obj = new URL("http://10.50.0.105:8060/launch/dev?url="+encodedUrl);
HttpURLConnection con = (HttpURLConnection) obj.openConnection();
//add request header
con.setRequestMethod("POST");
// Send post request
con.setDoOutput(true);
OutputStreamWriter wr = new OutputStreamWriter(con.getOutputStream());
System.out.println(con.getResponseCode());
System.out.println(con.getResponseMessage());
wr.flush();
wr.close();
wr.write("");
}
Any tips to lead me to the right direction?
Here is what I'm trying to do but in C#
using System.Net;
using System.Web;
class Program
{
static void Main()
{
string rokuIp = "192.168.0.6";
string channelId = "dev";
string videoUrl = "http://video.ted.com/talks/podcast/DavidBrooks_2011.mp4";
// Note that the query string parameters should be url-encoded
string requestUrl = $"http://{rokuIp}:8060/launch/{channelId}?url={HttpUtility.UrlEncode(videoUrl)}";
using (var wc = new WebClient())
{
// POST the query string with no data
wc.UploadString(requestUrl, "");
}
}
}
The following Post command to use in terminal works, this is essentially what I want to do, but in Java:
curl -d "" "http://10.50.0.46:8060/launch/12?url=http%3A%2F%2Fvideo.ted.com%2Ftalks%2Fpodcast%2FDavidBrooks_2011.mp4"

You are never writing anything to the output stream. You have to call wr.write() to write your data to the stream.
Also, you can't encode the URL like that inside the String. You need to concatenate the two Strings together after you've encoded the url separately. Like this:
String encodedUrl = URLEncoder.encode("http://video.ted.com/talks/podcast/DavidBrooks_2011.mp4");
URL obj = new URL("http://10.50.0.105:8060/launch/dev?url="+encodedUrl);

Android: not getting xml out of http get request with basic authentication

My goal is to get the xml from an API. The API uri I use, including parameters is http://webservices.ns.nl/ns-api-treinplanner?fromStation=Roosendaal&toStation=Eindhoven. I am given a username and password, for what I think probably is basic authorization.
I tried various things like something with an Authenticator, the format http://username:password#webservices.ns.nl/ns-api-treinplanner, but at the end of a lot of SO searching I ended up with something with a setRequestProperty with the basic authorization.
I put the code into an AsyncTask which seems to work correctly so I will just put the code from inside doInBackground in here.
As the java FileNotFoundException I first got didn't give me much information, I found out how to use the getErrorStream to find out more.
InputStream in;
int resCode;
try {
URL url = new URL("http://webservices.ns.nl/ns-api-treinplanner?fromStation=Roosendaal&toStation=Eindhoven");
String userCredentials = "username:password";
String encoding = new String(android.util.Base64.encode(userCredentials.getBytes(), Base64.DEFAULT));
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
urlConnection.setRequestProperty("Authorization", "Basic " + encoding);
urlConnection.setRequestMethod("GET");
urlConnection.setDoOutput(true);
try {
resCode = urlConnection.getResponseCode();
if (resCode == HttpURLConnection.HTTP_OK) {
Log.i("rescode","ok");
in = urlConnection.getInputStream();
} else {
Log.i("rescode","not ok");
in = urlConnection.getErrorStream();
}
BufferedReader bufferedReader = new BufferedReader(
new InputStreamReader(in));
StringBuilder stringBuilder = new StringBuilder();
String line;
while ((line = bufferedReader.readLine()) != null) {
stringBuilder.append(line).append("\n");
}
bufferedReader.close();
return stringBuilder.toString();
}
finally{
urlConnection.disconnect();
}
}
catch(Exception e) {
Log.e("ERROR", e.getMessage(), e);
return null;
}
Then, in onPostExecute I print the response, but the response I get is
<?xml version="1.0"?>
<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope" soap:encodingStyle="http://www.w3.org/2003/05/soap-encoding">
<soap:Header></soap:Header>
<soap:Body><soap:Fault>
<faultcode>soap:Server</faultcode>
<faultstring>006:No customer found for the specified username and password</faultstring></soap:Fault>
</soap:Body></soap:Envelope>
This is of course not right, it should give a full xml of in this case a train voyage recommendation.
I tested with my browsers, and also using a HTTP request tool called Postman which returned the correct xml so all the uri's, parameters, username and password are correct.

The encoding used is wrong. The base64 encoding used randomly returns whitespaces in the middle, adding encoding = encoding.replaceAll("\\s+",""); actually fixed it.

How to enable cookies when opening a URL's connection

I am trying to use the following code to get the redirected URL and then do some processing on it. But when I print the redirected link, it goes to a page that informs the absence of cookies. how is it possible to enable cookies while opening the url's connection?
String url = "http://dx.doi.org/10.1137/0210059";
URLConnection con = new URL( url ).openConnection();
con.getInputStream();
String redirctedURL= con.getURL().toString();
System.out.println(redirctedURL);

When using java UrlConnection's, you should handle the cookies by yourself, to read and set the cookies, you can use the setRequestProperty() and getHeaderField() of URLConnection.
The remaining part is parsing the cookies by yourself, an example how the can be done is as follows:
Map<String, String> cookie = new HashMap<>();
public URLConnection doConnctionWithCookies(URL url) {
StringBuilder builder = new StringBuilder();
builder.append("&");
for(Map.Entry<String,String> entry : cookie.entrySet()) {
builder.append(urlenEncode(entry.getKey()))
.append("=")
.append(urlenEncode(entry.getValue()))
.append("&");
}
builder.setLength(builder.length() - 1);
URLConnection con = url.openConnection();
con.setRequestProperty("Cookie", builder.toString());
con.connect();
// Parse cookie headers
List<String> setCookie = con.getHeaderFields().get("set-cookie");
// HTTP Spec state header fields MUST be case INsentive, I expet the above to work with all servers
if(setCookie == null)
return con;
// Parse all the cookies
for (String str : setCookie) {
String[] cookieKeyValue = str.split(";")[0].split("=",2);
if (cookieKeyValue.length != 2) {
continue;
}
cookie.put(urlenDecode(cookieKeyValue[0]), urlenDecode(cookieKeyValue[1]));
}
return con;
}
public String urlenEncode(String en) {
return URLEncoder.encode(en, "UTF-8");
}
public String urlenDecode(String en) {
return URLDecoder.decode(en, "UTF-8");
}
The above implementation is a very stupid and crude way of implementation cookies, while it works, it totally ignores the fact that cookies can also have a host parameter to prevent identification cookies be shared across multiple hosts.
A better way than doing it yourself can be using a library dedicated for the task like Apache HttpClient.

A question on webpage representation in Java

I've followed a tutorial and came up with the following method to read the webpage content into a CharSequence
public static CharSequence getURLContent(URL url) throws IOException {
URLConnection conn = url.openConnection();
String encoding = conn.getContentEncoding();
if (encoding == null) {
encoding = "ISO-8859-1";
}
BufferedReader br = new BufferedReader(new
InputStreamReader(conn.getInputStream(),encoding));
StringBuilder sb = new StringBuilder(16384);
try {
String line;
while ((line = br.readLine()) != null) {
sb.append(line);
sb.append('\n');
}
} finally {
br.close();
}
return sb;
}
It will return a representation of the webpage specified by the url.
However,this representation is hugely different from what I use "view page source" in my Firefox,and since I need to scrape data from the original webpage(some data segement in the original "view page source" file),it will always fail to find required text on this Java representation.
Did I go wrong somewhere?I need your advice guys,thanks a lot for helping!

You need to use an HTML-parsing library to build a data structure representing the HTML text on this webpage. My recommendation is to use this library: http://htmlparser.sourceforge.net.

Things like the request useragent and cookies can change what the server returns in the response. So the problem is more likely in the details of the request you are sending rather than in how you are reading the response.
Things like HttpClient will allow you to more easily simulate the request being sent from a browser.