We Keep Coding

Easy way to check if a link is reachable or not from a java aplication [duplicate]

This question already has answers here:
How to check if a URL exists or returns 404 with Java?
(5 answers)
Closed 1 year ago.
I have a long list of links (absolute URLs) stored in a text file. I need to find out where the link is dead (Web page does not exist any more under the given adress). Example
Android http://www.android.com/
stackoverflow https://stackoverflow.com/
AIMS Desktop https://desktop.aims.ac.za/
google http://www.google.com/
blahblah http://www.ffgfgfgkzu.com
I do not care if there is a redirect, from http to https (if i type http://www.google.com/ in my browser this would be redirected to https://www.google.com/) or any other page which can have a total different url. I am only interested in finding out dead links, like the last entry above, where my browser also result in: (german text for page not found)
I have looked in to Selenium and some other web scraping tutorials. I don't want to scrape any content. I only need to remove the dead links from my list.

You can send HEAD request to the URL and see what response code you are getting. If response code is 404 then you say the URL is not exists. The HEAD request is much faster than GET. Also, The HEAD request will not return response body. This is a standard way to check if URL is exists or not. Please see below code snippet which uses apache HttpClient version 4.5.5 to check if URL exists or not:
/**
* #param url
* #return
*/
public static boolean isReachable(String url) {
boolean isReachable = true;
try (CloseableHttpClient httpClient = HttpClients.custom()
.setSSLContext(new SSLContextBuilder().loadTrustMaterial(null, TrustAllStrategy.INSTANCE).build())
.setSSLHostnameVerifier(NoopHostnameVerifier.INSTANCE)
.build())
{
HttpHead request = new HttpHead(url);
CloseableHttpResponse response = httpClient.execute(request);
if (response.getStatusLine().getStatusCode() == 404) {
System.out.println("URL " + url + " Not found");
isReachable = false;
}
} catch (Exception e) {
e.printStackTrace();
isReachable = false;
}
return isReachable;
}

InetAddress has a function to check for availability InetAddress.isReachable():
boolean reachable = InetAddress.getByName(host).isReachable();
Or if your prefer to catch and check exceptions you can also use Sockets.
public static boolean pingHost(String host, int port, int timeout) {
try (Socket socket = new Socket()){
socket.connect(new InetSocketAddress(host, port), timeout);
return true;
} catch (IOException e) {
return false;
}
}
Or HttpUrlConnection:
public static boolean isInternetReachable(String urlStr)
{
try {
URL url = new URL(urlStr);
HttpURLConnection urlConnection = (HttpURLConnection)url.openConnection();
urlConnection.setInstanceFollowRedirects(true);
Object objData = urlConnection.getContent();
} catch (Exception e) {
e.printStackTrace();
return false;
}
return true;
}

How to login to server and then hit a url on the same server to get JSON in return

Whenever I try to hit a url using java it will redirect me to login page. How can I first login then hit a specific url to get JSON in return ?
Here what I tried so far:
try {
URL url = new URL(GET_EXPENSE_FOR_VENDOR_URL);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.setRequestProperty("Accept", "application/json");
BufferedReader br = new BufferedReader(new InputStreamReader((conn.getInputStream())));
String response;
System.out.println("Output from Server .... \n");
while ((response = br.readLine()) != null) {
System.out.println(response);
Gson gson = new Gson();
gson.fromJson(response, ExpenseAllocationDTO[].class);
Type collectionType = new TypeToken<Collection<ExpenseAllocationDTO>>() {
}.getType();
expenseAllocationList = gson.fromJson(response, collectionType);
expenseAllocationDTODataModel = (new ExpenseAllocationDTODataModel(expenseAllocationList));
if (expenseAllocationList.isEmpty() || expenseAllocationList == null) {
expenseExists = true;
}
conn.disconnect();
}
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}

The problem
I believe the initial request is missing some headers. Only the Accept header is set.
How to solve it ?
Option #1
In order to discover the missing headers, open your favorite browser and browse to GET_EXPENSE_FOR_VENDOR_URL. Before browsing, open the webdeveloper toolbar in order to see headers sent by the browser.
Here is a sample screenshot of the webdeveloper toolbar under Chrome on Windows.
.
Option #2
If your browser doesn't have such a toolbar, you can use a tool like Fiddler for finding the missing headers.
Option #3
You can also use a tool like hurl.it in order to test the headers expected by the target server as you discover them. IMO, this tool can be more straight forward than Fiddler during the debugging phase.
Get back to your code
Once you have identified the missing headers, add them to your Java code like this:
URL url = new URL(GET_EXPENSE_FOR_VENDOR_URL);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
conn.setRequestProperty("Accept", "application/json");
conn.setRequestProperty("Missing-Header-1", "...");
conn.setRequestProperty("Missing-Header-2", "...");

Check if URL is HTTPS or HTTP protocol?

I am currently using the following to read a file from android docs here and here. The user selects (in the settings screen) if their site uses HTTP or HTTPS protocol. If their website uses the HTTP protocol then it works for both HttpURLConnection and HttpsURLConnection, but if their site uses HTTPS protocol then it doesn't work for HttpURLConnection protocol and worst of all it doesn't give me an exception error. Below is the sample code that I am using.
So in essence, how can I check to see if the web url is HTTPS protocol so checking if the user selected the correct protocol?
InputStream inputStream;
HttpURLConnection urlConnection;
HttpsURLConnection urlHttpsConnection;
boolean httpYes, httpsYes;
try {
if (httpSelection.equals("http://")) {
URL url = new URL(weburi);
urlConnection = (HttpURLConnection) url.openConnection();
inputStream = new BufferedInputStream((urlConnection.getInputStream()));
httpYes = True;
}
if (httpSelection.equals("https://")) {
URL url = new URL(weburi);
urlHttpsConnection = (HttpsURLConnection) url.openConnection();
urlHttpsConnection.setSSLSocketFactory(context.getSocketFactory());
inputStream = urlHttpsConnection.getInputStream();
https=True;
}
catch (Exception e) {
//Toast Message displays and settings intent re-starts
}
finally {
readFile(in);
if(httpYes){
urlConnection.disconnect();
httpYes = False;
}
if(httpsYes){
urlHttpsConnection.disconnect();
httpsYes = False;
}
}
}
EDIT:
To elaborate some more. I need to see if it returns a valid response from a website? So if the user selected http instead of https how can I check to see if http is the incorrect prefix/protocol?
How can I check if the website uses HTTPS or HTTP protocol? If the user then only puts in say www.google.com and I append https:// or http:// prefix to it, how do I know which one is the correct one to use?

You can Use android URLUtil to check whether url is HTTP or HTTPS:
public static boolean isHttpUrl (String url)
Returns True iff the url is an http: url.
public static boolean isHttpsUrl (String url)
Returns True iff the url is an https: url.
Edit:
public static boolean isValidUrl (String url)
Returns True iff the url is valid.

URLConnection result = url.openConnection();
if (result instanceof HttpsURLConnection) {
// https
}
else if (result instanceof HttpURLConnection) {
// http
}
else {
// null or something bad happened
}

Try this code:
mConnexion = (URLUtil.isHttpsUrl(mStringUrl)) ? (HttpsURLConnection) url.openConnection() : (HttpURLConnection) url.openConnection();

I have checked URLUtil class and checked that its contain so many methods for these kind of stuff, but i just extend the answer that you can simply do as below also :-
public static boolean isHttpOrHttpsUrl(String url) {
return url.matches("^(http|https|ftp)://.*$");
}

This can be checked by using Util.
isHttpUrl returns True iff the url is an http: url.
isHttpsUrl returns True iff the url is an https: url.

you can try this
boolean httpYes, httpsYes;
try {
URL url = new URL(weburi);
urlConnection = (HttpURLConnection) url.openConnection();
inputStream = new BufferedInputStream((urlConnection.getInputStream()));
httpYes = True;
}
catch (Exception e) {
//Toast Message displays and settings intent re-starts
URL url = new URL(weburi);
urlHttpsConnection = (HttpsURLConnection) url.openConnection();
urlHttpsConnection.setSSLSocketFactory(context.getSocketFactory());
inputStream = urlHttpsConnection.getInputStream();
https=True;
}

I think this works, at least it seems to be working, what do you guys think? I place this if statement just before the httpsYes = True and httpYes = True.
It seems that when the HTTPS protocol is selected it wants to redirect using response code 302, but for all other instances it connects with response code 200. I throw a new ConnectionException() error as that takes the user back to the settings screen to correct the URL error.
For the HTTPS protocol:
if (httpsURLConnection.getResponseCode() != 200) {
throw new ConnectException();
}
For the HTTP protocol:
if (urlConnection.getResponseCode() != 200) {
throw new ConnectException();
}
Comments? Should I use urlConnection.getResponseCode() > 199 && < 300? To cover all successful connects?

My recommendation is created function expression regular.
for example:
public void testUrl(Object urlHttp){
String url = "https://www.google.com";
if(url.matches("^(https?)://.*$")){
Object o = (HttpsURLConnection) urlHttp;
}else{
Object o = (HttpURLConnection) urlHttp;
}
}
Regards

I need an alternative option to HttpClient in Android to send data to PHP as it is no longer supported

Currently I'm using HttpClient, HttpPost to send data to my PHP server from an Android app but all those methods were deprecated in API 22 and removed in API 23, so what are the alternative options to it?
I searched everywhere but I didn't find anything.

I've also encountered with this problem to solve that I've made my own class.
Which based on java.net, and supports up to android's API 24
please check it out:
HttpRequest.java
Using this class you can easily:
Send Http GET request
Send Http POST request
Send Http PUT request
Send Http DELETE
Send request without extra data params & check response HTTP status code
Add custom HTTP Headers to request (using varargs)
Add data params as String query to request
Add data params as HashMap {key=value}
Accept Response as String
Accept Response as JSONObject
Accept response as byte [] Array of bytes (useful for files)
and any combination of those - just with one single line of code)
Here are a few examples:
//Consider next request:
HttpRequest req=new HttpRequest("http://host:port/path");
Example 1:
//prepare Http Post request and send to "http://host:port/path" with data params name=Bubu and age=29, return true - if worked
req.prepare(HttpRequest.Method.POST).withData("name=Bubu&age=29").send();
Example 2:
// prepare http get request, send to "http://host:port/path" and read server's response as String
req.prepare().sendAndReadString();
Example 3:
// prepare Http Post request and send to "http://host:port/path" with data params name=Bubu and age=29 and read server's response as JSONObject
HashMap<String, String>params=new HashMap<>();
params.put("name", "Groot");
params.put("age", "29");
req.prepare(HttpRequest.Method.POST).withData(params).sendAndReadJSON();
Example 4:
//send Http Post request to "http://url.com/b.c" in background using AsyncTask
new AsyncTask<Void, Void, String>(){
protected String doInBackground(Void[] params) {
String response="";
try {
response=new HttpRequest("http://url.com/b.c").prepare(HttpRequest.Method.POST).sendAndReadString();
} catch (Exception e) {
response=e.getMessage();
}
return response;
}
protected void onPostExecute(String result) {
//do something with response
}
}.execute();
Example 5:
//Send Http PUT request to: "http://some.url" with request header:
String json="{\"name\":\"Deadpool\",\"age\":40}";//JSON that we need to send
String url="http://some.url";//URL address where we need to send it
HttpRequest req=new HttpRequest(url);//HttpRequest to url: "http://some.url"
req.withHeaders("Content-Type: application/json");//add request header: "Content-Type: application/json"
req.prepare(HttpRequest.Method.PUT);//Set HttpRequest method as PUT
req.withData(json);//Add json data to request body
JSONObject res=req.sendAndReadJSON();//Accept response as JSONObject
Example 6:
//Equivalent to previous example, but in a shorter way (using methods chaining):
String json="{\"name\":\"Deadpool\",\"age\":40}";//JSON that we need to send
String url="http://some.url";//URL address where we need to send it
//Shortcut for example 5 complex request sending & reading response in one (chained) line
JSONObject res=new HttpRequest(url).withHeaders("Content-Type: application/json").prepare(HttpRequest.Method.PUT).withData(json).sendAndReadJSON();
Example 7:
//Downloading file
byte [] file = new HttpRequest("http://some.file.url").prepare().sendAndReadBytes();
FileOutputStream fos = new FileOutputStream("smile.png");
fos.write(file);
fos.close();

The HttpClient was deprecated and now removed:
org.apache.http.client.HttpClient:
This interface was deprecated in API level 22.
Please use openConnection() instead. Please visit this webpage for further details.
means that you should switch to java.net.URL.openConnection().
See also the new HttpURLConnection documentation.
Here's how you could do it:
URL url = new URL("http://some-server");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("POST");
// read the response
System.out.println("Response Code: " + conn.getResponseCode());
InputStream in = new BufferedInputStream(conn.getInputStream());
String response = org.apache.commons.io.IOUtils.toString(in, "UTF-8");
System.out.println(response);
IOUtils documentation: Apache Commons IO
IOUtils Maven dependency: http://search.maven.org/#artifactdetails|org.apache.commons|commons-io|1.3.2|jar

The following code is in an AsyncTask:
In my background process:
String POST_PARAMS = "param1=" + params[0] + "&param2=" + params[1];
URL obj = null;
HttpURLConnection con = null;
try {
obj = new URL(Config.YOUR_SERVER_URL);
con = (HttpURLConnection) obj.openConnection();
con.setRequestMethod("POST");
// For POST only - BEGIN
con.setDoOutput(true);
OutputStream os = con.getOutputStream();
os.write(POST_PARAMS.getBytes());
os.flush();
os.close();
// For POST only - END
int responseCode = con.getResponseCode();
Log.i(TAG, "POST Response Code :: " + responseCode);
if (responseCode == HttpURLConnection.HTTP_OK) { //success
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine);
}
in.close();
// print result
Log.i(TAG, response.toString());
} else {
Log.i(TAG, "POST request did not work.");
}
} catch (IOException e) {
e.printStackTrace();
}
Reference:
http://www.journaldev.com/7148/java-httpurlconnection-example-to-send-http-getpost-requests

This is the solution that I have applied to the problem that httpclient deprecated in this version of android 22`
public static final String USER_AGENT = "Mozilla/5.0";
public static String sendPost(String _url,Map<String,String> parameter) {
StringBuilder params=new StringBuilder("");
String result="";
try {
for(String s:parameter.keySet()){
params.append("&"+s+"=");
params.append(URLEncoder.encode(parameter.get(s),"UTF-8"));
}
String url =_url;
URL obj = new URL(_url);
HttpsURLConnection con = (HttpsURLConnection) obj.openConnection();
con.setRequestMethod("POST");
con.setRequestProperty("User-Agent", USER_AGENT);
con.setRequestProperty("Accept-Language", "UTF-8");
con.setDoOutput(true);
OutputStreamWriter outputStreamWriter = new OutputStreamWriter(con.getOutputStream());
outputStreamWriter.write(params.toString());
outputStreamWriter.flush();
int responseCode = con.getResponseCode();
System.out.println("\nSending 'POST' request to URL : " + url);
System.out.println("Post parameters : " + params);
System.out.println("Response Code : " + responseCode);
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
String inputLine;
StringBuffer response = new StringBuffer();
while ((inputLine = in.readLine()) != null) {
response.append(inputLine + "\n");
}
in.close();
result = response.toString();
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (ProtocolException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}catch (Exception e) {
e.printStackTrace();
}finally {
return result;
}
}

You are free to continue using HttpClient. Google deprecated only their own version of Apache's components. You can install fresh, powerful and non deprecated version of Apache's HttpClient like I described in this post: https://stackoverflow.com/a/37623038/1727132

if targeted for API 22 and older, then should add the following line into build.gradle
dependencies {
compile group: 'org.apache.httpcomponents' , name: 'httpclient-android' , version: '4.3.5.1'
}
if targeted for API 23 and later, then should add the following line into build.gradle
dependencies {
compile group: 'cz.msebera.android' , name: 'httpclient', version: '4.4.1.1'
}
If still want to use httpclient library, in Android Marshmallow (sdk 23), you can add:
useLibrary 'org.apache.http.legacy'
to build.gradle in the android {} section as a workaround. This seems to be necessary for some of Google's own gms libraries!

Which client is best?
Apache HTTP client has fewer bugs on Eclair and Froyo. It is the best
choice for these releases.
For Gingerbread and better, HttpURLConnection is the best choice. Its
simple API and small size makes it great fit for Android...
Reference here for more info (Android developers blog)

You can use my easy to use custom class.
Just create an object of the abstract class(Anonymous) and define onsuccess() and onfail() method.
https://github.com/creativo123/POSTConnection

i had similar issues in using HttpClent and HttpPost method since i didn't wanted change my code so i found alternate option in build.gradle(module) file by removing 'rc3' from buildToolsVersion "23.0.1 rc3" and it worked for me. Hope that Helps.

What's the fastest way to get a URLs status code using HttpClient

What is the fastest method of getting a URLs status using HttpClient? I don't want to download the page/file, I just want to know if the page/file exists?(If it's a redirect I want it to follow the redirect)

Here is how I get status code from HttpClient, which I like very much:
public boolean exists(){
CloseableHttpResponse response = null;
try {
CloseableHttpClient client = HttpClients.createDefault();
HttpHead headReq = new HttpHead(this.uri);
response = client.execute(headReq);
StatusLine sl = response.getStatusLine();
switch (sl.getStatusCode()) {
case 404: return false;
default: return true;
}
} catch (Exception e) {
log.error("Error in HttpGroovySourse : "+e.getMessage(), e );
} finally {
try {
response.close();
} catch (Exception e) {
log.error("Error in HttpGroovySourse : "+e.getMessage(), e );
}
}
return false;
}

Use the HEAD call. It's basically a GET call where the server does not return a body. Example from their documentation:
HeadMethod head = new HeadMethod("http://jakarta.apache.org");
// execute the method and handle any error responses.
...
// Retrieve all the headers.
Header[] headers = head.getResponseHeaders();
// Retrieve just the last modified header value.
String lastModified = head.getResponseHeader("last-modified").getValue();

You can get this Info with java.net.HttpURLConnection:
URL url = new URL("http://stackoverflow.com/");
URLConnection urlConnection = url.openConnection();
if (urlConnection instanceof HttpURLConnection) {
int responseCode = ((HttpURLConnection) urlConnection).getResponseCode();
switch (responseCode) {
case HttpURLConnection.HTTP_OK:
// HTTP Status-Code 302: Temporary Redirect.
break;
case HttpURLConnection.HTTP_MOVED_TEMP:
// HTTP Status-Code 302: Temporary Redirect.
break;
case HttpURLConnection.HTTP_NOT_FOUND:
// HTTP Status-Code 404: Not Found.
break;
}
}

You can use:
HeadMethod head = new HeadMethod("http://www.myfootestsite.com");
head.setFollowRedirects(true);
// Header stuff
Header[] headers = head.getResponseHeaders();
Do make sure that your web server supports the HEAD command.
See Section 9.4 in the HTTP 1.1 Spec