We Keep Coding

How to Perform MalformedURLException / 404 Not Found

How do I make my program return MalformedUrlException and not just a generic Exception?
I am making a simple function that reads a URL a user enters in the console and it returns the content from the URL. I need it to check if the URL is a valid URL or if it's not a working URL.
Example urls:
http://google.com/not-found.html
http:/google.com
I created two catch exceptions but it seems like the overall exception is always returned instead of MalformedUrlException.
public static String getUrlContents(String theUrl) {
String content = "";
try {
URL url = new URL(theUrl);
//Create a url connection object
URLConnection urlConnection = url.openConnection();
//wrap the url connection a buffered reader
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
String line;
while((line = bufferedReader.readLine()) != null) {
content += line + "\n";
}
bufferedReader.close();
} catch (MalformedURLException e) {
System.out.println("The following url is invalid'" + theUrl + "'");
//logging error should go here
} catch (Exception e) {
System.out.println("Something went wrong, try agian");
}
return content;
}

First, java.net.MalformedURLException is not the case for a "not found" resource:
public class MalformedURLException extends IOException
Thrown to indicate that a malformed URL has occurred. Either no legal
protocol could be found in a specification string or the string could
not be parsed.
I understand that you want to catch a situation when the URL results in a not found return code (404). To do this you need to examine the HTTP response code.
The easiest way is to use java.net.HttpURLConnection:
https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/net/HttpURLConnection.html
public abstract class HttpURLConnection extends URLConnection
A URLConnection with support for HTTP-specific features. See the spec
for details.
Each HttpURLConnection instance is used to make a single request but
the underlying network connection to the HTTP server may be
transparently shared by other instances. Calling the close() methods
on the InputStream or OutputStream of an HttpURLConnection after a
request may free network resources associated with this instance but
has no effect on any shared persistent connection. Calling the
disconnect() method may close the underlying socket if a persistent
connection is otherwise idle at that time.
You can check the response code by calling getResponseCode(). If the result is less than 400, you got a valid response, otherwise there was a client error (4xx) or a server error (5xx).
Something like this:
public static String getUrlContents(String theUrl) {
String content = "";
try {
URL url = new URL(theUrl);
//Create a url connection object
URLConnection urlConnection = url.openConnection();
if (urlConnection instanceof HttpURLConnection) {
HttpURLConnection conn = (HttpURLConnection) urlConnection;
if (conn.getResponseCode() < 400) {
// read contents
} else {
System.out.println(conn.getResponseMessage());
// treat the error as you like
}
} else {
// not a HTTP connection, treat as you like
}
} catch (MalformedURLException e) {
System.out.println("The following url is invalid'" + theUrl + "'");
//logging error should go here
} catch (Exception e) {
System.out.println("Something went wrong, try agian");
}
return content;
}
I have not checked the code, but I think you can get the overall idea.

Default Proxy Java

I'm trying to call Webservices API using JAVA but I have connection issue probably due to a proxy.
I already did it with C# WPF application, and to avoid connection issue I set following option on App.config file
<system.net>
<defaultProxy useDefaultCredentials="true"></defaultProxy>
I don't find a way to do the same setup on JAVA and my connection is rejected :
Do you know how can I setup the default proxy ?
JAVA code
/**
* Executes the request and returns its response.
*
* #return the request's response
* #throws IOException if the underlying {#link HttpsURLConnection} could
* not be set up or executed
*/
public String execute() throws IOException {
HttpsURLConnection connection = null;
try {
connection = (HttpsURLConnection) url.openConnection();
connection.setRequestMethod("POST");
// write POST data to request
// Exception is raised at this level
if (postData != null && !postData.toString().isEmpty()) {
connection.setDoOutput(true);
try (OutputStreamWriter out = new OutputStreamWriter(connection.getOutputStream()))
{
out.write(postData.toString());
}
catch(Exception e){
System.out.println(e.getMessage());
}
}
// execute request and read response
try (BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()))) {
StringBuilder response = new StringBuilder();
String line;
while ((line = in.readLine()) != null) {
response.append(line);
}
return response.toString();
}
}
finally {
connection.disconnect();
}
}
Many thanks for your help

How to close a persistent HTTP Connection without reading

I have an URLConnection which I want to cancel depending on the response code without reading any data. I closely followed the android training to build the following minimal example which
floods the server with requests since no connection is ever released back to the handle pool for reuse
private String downloadUrl(String myurl) throws IOException {
InputStream is = null;
try {
URL url = new URL(myurl);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setReadTimeout(10000 /* milliseconds */);
conn.setConnectTimeout(15000 /* milliseconds */);
conn.setRequestMethod("GET");
conn.setDoInput(true);
// Starts the query
conn.connect();
int response = conn.getResponseCode();
Log.d(TAG, "The response code is: " + response);
is = conn.getInputStream();
// Do not read anything //String contentAsString = readIt(is, len);
String contentAsString = "notReadingAnything";
return contentAsString;
} finally {
if (is != null) {
is.close();
}
}
}
private class DownloadWebpageTask extends AsyncTask<String, Void, String> {
#Override
protected String doInBackground(String... urls) {
try {
String result = new String();
for (int i=0; i<100; i++) {
result += downloadUrl(urls[0]);
}
return result;
} catch (IOException e) {
return "Unable to retrieve web page. URL may be invalid.";
}
}
#Override
protected void onPostExecute(String result) {
Log.d(TAG, "The response is: " + result);
}
}
Despite the docs explicitly stating
But if the response body is long and you are not interested in the rest of it after seeing the beginning, you can close the InputStream
the server quickly reaches its maximum number of connections (50) and goes to 99% workload if I don't read the stream but works fine if I do read it. What is my mistake?
EDIT: Failed solution attempts so far (thanks to #Blackbelt for most of them)
calling conn.disconnect() in the finally block
calling conn.disconnect() instead of is.close() in the finally block
Setting System.setProperty("http.keepAlive", "false"); before the first call
Setting conn.setRequestProperty("Connection", "Close"); before connecting
Setting "{enable_keep_alive", "no"} on the used backend server (Civetweb)

you should call disconnect() too. Accordingly to the documentation
Disconnect. Once the response body has been read, the
HttpURLConnection should be closed by calling disconnect().
Disconnecting releases the resources held by a connection so they may
be closed or reused.
InputStream is = null;
HttpURLConnection conn = null;
try {
URL url = new URL(myurl);
conn = (HttpURLConnection) url.openConnection();
} finally {
if (is != null) {
is.close();
}
if (conn != null) {
conn.disconnect();
}
}
if you still are experiencing issues, is also possible that the bug is backend side

java.net.ProtocolException: Connection already established ( Not Found Proper Solution in existing Post )

In one of my android app, i am getting error "java.net.ProtocolException: Connection already established" during call the API. (I refered "Connection already established" exception in HttpsURLConnection for solution but not useful)
Below is my code to call API. Code is working for all the API except the one. That API is the same way developed as the Others and still below code throwing error for that API.
HttpURLConnection con = null;
BufferedReader in = null;
StringBuilder sb = new StringBuilder();
try {
URL uri = null;
uri = new URL(url);
LogUtil.v("~~URL - "+url);
con = (HttpURLConnection) uri.openConnection();
System.setProperty("http.keepAlive", "false");
con.setRequestMethod(requestType); //type: POST, PUT, DELETE, GET
con.setDoOutput(true);
// con.setDoInput(true);
con.setConnectTimeout(TIMEOUT); //20 secs
con.setReadTimeout(TIMEOUT); //20 secs
con.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
String authorizationString = "Basic " + Base64.encodeToString((myAuthString).getBytes(), Base64.DEFAULT);
con.setRequestProperty("Authorization", authorizationString);
if( body != null){
LogUtil.v("~~BODY - "+body);
DataOutputStream out = new DataOutputStream(con.getOutputStream());
out.writeBytes(body);
out.flush();
out.close();
}
con.connect();
InputStream inst = con.getInputStream();
in = new BufferedReader(new InputStreamReader(inst));
String temp = null;
while((temp = in.readLine()) != null){
sb.append(temp).append(" ");
}
} catch (IOException e) {
e.printStackTrace();
}catch (Exception e) {
e.printStackTrace();
}finally{
if(in!=null){
try {
in.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if(cb!=null){
if (sb.toString().length() >0) {
LogUtil.v("~~ RESPONSE - "+ sb.toString());
cb.onResponse(sb.toString(), actionCode);
}else{
LogUtil.v("~~ RESPONSE - null");
cb.onResponse(null, actionCode);
}
}
if(con!=null)
con.disconnect();
}
Please don't suggest to use DefaultHttpClient here.
I want to solve the problem using above BECAUSE IT IS WORKING FOR ALL OTHER APIS.
FYI: API CALL IS ALREADY CHECKED USING BROWSER AND IT IS RETURNING RETURNING DATA. IT IS JUST THROWING ERROR WHEN CALLING FROM ANDROID.
~~~~~~ EDITED ~~~~~~~~~ (SERVER SIDE APIS ARE DEVELOPED IN PHP)
Let me simplify my problem explanation:
function API1Call(){
// callAPI(param...)
}
function API2Call(){
// callAPI(param...)
}
callAPI(param..){
// above API call code goes here
}
if you see the function callAPI(param...), it executing whenever required to call API.
***> Now in my proj, i m calling many apis written on different activites.
Only for One API call, above callAPI(param...) throwing exception that
i mentioned above.***
~~~~~ MY LOGCAT ~~~~~~~
08-31 14:22:45.309: W/System.err(27364): java.net.ProtocolException: Connection already established
08-31 14:22:45.314: W/System.err(27364): at java.net.HttpURLConnection.setRequestMethod(HttpURLConnection.java:666)
08-31 14:22:45.324: W/System.err(27364): at libcore.net.http.HttpsURLConnectionImpl.setRequestMethod(HttpsURLConnectionImpl.java:144)
08-31 14:22:45.329: W/System.err(27364): at
com.indapoint.ewe.api.EWeAPI$1.run(EWeAPI.java:245)
08-31 14:22:45.334: W/System.err(27364): at java.lang.Thread.run(Thread.java:856)

Don't use con.connect() after the getOutputStream() call, as the second will have called connect() anyways.

URLConnection is not allowing me to access data on Http errors (404,500,etc)

I am making a crawler, and need to get the data from the stream regardless if it is a 200 or not. CURL is doing it, as well as any standard browser.
The following will not actually get the content of the request, even though there is some, an exception is thrown with the http error status code. I want the output regardless, is there a way? I prefer to use this library as it will actually do persistent connections, which is perfect for the type of crawling I am doing.
package test;
import java.net.*;
import java.io.*;
public class Test {
public static void main(String[] args) {
try {
URL url = new URL("http://github.com/XXXXXXXXXXXXXX");
URLConnection connection = url.openConnection();
DataInputStream inStream = new DataInputStream(connection.getInputStream());
String inputLine;
while ((inputLine = inStream.readLine()) != null) {
System.out.println(inputLine);
}
inStream.close();
} catch (MalformedURLException me) {
System.err.println("MalformedURLException: " + me);
} catch (IOException ioe) {
System.err.println("IOException: " + ioe);
}
}
}
Worked, thanks: Here is what I came up with - just as a rough proof of concept:
import java.net.*;
import java.io.*;
public class Test {
public static void main(String[] args) {
//InputStream error = ((HttpURLConnection) connection).getErrorStream();
URL url = null;
URLConnection connection = null;
String inputLine = "";
try {
url = new URL("http://verelo.com/asdfrwdfgdg");
connection = url.openConnection();
DataInputStream inStream = new DataInputStream(connection.getInputStream());
while ((inputLine = inStream.readLine()) != null) {
System.out.println(inputLine);
}
inStream.close();
} catch (MalformedURLException me) {
System.err.println("MalformedURLException: " + me);
} catch (IOException ioe) {
System.err.println("IOException: " + ioe);
InputStream error = ((HttpURLConnection) connection).getErrorStream();
try {
int data = error.read();
while (data != -1) {
//do something with data...
//System.out.println(data);
inputLine = inputLine + (char)data;
data = error.read();
//inputLine = inputLine + (char)data;
}
error.close();
} catch (Exception ex) {
try {
if (error != null) {
error.close();
}
} catch (Exception e) {
}
}
}
System.out.println(inputLine);
}
}

Simple:
URLConnection connection = url.openConnection();
InputStream is = connection.getInputStream();
if (connection instanceof HttpURLConnection) {
HttpURLConnection httpConn = (HttpURLConnection) connection;
int statusCode = httpConn.getResponseCode();
if (statusCode != 200 /* or statusCode >= 200 && statusCode < 300 */) {
is = httpConn.getErrorStream();
}
}
You can refer to Javadoc for explanation. The best way I would handle this is as follows:
URLConnection connection = url.openConnection();
InputStream is = null;
try {
is = connection.getInputStream();
} catch (IOException ioe) {
if (connection instanceof HttpURLConnection) {
HttpURLConnection httpConn = (HttpURLConnection) connection;
int statusCode = httpConn.getResponseCode();
if (statusCode != 200) {
is = httpConn.getErrorStream();
}
}
}

You need to do the following after calling openConnection.
Cast the URLConnection to HttpURLConnection
Call getResponseCode
If the response is a success, use getInputStream, otherwise use getErrorStream
(The test for success should be 200 <= code < 300 because there are valid HTTP success codes apart from than 200.)
I am making a crawler, and need to get the data from the stream regardless if it is a 200 or not.
Just be aware that it if the code is a 4xx or 5xx, then the "data" is likely to be an error page of some kind.
The final point that should be made is that you should always respect the "robots.txt" file ... and read the Terms of Service before crawling / scraping the content of a site whose owners might care. Simply blatting off GET requests is likely to annoy site owners ... unless you've already come to some sort of "arrangement" with them.