We Keep Coding

How to Perform MalformedURLException / 404 Not Found

How do I make my program return MalformedUrlException and not just a generic Exception?
I am making a simple function that reads a URL a user enters in the console and it returns the content from the URL. I need it to check if the URL is a valid URL or if it's not a working URL.
Example urls:
http://google.com/not-found.html
http:/google.com
I created two catch exceptions but it seems like the overall exception is always returned instead of MalformedUrlException.
public static String getUrlContents(String theUrl) {
String content = "";
try {
URL url = new URL(theUrl);
//Create a url connection object
URLConnection urlConnection = url.openConnection();
//wrap the url connection a buffered reader
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
String line;
while((line = bufferedReader.readLine()) != null) {
content += line + "\n";
}
bufferedReader.close();
} catch (MalformedURLException e) {
System.out.println("The following url is invalid'" + theUrl + "'");
//logging error should go here
} catch (Exception e) {
System.out.println("Something went wrong, try agian");
}
return content;
}

First, java.net.MalformedURLException is not the case for a "not found" resource:
public class MalformedURLException extends IOException
Thrown to indicate that a malformed URL has occurred. Either no legal
protocol could be found in a specification string or the string could
not be parsed.
I understand that you want to catch a situation when the URL results in a not found return code (404). To do this you need to examine the HTTP response code.
The easiest way is to use java.net.HttpURLConnection:
https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/net/HttpURLConnection.html
public abstract class HttpURLConnection extends URLConnection
A URLConnection with support for HTTP-specific features. See the spec
for details.
Each HttpURLConnection instance is used to make a single request but
the underlying network connection to the HTTP server may be
transparently shared by other instances. Calling the close() methods
on the InputStream or OutputStream of an HttpURLConnection after a
request may free network resources associated with this instance but
has no effect on any shared persistent connection. Calling the
disconnect() method may close the underlying socket if a persistent
connection is otherwise idle at that time.
You can check the response code by calling getResponseCode(). If the result is less than 400, you got a valid response, otherwise there was a client error (4xx) or a server error (5xx).
Something like this:
public static String getUrlContents(String theUrl) {
String content = "";
try {
URL url = new URL(theUrl);
//Create a url connection object
URLConnection urlConnection = url.openConnection();
if (urlConnection instanceof HttpURLConnection) {
HttpURLConnection conn = (HttpURLConnection) urlConnection;
if (conn.getResponseCode() < 400) {
// read contents
} else {
System.out.println(conn.getResponseMessage());
// treat the error as you like
}
} else {
// not a HTTP connection, treat as you like
}
} catch (MalformedURLException e) {
System.out.println("The following url is invalid'" + theUrl + "'");
//logging error should go here
} catch (Exception e) {
System.out.println("Something went wrong, try agian");
}
return content;
}
I have not checked the code, but I think you can get the overall idea.

How to close a persistent HTTP Connection without reading

I have an URLConnection which I want to cancel depending on the response code without reading any data. I closely followed the android training to build the following minimal example which
floods the server with requests since no connection is ever released back to the handle pool for reuse
private String downloadUrl(String myurl) throws IOException {
InputStream is = null;
try {
URL url = new URL(myurl);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setReadTimeout(10000 /* milliseconds */);
conn.setConnectTimeout(15000 /* milliseconds */);
conn.setRequestMethod("GET");
conn.setDoInput(true);
// Starts the query
conn.connect();
int response = conn.getResponseCode();
Log.d(TAG, "The response code is: " + response);
is = conn.getInputStream();
// Do not read anything //String contentAsString = readIt(is, len);
String contentAsString = "notReadingAnything";
return contentAsString;
} finally {
if (is != null) {
is.close();
}
}
}
private class DownloadWebpageTask extends AsyncTask<String, Void, String> {
#Override
protected String doInBackground(String... urls) {
try {
String result = new String();
for (int i=0; i<100; i++) {
result += downloadUrl(urls[0]);
}
return result;
} catch (IOException e) {
return "Unable to retrieve web page. URL may be invalid.";
}
}
#Override
protected void onPostExecute(String result) {
Log.d(TAG, "The response is: " + result);
}
}
Despite the docs explicitly stating
But if the response body is long and you are not interested in the rest of it after seeing the beginning, you can close the InputStream
the server quickly reaches its maximum number of connections (50) and goes to 99% workload if I don't read the stream but works fine if I do read it. What is my mistake?
EDIT: Failed solution attempts so far (thanks to #Blackbelt for most of them)
calling conn.disconnect() in the finally block
calling conn.disconnect() instead of is.close() in the finally block
Setting System.setProperty("http.keepAlive", "false"); before the first call
Setting conn.setRequestProperty("Connection", "Close"); before connecting
Setting "{enable_keep_alive", "no"} on the used backend server (Civetweb)

you should call disconnect() too. Accordingly to the documentation
Disconnect. Once the response body has been read, the
HttpURLConnection should be closed by calling disconnect().
Disconnecting releases the resources held by a connection so they may
be closed or reused.
InputStream is = null;
HttpURLConnection conn = null;
try {
URL url = new URL(myurl);
conn = (HttpURLConnection) url.openConnection();
} finally {
if (is != null) {
is.close();
}
if (conn != null) {
conn.disconnect();
}
}
if you still are experiencing issues, is also possible that the bug is backend side

HTTP Post request read the response [duplicate]

In Java, this code throws an exception when the HTTP result is 404 range:
URL url = new URL("http://stackoverflow.com/asdf404notfound");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.getInputStream(); // throws!
In my case, I happen to know that the content is 404, but I'd still like to read the body of the response anyway.
(In my actual case the response code is 403, but the body of the response explains the reason for rejection, and I'd like to display that to the user.)
How can I access the response body?

Here is the bug report (close, will not fix, not a bug).
Their advice there is to code like this:
HttpURLConnection httpConn = (HttpURLConnection)_urlConnection;
InputStream _is;
if (httpConn.getResponseCode() < HttpURLConnection.HTTP_BAD_REQUEST) {
_is = httpConn.getInputStream();
} else {
/* error from server */
_is = httpConn.getErrorStream();
}

It's the same problem I was having:
HttpUrlConnection returns FileNotFoundException if you try to read the getInputStream() from the connection.
You should instead use getErrorStream() when the status code is higher than 400.
More than this, please be careful since it's not only 200 to be the success status code, even 201, 204, etc. are often used as success statuses.
Here is an example of how I went to manage it
... connection code code code ...
// Get the response code
int statusCode = connection.getResponseCode();
InputStream is = null;
if (statusCode >= 200 && statusCode < 400) {
// Create an InputStream in order to extract the response object
is = connection.getInputStream();
}
else {
is = connection.getErrorStream();
}
... callback/response to your handler....
In this way, you'll be able to get the needed response in both success and error cases.
Hope this helps!

In .Net you have the Response property of the WebException that gives access to the stream ON an exception. So i guess this is a good way for Java,...
private InputStream dispatch(HttpURLConnection http) throws Exception {
try {
return http.getInputStream();
} catch(Exception ex) {
return http.getErrorStream();
}
}
Or an implementation i used. (Might need changes for encoding or other things. Works in current environment.)
private String dispatch(HttpURLConnection http) throws Exception {
try {
return readStream(http.getInputStream());
} catch(Exception ex) {
readAndThrowError(http);
return null; // <- never gets here, previous statement throws an error
}
}
private void readAndThrowError(HttpURLConnection http) throws Exception {
if (http.getContentLengthLong() > 0 && http.getContentType().contains("application/json")) {
String json = this.readStream(http.getErrorStream());
Object oson = this.mapper.readValue(json, Object.class);
json = this.mapper.writer().withDefaultPrettyPrinter().writeValueAsString(oson);
throw new IllegalStateException(http.getResponseCode() + " " + http.getResponseMessage() + "\n" + json);
} else {
throw new IllegalStateException(http.getResponseCode() + " " + http.getResponseMessage());
}
}
private String readStream(InputStream stream) throws Exception {
StringBuilder builder = new StringBuilder();
try (BufferedReader in = new BufferedReader(new InputStreamReader(stream))) {
String line;
while ((line = in.readLine()) != null) {
builder.append(line); // + "\r\n"(no need, json has no line breaks!)
}
in.close();
}
System.out.println("JSON: " + builder.toString());
return builder.toString();
}

I know that this doesn't answer the question directly, but instead of using the HTTP connection library provided by Sun, you might want to take a look at Commons HttpClient, which (in my opinion) has a far easier API to work with.

First check the response code and then use HttpURLConnection.getErrorStream()

InputStream is = null;
if (httpConn.getResponseCode() !=200) {
is = httpConn.getErrorStream();
} else {
/* error from server */
is = httpConn.getInputStream();
}

My running code.
HttpURLConnection httpConn = (HttpURLConnection) urlConn;
if (httpConn.getResponseCode() < HttpURLConnection.HTTP_BAD_REQUEST) {
in = new InputStreamReader(urlConn.getInputStream());
BufferedReader bufferedReader = new BufferedReader(in);
if (bufferedReader != null) {
int cp;
while ((cp = bufferedReader.read()) != -1) {
sb.append((char) cp);
}
bufferedReader.close();
}
in.close();
} else {
/* error from server */
in = new InputStreamReader(httpConn.getErrorStream());
BufferedReader bufferedReader = new BufferedReader(in);
if (bufferedReader != null) {
int cp;
while ((cp = bufferedReader.read()) != -1) {
sb.append((char) cp);
}
bufferedReader.close();
}
in.close();
}
System.out.println("sb="+sb);

How to read 404 response body in java:
Use Apache library - https://hc.apache.org/httpcomponents-client-4.5.x/httpclient/apidocs/
or
Java 11 - https://docs.oracle.com/en/java/javase/11/docs/api/java.net.http/java/net/http/HttpClient.html
Snippet given below uses Apache:
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.util.EntityUtils;
CloseableHttpClient client = HttpClients.createDefault();
CloseableHttpResponse resp = client.execute(new HttpGet(domainName + "/blablablabla.html"));
String response = EntityUtils.toString(resp.getEntity());

Java, HttpURLConnection handle CLOSE_WAIT connections

I am using java HttpURLConnection for REST calls. After working(testing of rest calls using ant targets) with all the rest api's I see there are many socket connections that are in 'CLOSE_WAIT' state.
I tried by calling the close() methods on the InputStream or OutputStream of an HttpURLConnection, but still the connections are in CLOSE_WAIT state only.
One other observation is even with con.disconnect() method also the connection is not being closed.
Please help me regarding this issue as CLOSE_WAIT connections indicate an error in the software.
Below is the code for get call. Other POST/PUT/DELETE calls are also like 'get'
public void get(String url, Header[] headers,
NameValuePair[] data, ResponseHandler handler) throws HttpException {
HttpURLConnection con = null;
try
{
String query = null;
if (data != null) {
query = getQueryString(data);
}
URL obj;
if(query == null || query == "")
obj = new URL(url);
else
obj = new URL(url+"?"+query);
con = (HttpURLConnection) obj.openConnection();
setDoAuthentication(con);
con.setRequestMethod("GET");
con.setDoOutput(true);
if (headers != null) {
for (int i = 0; i < headers.length; i++) {
con.setRequestProperty(headers[i].getName(), headers[i].getValue());
}
}
con.setRequestProperty("Accept-encoding", "gzip");
con.setRequestProperty("Authorization", this.auth);
int responseCode = con.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_OK) {
if (handler!=null) handler.handleResponse(con); // in handleResponse(con) method, I am closing the con input stream
} else if (responseCode == HttpURLConnection.HTTP_UNAUTHORIZED) {
throw new HttpException(new UnauthorizedException());
} else if (!is2xx(responseCode)) {
ErrorHandler errHandler = new ErrorHandler();
errHandler.handleResponse(con);
throw errHandler.getResult();
}
}
catch (MalformedURLException e) {
throw new HttpException(e);
}
catch (IOException e) {
handleSessionConnectionError(e, url);
}
finally {
con.disconnect();
}
}
Thanks
Mohan G

Calling con.disconnect() may cause the issue. Check this out this link link.
It will create sockets on and on, and if the connections are multiple, it will overpass the max limit of files open per process, e.g. in linux is usually 1024.
Instead of using disconnect, always close the input stream, the values defined in http.maxConnection in java system properties will do the work of closing idle connections when the limit is reach.
Check also javadoc of HttpUrlConnection in class and the method disconnect.

Observe/capture 302 redirects in java

I have a web application under test. Using Fiddler/Httpfox, I can see that on logging in to the web app, there are TWO 302 HTTP redirects before a 200 OK is response is received. Is it possible to observe the two redirects using Java code?
This is what I coded:
public class HttpReq {
HttpURLConnection con = null;
StringBuilder str = new StringBuilder();
BufferedReader br = null;
URL address = null;
String line = null;
HttpReq () {
try {
address = new URL("http://walhs24002v.us.oracle.com/t1mockapp1/");
con = (HttpURLConnection)address.openConnection();
con.setRequestMethod("GET");
con.setReadTimeout(60000);
con.setConnectTimeout(60000);
con.setDoOutput(true);
con.setInstanceFollowRedirects(true);
con.connect();
InputStreamReader is = new InputStreamReader(con.getInputStream());
br = new BufferedReader(is);
while((line = br.readLine()) != null)
{
str.append(line + '\n');
}
//System.out.println(str);
System.out.println(con.getResponseCode());
System.out.println(con.getResponseMessage());
}
catch (MalformedURLException m)
{
m.printStackTrace();
}
catch (IOException i)
{
i.printStackTrace();
}
finally
{
br = null;
str = null;
con = null;
}
}
public static void main(String[] args) {
HttpReq http = new HttpReq();
}
}
The program gives the output:
200
OK
No surprises there. Is there a way to capture the two 302 redirects before the 200 ok is received?

With defaulthttpclient you most certainly can:
http://hc.apache.org/httpcomponents-client-ga/tutorial/html/httpagent.html
ClientPNames.HANDLE_REDIRECTS='http.protocol.handle-redirects':
defines whether redirects should be handled automatically. This
parameter expects a value of type java.lang.Boolean. If this parameter
is not set HttpClient will handle redirects automatically.