How do I make my program return MalformedUrlException and not just a generic Exception?
I am making a simple function that reads a URL a user enters in the console and it returns the content from the URL. I need it to check if the URL is a valid URL or if it's not a working URL.
Example urls:
http://google.com/not-found.html
http:/google.com
I created two catch exceptions but it seems like the overall exception is always returned instead of MalformedUrlException.
public static String getUrlContents(String theUrl) {
String content = "";
try {
URL url = new URL(theUrl);
//Create a url connection object
URLConnection urlConnection = url.openConnection();
//wrap the url connection a buffered reader
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
String line;
while((line = bufferedReader.readLine()) != null) {
content += line + "\n";
}
bufferedReader.close();
} catch (MalformedURLException e) {
System.out.println("The following url is invalid'" + theUrl + "'");
//logging error should go here
} catch (Exception e) {
System.out.println("Something went wrong, try agian");
}
return content;
}
First, java.net.MalformedURLException is not the case for a "not found" resource:
public class MalformedURLException extends IOException
Thrown to indicate that a malformed URL has occurred. Either no legal
protocol could be found in a specification string or the string could
not be parsed.
I understand that you want to catch a situation when the URL results in a not found return code (404). To do this you need to examine the HTTP response code.
The easiest way is to use java.net.HttpURLConnection:
https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/net/HttpURLConnection.html
public abstract class HttpURLConnection extends URLConnection
A URLConnection with support for HTTP-specific features. See the spec
for details.
Each HttpURLConnection instance is used to make a single request but
the underlying network connection to the HTTP server may be
transparently shared by other instances. Calling the close() methods
on the InputStream or OutputStream of an HttpURLConnection after a
request may free network resources associated with this instance but
has no effect on any shared persistent connection. Calling the
disconnect() method may close the underlying socket if a persistent
connection is otherwise idle at that time.
You can check the response code by calling getResponseCode(). If the result is less than 400, you got a valid response, otherwise there was a client error (4xx) or a server error (5xx).
Something like this:
public static String getUrlContents(String theUrl) {
String content = "";
try {
URL url = new URL(theUrl);
//Create a url connection object
URLConnection urlConnection = url.openConnection();
if (urlConnection instanceof HttpURLConnection) {
HttpURLConnection conn = (HttpURLConnection) urlConnection;
if (conn.getResponseCode() < 400) {
// read contents
} else {
System.out.println(conn.getResponseMessage());
// treat the error as you like
}
} else {
// not a HTTP connection, treat as you like
}
} catch (MalformedURLException e) {
System.out.println("The following url is invalid'" + theUrl + "'");
//logging error should go here
} catch (Exception e) {
System.out.println("Something went wrong, try agian");
}
return content;
}
I have not checked the code, but I think you can get the overall idea.
Related
Unsure how many of you are familiar with Bukkit / Mojang API, however, this is what I'm working with today. Though I don't believe this will suit any relevance to the problem as it seems to be a native issue.
I connect to the authentication service following Mojang's authentication wiki:
http://wiki.vg/Authentication#Authenticate
In my code, the 'MinecraftAccount' is just an object representing the payload in the above link, which I serialize with GSON.
I then want to grab the input response sent back, with properties you can also find in the link above. I use another object to hold the json values and deserialize it with Json (AuthResponse).
It seems I am doing everything right, but I get a 403 Error. I'm unsure why because after several attempts of getting this right, it worked, and then spontaneously began spitting out the 403 again. Hopefully you guys can see the issue here.
public String getAuthToken(MinecraftAccount minecraftAccount) {
try {
HttpURLConnection httpURLConnection = Webs.getConnection("https://authserver.mojang.com/authenticate");
httpURLConnection.setRequestMethod("POST");
httpURLConnection.setDoOutput(true);
httpURLConnection.setDoInput(true);
httpURLConnection.setInstanceFollowRedirects(false);
httpURLConnection.setRequestProperty("Accept-Charset", "UTF-8");
httpURLConnection.setRequestProperty("Content-Type", "application/json");
httpURLConnection.setUseCaches(false);
httpURLConnection.addRequestProperty("User-Agent", "Mozilla/4.0");
IOUtils.write(GsonProvider.standard().toJson(minecraftAccount), httpURLConnection.getOutputStream(), StandardCharsets.UTF_8);
return GsonProvider.standard().fromJson(IOUtils.toString(httpURLConnection.getInputStream(), StandardCharsets.UTF_8), AuthResponse.class).getAccessToken();
} catch (IOException exception) {
exception.printStackTrace();
}
return null;
}
public void uploadSkin(UUID uuid, String url, boolean normal) {
try {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
ImageIO.write(ImageIO.read(new URL(url)), "png", baos);
String authenticityToken = this.getAuthToken(this.minecraftAccount);
try {
System.out.println("Token: " + authenticityToken);
if (authenticityToken == null) {
return;
}
HttpURLConnection httpURLConnection = Webs.getConnection(String.format(UPLOAD_SKIN_URL, UUIDTypeAdapter.fromUUID(uuid)));
httpURLConnection.setRequestMethod("POST");
httpURLConnection.setDoOutput(true);
httpURLConnection.addRequestProperty("Authorization", "Bearer " + authenticityToken);
try (BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(httpURLConnection.getOutputStream(), StandardCharsets.UTF_8))) {
writer.write("model=");
if (!normal) {
writer.write("slim");
}
writer.write("&url=" + UrlEscapers.urlPathSegmentEscaper().escape(url));
System.out.println("status: " + httpURLConnection.getResponseMessage() + "(" + httpURLConnection.getResponseCode() + ")");
} catch (Exception exception) {
exception.printStackTrace();
}
} catch (Exception exception) {
exception.printStackTrace();
}
} catch (IOException exception) {
exception.printStackTrace();
}
}
The Webs#getConnection is simply this (Using Proxy.NO_PROXY):
public static HttpURLConnection getConnection(String url, Proxy proxy) throws IOException {
HttpURLConnection httpConnection = (HttpURLConnection) new URL(url).openConnection(proxy);
httpConnection.setConnectTimeout(TIMEOUT);
httpConnection.setReadTimeout(2 * TIMEOUT);
httpConnection.setRequestProperty(HttpHeaders.CONTENT_TYPE, "application/json");
httpConnection.setRequestProperty(HttpHeaders.USER_AGENT, "Mozilla/5.0");
return httpConnection;
}
PS: Yes, I have looked at several other 403 threads, however, their solutions (such as setting the user agent as a fix), doesn't work.
Ultimately, there's no error when calling #uploadSkin, it just returns a 403 and doesn't actually upload the file to the site or anything. The method just outputs this:
Output
I have an URLConnection which I want to cancel depending on the response code without reading any data. I closely followed the android training to build the following minimal example which
floods the server with requests since no connection is ever released back to the handle pool for reuse
private String downloadUrl(String myurl) throws IOException {
InputStream is = null;
try {
URL url = new URL(myurl);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setReadTimeout(10000 /* milliseconds */);
conn.setConnectTimeout(15000 /* milliseconds */);
conn.setRequestMethod("GET");
conn.setDoInput(true);
// Starts the query
conn.connect();
int response = conn.getResponseCode();
Log.d(TAG, "The response code is: " + response);
is = conn.getInputStream();
// Do not read anything //String contentAsString = readIt(is, len);
String contentAsString = "notReadingAnything";
return contentAsString;
} finally {
if (is != null) {
is.close();
}
}
}
private class DownloadWebpageTask extends AsyncTask<String, Void, String> {
#Override
protected String doInBackground(String... urls) {
try {
String result = new String();
for (int i=0; i<100; i++) {
result += downloadUrl(urls[0]);
}
return result;
} catch (IOException e) {
return "Unable to retrieve web page. URL may be invalid.";
}
}
#Override
protected void onPostExecute(String result) {
Log.d(TAG, "The response is: " + result);
}
}
Despite the docs explicitly stating
But if the response body is long and you are not interested in the rest of it after seeing the beginning, you can close the InputStream
the server quickly reaches its maximum number of connections (50) and goes to 99% workload if I don't read the stream but works fine if I do read it. What is my mistake?
EDIT: Failed solution attempts so far (thanks to #Blackbelt for most of them)
calling conn.disconnect() in the finally block
calling conn.disconnect() instead of is.close() in the finally block
Setting System.setProperty("http.keepAlive", "false"); before the first call
Setting conn.setRequestProperty("Connection", "Close"); before connecting
Setting "{enable_keep_alive", "no"} on the used backend server (Civetweb)
you should call disconnect() too. Accordingly to the documentation
Disconnect. Once the response body has been read, the
HttpURLConnection should be closed by calling disconnect().
Disconnecting releases the resources held by a connection so they may
be closed or reused.
InputStream is = null;
HttpURLConnection conn = null;
try {
URL url = new URL(myurl);
conn = (HttpURLConnection) url.openConnection();
} finally {
if (is != null) {
is.close();
}
if (conn != null) {
conn.disconnect();
}
}
if you still are experiencing issues, is also possible that the bug is backend side
I am using java HttpURLConnection for REST calls. After working(testing of rest calls using ant targets) with all the rest api's I see there are many socket connections that are in 'CLOSE_WAIT' state.
I tried by calling the close() methods on the InputStream or OutputStream of an HttpURLConnection, but still the connections are in CLOSE_WAIT state only.
One other observation is even with con.disconnect() method also the connection is not being closed.
Please help me regarding this issue as CLOSE_WAIT connections indicate an error in the software.
Below is the code for get call. Other POST/PUT/DELETE calls are also like 'get'
public void get(String url, Header[] headers,
NameValuePair[] data, ResponseHandler handler) throws HttpException {
HttpURLConnection con = null;
try
{
String query = null;
if (data != null) {
query = getQueryString(data);
}
URL obj;
if(query == null || query == "")
obj = new URL(url);
else
obj = new URL(url+"?"+query);
con = (HttpURLConnection) obj.openConnection();
setDoAuthentication(con);
con.setRequestMethod("GET");
con.setDoOutput(true);
if (headers != null) {
for (int i = 0; i < headers.length; i++) {
con.setRequestProperty(headers[i].getName(), headers[i].getValue());
}
}
con.setRequestProperty("Accept-encoding", "gzip");
con.setRequestProperty("Authorization", this.auth);
int responseCode = con.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_OK) {
if (handler!=null) handler.handleResponse(con); // in handleResponse(con) method, I am closing the con input stream
} else if (responseCode == HttpURLConnection.HTTP_UNAUTHORIZED) {
throw new HttpException(new UnauthorizedException());
} else if (!is2xx(responseCode)) {
ErrorHandler errHandler = new ErrorHandler();
errHandler.handleResponse(con);
throw errHandler.getResult();
}
}
catch (MalformedURLException e) {
throw new HttpException(e);
}
catch (IOException e) {
handleSessionConnectionError(e, url);
}
finally {
con.disconnect();
}
}
Thanks
Mohan G
Calling con.disconnect() may cause the issue. Check this out this link link.
It will create sockets on and on, and if the connections are multiple, it will overpass the max limit of files open per process, e.g. in linux is usually 1024.
Instead of using disconnect, always close the input stream, the values defined in http.maxConnection in java system properties will do the work of closing idle connections when the limit is reach.
Check also javadoc of HttpUrlConnection in class and the method disconnect.
Please tell me some one, How to resolve this problem,
Sometime I am getting Filenotfound Exception and Some time this code working fine.
Below is my code,
public String sendSMS(String data, String url1) {
URL url;
String status = "Somthing wrong ";
try {
url = new URL(url1);
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
conn.setRequestProperty("User-Agent","Mozilla/5.0 ( compatible ) ");
conn.setRequestProperty("Accept","*/*");
OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
wr.write(data);
wr.flush();
// Get the response
try {
BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String s;
while ((s = rd.readLine()) != null) {
status = s;
}
rd.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
}
wr.close();
} catch (MalformedURLException e) {
status = "MalformedURLException Exception in sendSMS";
e.printStackTrace();
} catch (IOException e) {
status = "IO Exception in sendSMS";
e.printStackTrace();
}
return status;
}
Rewrite like this and let me know how you go... (note closing of reading and writing streams, also the cleanup of streams if an exception is thrown).
public String sendSMS(String data, String url1) {
URL url;
OutputStreamWriter wr = null;
BufferedReader rd = null;
String status = "Somthing wrong ";
try {
url = new URL(url1);
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
conn.setRequestProperty("User-Agent","Mozilla/5.0 ( compatible ) ");
conn.setRequestProperty("Accept","*/*");
wr = new OutputStreamWriter(conn.getOutputStream());
wr.write(data);
wr.flush();
wr.close();
rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String s;
while ((s = rd.readLine()) != null) {
status = s;
}
rd.close();
} catch (Exception e) {
if (wr != null) try { wr.close(); } catch (Exception x) {/*cleanup*/}
if (rd != null) try { rd.close(); } catch (Exception x) {/*cleanup*/}
e.printStackTrace();
}
return status;
}
This issue seems to be known, but for different reasons so its not clear why this happend.
Some threads would recommend closing the OutputStreamWriter as flushing it is not enough, therefor i would try to clos it directly after fushing as you are not using it in the code between the flush and close.
Other threads show that using a different connections like HttpURLConnection are avoiding this problem from occuring (Take a look here)
Another article suggests to use the URLEncoder class’ static method encode. This method takes a string and encodes it to a string that is ok to put in a URL.
Some similar questions:
URL is accessable with browser but still FileNotFoundException with URLConnection
URLConnection FileNotFoundException for non-standard HTTP port sources
URLConnection throwing FileNotFoundException
Wish you good luck.
It returns FileNotFoundException when the server response to HTTP request is code 404.
Check your URL.
I am making a crawler, and need to get the data from the stream regardless if it is a 200 or not. CURL is doing it, as well as any standard browser.
The following will not actually get the content of the request, even though there is some, an exception is thrown with the http error status code. I want the output regardless, is there a way? I prefer to use this library as it will actually do persistent connections, which is perfect for the type of crawling I am doing.
package test;
import java.net.*;
import java.io.*;
public class Test {
public static void main(String[] args) {
try {
URL url = new URL("http://github.com/XXXXXXXXXXXXXX");
URLConnection connection = url.openConnection();
DataInputStream inStream = new DataInputStream(connection.getInputStream());
String inputLine;
while ((inputLine = inStream.readLine()) != null) {
System.out.println(inputLine);
}
inStream.close();
} catch (MalformedURLException me) {
System.err.println("MalformedURLException: " + me);
} catch (IOException ioe) {
System.err.println("IOException: " + ioe);
}
}
}
Worked, thanks: Here is what I came up with - just as a rough proof of concept:
import java.net.*;
import java.io.*;
public class Test {
public static void main(String[] args) {
//InputStream error = ((HttpURLConnection) connection).getErrorStream();
URL url = null;
URLConnection connection = null;
String inputLine = "";
try {
url = new URL("http://verelo.com/asdfrwdfgdg");
connection = url.openConnection();
DataInputStream inStream = new DataInputStream(connection.getInputStream());
while ((inputLine = inStream.readLine()) != null) {
System.out.println(inputLine);
}
inStream.close();
} catch (MalformedURLException me) {
System.err.println("MalformedURLException: " + me);
} catch (IOException ioe) {
System.err.println("IOException: " + ioe);
InputStream error = ((HttpURLConnection) connection).getErrorStream();
try {
int data = error.read();
while (data != -1) {
//do something with data...
//System.out.println(data);
inputLine = inputLine + (char)data;
data = error.read();
//inputLine = inputLine + (char)data;
}
error.close();
} catch (Exception ex) {
try {
if (error != null) {
error.close();
}
} catch (Exception e) {
}
}
}
System.out.println(inputLine);
}
}
Simple:
URLConnection connection = url.openConnection();
InputStream is = connection.getInputStream();
if (connection instanceof HttpURLConnection) {
HttpURLConnection httpConn = (HttpURLConnection) connection;
int statusCode = httpConn.getResponseCode();
if (statusCode != 200 /* or statusCode >= 200 && statusCode < 300 */) {
is = httpConn.getErrorStream();
}
}
You can refer to Javadoc for explanation. The best way I would handle this is as follows:
URLConnection connection = url.openConnection();
InputStream is = null;
try {
is = connection.getInputStream();
} catch (IOException ioe) {
if (connection instanceof HttpURLConnection) {
HttpURLConnection httpConn = (HttpURLConnection) connection;
int statusCode = httpConn.getResponseCode();
if (statusCode != 200) {
is = httpConn.getErrorStream();
}
}
}
You need to do the following after calling openConnection.
Cast the URLConnection to HttpURLConnection
Call getResponseCode
If the response is a success, use getInputStream, otherwise use getErrorStream
(The test for success should be 200 <= code < 300 because there are valid HTTP success codes apart from than 200.)
I am making a crawler, and need to get the data from the stream regardless if it is a 200 or not.
Just be aware that it if the code is a 4xx or 5xx, then the "data" is likely to be an error page of some kind.
The final point that should be made is that you should always respect the "robots.txt" file ... and read the Terms of Service before crawling / scraping the content of a site whose owners might care. Simply blatting off GET requests is likely to annoy site owners ... unless you've already come to some sort of "arrangement" with them.