I have an URLConnection which I want to cancel depending on the response code without reading any data. I closely followed the android training to build the following minimal example which
floods the server with requests since no connection is ever released back to the handle pool for reuse
private String downloadUrl(String myurl) throws IOException {
InputStream is = null;
try {
URL url = new URL(myurl);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setReadTimeout(10000 /* milliseconds */);
conn.setConnectTimeout(15000 /* milliseconds */);
conn.setRequestMethod("GET");
conn.setDoInput(true);
// Starts the query
conn.connect();
int response = conn.getResponseCode();
Log.d(TAG, "The response code is: " + response);
is = conn.getInputStream();
// Do not read anything //String contentAsString = readIt(is, len);
String contentAsString = "notReadingAnything";
return contentAsString;
} finally {
if (is != null) {
is.close();
}
}
}
private class DownloadWebpageTask extends AsyncTask<String, Void, String> {
#Override
protected String doInBackground(String... urls) {
try {
String result = new String();
for (int i=0; i<100; i++) {
result += downloadUrl(urls[0]);
}
return result;
} catch (IOException e) {
return "Unable to retrieve web page. URL may be invalid.";
}
}
#Override
protected void onPostExecute(String result) {
Log.d(TAG, "The response is: " + result);
}
}
Despite the docs explicitly stating
But if the response body is long and you are not interested in the rest of it after seeing the beginning, you can close the InputStream
the server quickly reaches its maximum number of connections (50) and goes to 99% workload if I don't read the stream but works fine if I do read it. What is my mistake?
EDIT: Failed solution attempts so far (thanks to #Blackbelt for most of them)
calling conn.disconnect() in the finally block
calling conn.disconnect() instead of is.close() in the finally block
Setting System.setProperty("http.keepAlive", "false"); before the first call
Setting conn.setRequestProperty("Connection", "Close"); before connecting
Setting "{enable_keep_alive", "no"} on the used backend server (Civetweb)
you should call disconnect() too. Accordingly to the documentation
Disconnect. Once the response body has been read, the
HttpURLConnection should be closed by calling disconnect().
Disconnecting releases the resources held by a connection so they may
be closed or reused.
InputStream is = null;
HttpURLConnection conn = null;
try {
URL url = new URL(myurl);
conn = (HttpURLConnection) url.openConnection();
} finally {
if (is != null) {
is.close();
}
if (conn != null) {
conn.disconnect();
}
}
if you still are experiencing issues, is also possible that the bug is backend side
Related
How do I make my program return MalformedUrlException and not just a generic Exception?
I am making a simple function that reads a URL a user enters in the console and it returns the content from the URL. I need it to check if the URL is a valid URL or if it's not a working URL.
Example urls:
http://google.com/not-found.html
http:/google.com
I created two catch exceptions but it seems like the overall exception is always returned instead of MalformedUrlException.
public static String getUrlContents(String theUrl) {
String content = "";
try {
URL url = new URL(theUrl);
//Create a url connection object
URLConnection urlConnection = url.openConnection();
//wrap the url connection a buffered reader
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
String line;
while((line = bufferedReader.readLine()) != null) {
content += line + "\n";
}
bufferedReader.close();
} catch (MalformedURLException e) {
System.out.println("The following url is invalid'" + theUrl + "'");
//logging error should go here
} catch (Exception e) {
System.out.println("Something went wrong, try agian");
}
return content;
}
First, java.net.MalformedURLException is not the case for a "not found" resource:
public class MalformedURLException extends IOException
Thrown to indicate that a malformed URL has occurred. Either no legal
protocol could be found in a specification string or the string could
not be parsed.
I understand that you want to catch a situation when the URL results in a not found return code (404). To do this you need to examine the HTTP response code.
The easiest way is to use java.net.HttpURLConnection:
https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/net/HttpURLConnection.html
public abstract class HttpURLConnection extends URLConnection
A URLConnection with support for HTTP-specific features. See the spec
for details.
Each HttpURLConnection instance is used to make a single request but
the underlying network connection to the HTTP server may be
transparently shared by other instances. Calling the close() methods
on the InputStream or OutputStream of an HttpURLConnection after a
request may free network resources associated with this instance but
has no effect on any shared persistent connection. Calling the
disconnect() method may close the underlying socket if a persistent
connection is otherwise idle at that time.
You can check the response code by calling getResponseCode(). If the result is less than 400, you got a valid response, otherwise there was a client error (4xx) or a server error (5xx).
Something like this:
public static String getUrlContents(String theUrl) {
String content = "";
try {
URL url = new URL(theUrl);
//Create a url connection object
URLConnection urlConnection = url.openConnection();
if (urlConnection instanceof HttpURLConnection) {
HttpURLConnection conn = (HttpURLConnection) urlConnection;
if (conn.getResponseCode() < 400) {
// read contents
} else {
System.out.println(conn.getResponseMessage());
// treat the error as you like
}
} else {
// not a HTTP connection, treat as you like
}
} catch (MalformedURLException e) {
System.out.println("The following url is invalid'" + theUrl + "'");
//logging error should go here
} catch (Exception e) {
System.out.println("Something went wrong, try agian");
}
return content;
}
I have not checked the code, but I think you can get the overall idea.
I am using java HttpURLConnection for REST calls. After working(testing of rest calls using ant targets) with all the rest api's I see there are many socket connections that are in 'CLOSE_WAIT' state.
I tried by calling the close() methods on the InputStream or OutputStream of an HttpURLConnection, but still the connections are in CLOSE_WAIT state only.
One other observation is even with con.disconnect() method also the connection is not being closed.
Please help me regarding this issue as CLOSE_WAIT connections indicate an error in the software.
Below is the code for get call. Other POST/PUT/DELETE calls are also like 'get'
public void get(String url, Header[] headers,
NameValuePair[] data, ResponseHandler handler) throws HttpException {
HttpURLConnection con = null;
try
{
String query = null;
if (data != null) {
query = getQueryString(data);
}
URL obj;
if(query == null || query == "")
obj = new URL(url);
else
obj = new URL(url+"?"+query);
con = (HttpURLConnection) obj.openConnection();
setDoAuthentication(con);
con.setRequestMethod("GET");
con.setDoOutput(true);
if (headers != null) {
for (int i = 0; i < headers.length; i++) {
con.setRequestProperty(headers[i].getName(), headers[i].getValue());
}
}
con.setRequestProperty("Accept-encoding", "gzip");
con.setRequestProperty("Authorization", this.auth);
int responseCode = con.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_OK) {
if (handler!=null) handler.handleResponse(con); // in handleResponse(con) method, I am closing the con input stream
} else if (responseCode == HttpURLConnection.HTTP_UNAUTHORIZED) {
throw new HttpException(new UnauthorizedException());
} else if (!is2xx(responseCode)) {
ErrorHandler errHandler = new ErrorHandler();
errHandler.handleResponse(con);
throw errHandler.getResult();
}
}
catch (MalformedURLException e) {
throw new HttpException(e);
}
catch (IOException e) {
handleSessionConnectionError(e, url);
}
finally {
con.disconnect();
}
}
Thanks
Mohan G
Calling con.disconnect() may cause the issue. Check this out this link link.
It will create sockets on and on, and if the connections are multiple, it will overpass the max limit of files open per process, e.g. in linux is usually 1024.
Instead of using disconnect, always close the input stream, the values defined in http.maxConnection in java system properties will do the work of closing idle connections when the limit is reach.
Check also javadoc of HttpUrlConnection in class and the method disconnect.
I'm trying to make a POST-request using android HttpUrlConnection. First I use the example for GET-request from here:
http://developer.android.com/training/basics/network-ops/connecting.html#http-client
It works perfectly (for example I get google.com page). Then I make some changes to make a POST-request: change the request method on POST:
conn.setRequestMethod("POST");
and add this code (got from here: http://developer.android.com/reference/java/net/HttpURLConnection.html):
conn.setDoOutput(true);
conn.setChunkedStreamingMode(0);
OutputStream out = new BufferedOutputStream(conn.getOutputStream());
out.close();
So now the method downloadUrl looks like this:
private String downloadUrl(String myurl) throws IOException {
InputStream is = null;
// Only display the first 500 characters of the retrieved
// web page content.
int len = 500;
try {
URL url = new URL(myurl);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setReadTimeout(10000 /* milliseconds */);
conn.setConnectTimeout(15000 /* milliseconds */);
conn.setDoInput(true);
conn.setRequestMethod("POST");
conn.setDoOutput(true);
conn.setChunkedStreamingMode(0);
OutputStream out = new BufferedOutputStream(conn.getOutputStream());
out.close();
// Starts the query
conn.connect();
int response = conn.getResponseCode();
Log.d(DEBUG_TAG, "The response is: " + response);
is = conn.getInputStream();
// Convert the InputStream into a string
String contentAsString = readIt(is, len);
return contentAsString;
// Makes sure that the InputStream is closed after the app is
// finished using it.
} finally {
if (is != null) {
is.close();
}
}
}
And it always throws IOException. Could you help me, what is wrong?
It's because Android won't let you start a network connection on the main UI thread. You have to start a background thread (use AsyncTask) and do it from there.
More detail in this question.
I've sold this problem: the thing was the server didn't accept POST-requests on the selected URL. Changing URL (and server) led to successful request without throwing an exception.
When I am testing the list of url's to see if the connection is "alive" or "dead", I run into this site: (www.abbapregnancy.org) then site is blank with no information. How can I make an if statement to test for sites like this? Feel free to comment or suggest ideas. My code is currently like this:
try
{// Test URL Connection
URL url = new URL("http://www."+line);
URLConnection conn = url.openConnection();
conn.setDoOutput(true);
wr = new OutputStreamWriter(conn.getOutputStream());
wr.flush();
rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
while( !rd.ready() ) {}
if( rd.ready())
{
//write to output file
System.out.println("Good URL: " + line);
urlAlive ++;
//increment alive line to Alive url file
aliveUrl += (line + System.getProperty("line.separator"));
}
else // increment dead url
urlDead++;
}
catch (Exception e)
// increment dead url
{
urlDead++;
}
So you want to check more than just "alive" or "dead". If alive, you also want to know if the content is empty, right?
Since you seem to want to check websites, using http connections makes sense here. For the alive/dead condition, connection attempts will throw an exception if dead. If alive, you can use the status code to check if the request was really successful. Finally, you can use the content-length header to check if the site is up but actually returning empty content. For example like this:
public void checkURL(URL url) throws IOException {
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("GET");
System.out.println(String.format("Fetching %s ...", url));
try {
int responseCode = conn.getResponseCode();
if (responseCode == 200) {
System.out.println(String.format("Site is up, content length = %s", conn.getHeaderField("content-length")));
} else {
System.out.println(String.format("Site is up, but returns non-ok status = %d", responseCode));
}
} catch (java.net.UnknownHostException e) {
System.out.println("Site is down");
}
}
public static boolean IsReachable(Context context, String check_url) {
// First, check we have any sort of connectivity
final ConnectivityManager connMgr = (ConnectivityManager) context.getSystemService(Context.CONNECTIVITY_SERVICE);
final NetworkInfo netInfo = connMgr.getActiveNetworkInfo();
boolean isReachable = false;
if (netInfo != null && netInfo.isConnected()) {
// Some sort of connection is open, check if server is reachable
try {
URL url = new URL(check_url);
// URL url = new URL("http://192.168.100.93/office_com/www/api.php/office_com/Logins/SignUp?api_user=endpoint");
//URL url = new URL("http://10.0.2.2");
HttpURLConnection urlc = (HttpURLConnection) url.openConnection();
urlc.setRequestProperty("User-Agent", "Android Application");
urlc.setRequestProperty("Connection", "close");
urlc.setConnectTimeout(10 * 1000);
try {
urlc.connect();
System.out.println("-----fffff");
} catch (Exception e) {
System.out.println("-----fffff " + e);
}
isReachable = (urlc.getResponseCode() == 200);
} catch (IOException e) {
}
}
return isReachable;
}
I've already looked through resources which describe specifics of work with HttpURLConnection as in Java so in Android (where there is not default implementation of working with connection pool) and my question is: if I close a stream gotten from HttpURLConnection, will the system create a new one Socket at a next time when I establish a HttpURLConnection with the same URL or it will try to use an exist one? Considering the following code:
private byte[] downloadText(URL url, String data) {
OutputStream out = null;
InputStream in = null;
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
try {
conn.setRequestMethod("POST");
conn.setReadTimeout(20 * 1000);
conn.setConnectTimeout(15 * 000);
conn.setDoInput(true);
conn.setDoOutput(true);
conn.setRequestProperty("Content-Type", "application/json;charset=utf-8");
byte[] payload = data.getBytes("utf-8");
conn.setFixedLengthStreamingMode(payload.length);
conn.connect();
out = new BufferedOutputStream(conn.getOutputStream());
out.write(payload);
out.flush();
final int responseCode = conn.getResponseCode();
final String responseMessage = conn.getResponseMessage();
is = conn.getInputStream();
if((responseCode / 100) == 2 || responseMessage.equals("OK")) {
return readFromStream(is);
}
} catch (IOException e) {
Log.e(TAG, "Error occurred while trying to connect to the server" + e.toString());
} finally {
try {
if(out != null) {
out.close();
}
if(is != null) {
is.close();
}
} catch(IOException e) {
Log.e(TAG, "Error occurred while trying to close data streams" + e.toString());
}
}
}
Your question doesn't make sense. If there isn't a connection pool, there is no 'existing socket' to reuse.