I am implementing a Java method that measures a number of metrics while loading a webpage. The metrics include : resolve time, the connect time and download time.
The challenge seems to be the name resolution, since the code should never trigger two NS look-ups by any means (even when DNS caching is disabled).
My first thought was to trigger the name resolution before connecting to the server, and then prevent java from running a second one upon connect.
Using InetAddress.getByName() for the name lookup and then HttpURLConnection and it's setRequestProperty method to set the a host header seemed to do the trick.
So here is my question: Do those two snippets below have the same effect? Do they always give the exact same result for all possible hosts? If not, what other options do I have?
VERSION 1: Implicit name resolution
/**
* Site content download Test
*
* #throws IOException
*/
public static void testMethod() throws IOException {
String protocol = "http";
String host = "stackoverflow.com";
String file = "/";
// create a URL object
URL url = new URL(protocol, host, file);
// create the connection object
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
// connect
conn.connect();
// create a stream reader
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
// read contents and print on std out
while ((inputLine = in.readLine()) != null) {
System.out.println(inputLine);
}
// close the stream
in.close();
}
VERSION 2: Explicit name resolution
/**
* Enhanced Site content download Test
*
* #throws IOException
*/
public static void testMethod2() throws IOException {
String protocol = "http";
String host = "stackoverflow.com";
String file = "/";
// Do a name lookup.
// If a literal IP address is supplied, only the validity of the address format is checked.
InetAddress address = InetAddress.getByName(host);
// create a URL object
URL url = new URL(protocol, address.getHostAddress(), file);
// create the connection object
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
// allow overriding Host and other restricted headers
System.setProperty("sun.net.http.allowRestrictedHeaders", "true");
// set the host header
conn.setRequestProperty("Host", host);
// connect
conn.connect();
// create a stream reader
BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String inputLine;
// read contents and print on std out
while ((inputLine = in.readLine()) != null) {
System.out.println(inputLine);
}
// close the stream
in.close();
}
TIA for the help.
-Dimi
I've browsed through Java's source code to see what happens when you pass a domain name to HttpURLConnection and it eventually ends up in NetworkClient.doConnect:
if (connectTimeout >= 0) {
s.connect(new InetSocketAddress(server, port), connectTimeout);
} else {
if (defaultConnectTimeout > 0) {
s.connect(new InetSocketAddress(server, port), defaultConnectTimeout);
} else {
s.connect(new InetSocketAddress(server, port));
}
}
As you see, the domain resolution is always handled by InetSocketAddress:
public InetSocketAddress(String hostname, int port) {
if (port < 0 || port > 0xFFFF) {
throw new IllegalArgumentException("port out of range:" + port);
}
if (hostname == null) {
throw new IllegalArgumentException("hostname can't be null");
}
try {
addr = InetAddress.getByName(hostname);
} catch(UnknownHostException e) {
this.hostname = hostname;
addr = null;
}
this.port = port;
}
As you can see, InetAddress.getByName is called everytime. I think that you method is safe.
Related
I am using Api in my Java App and triggering this URL (http://checkdnd.com/api/check_dnd_no_api.php?mobiles=9999999999). I am getting HTTP 403 error in console but in web browser no error occurs and getting the expected response. I also tried other URL and they work fine without problem or any errors.
So, what is the problem in URL and what should I do?
Here is source code :
Main.java
import org.json.simple.*;
import org.json.simple.parser.*;
public class Main
{
public static void main(String[] args) throws Exception
{
String numb = "9999999999,8888888888";
String response = new http_client("http://checkdnd.com/api/check_dnd_no_api.php?mobiles="+numb).response;
System.out.println(response);
// encoding response
Object obj = JSONValue.parse(response);
JSONObject jObj = (JSONObject) obj;
String msg = (String) jObj.get("msg");
System.out.println("MESSAGE : "+msg);
JSONObject msg_text = (JSONObject) jObj.get("msg_text");
String[] numbers = numb.split(",");
for(String number : numbers)
{
if(number.length() != 10 || number.matches(".*[A-Za-z].*")){
System.out.println(number+" is invalid.");
}else{
if(msg_text.get(number).equals("Y"))
{
System.out.println(number+" is DND Activated.");
}else{
System.out.println(number+" is not DND Activated.");
}
}
}
}
}
Now , http_client.java
import java.net.*;
import java.io.*;
public class http_client
{
String response = "";
http_client(String URL) throws Exception
{
URL url = new URL(URL);
HttpURLConnection con = (HttpURLConnection) url.openConnection();
con.setRequestMethod("GET");
BufferedReader bs = new BufferedReader(new InputStreamReader(con.getInputStream()));
String data ="";
String response = "";
while((data = bs.readLine()) != null){
response = response + data;
}
con.disconnect();
url = null;
con = null;
this.response = response;
}
}
Without you showing us the code you're using to access the supplied URL (http://checkdnd.com/api/check_dnd_no_api.php?mobiles=9999999999) it makes it a wee bit difficult to determine exactly where your problem lies but my first guess would be that the link you provided is only accessible through a Secure Socket Layer (SSL). In other words, the link should start with https:// instead of http://
To validate this simply make the change to your url string: https://checkdnd.com/api/check_dnd_no_api.php?mobiles=9999999999 and try again.
You're not going to have an issue with a browser for the simple reason that generally browsers will always try both protocols to make a connection. It is also up to the Website what protocol is acceptable, lots allow for both and some just don't.
To check if a url string is utilizing a valid protocol you can use this little method I quickly whipped up:
/**
* This method will take the supplied URL String regardless of the protocol (http or https)
* specified at the beginning of the string, and will return whether or not it is an actual
* "http" (no SSL) or "https" (is SSL) protocol. A connection to the URL is attempted first
* with the http protocol and if successful (by way of data acquisition) will then return
* that protocol. If not however, then the https protocol is attempted and if successful then
* that protocol is returned. If neither protocols were successful then Null is returned.<br><br>
*
* Returns null if the supplied URL String is invalid, a protocol does not
* exist, or a valid connection to the URL can not be established.<br><br>
*
* #param webLink (String) The full link path.<br>
*
* #return (String) Either "http" for Non SLL link, "https" for a SSL link.
* Null is returned if the supplied URL String is invalid, a protocol does
* not exist, or a valid connection to the URL can not be established.
*/
public static String isHttpOrHttps(String webLink) {
URL url;
try {
url = new URL(webLink);
} catch (MalformedURLException ex) { return null; }
String protocol = url.getProtocol();
if (protocol.equals("")) { return null; }
URLConnection yc;
try {
yc = url.openConnection();
BufferedReader in = new BufferedReader(new InputStreamReader(yc.getInputStream()));
in.close();
return "http";
} catch (IOException e) {
// Do nothing....check for https instead.
}
try {
yc = new URL(webLink).openConnection();
//send request for page data...
yc.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
yc.connect();
BufferedReader in = new BufferedReader(new InputStreamReader(yc.getInputStream()));
in.close();
return "https";
} catch (IOException e) {
// Do Nothing....allow for Null to be returned.
}
return null;
}
To use this method:
// Note that the http protocol is supplied within the url string:
String protocol = isHttpOrHttps("http://checkdnd.com/api/check_dnd_no_api.php?mobiles=9999999999");
System.out.println(protocol);
The output to console will be: https. The isHttpOrHttps() method has determined that the https protocol is the successful protocol to use in order to acquire data (or whatever) even though http was supplied.
To pull the page source from the web page you can perhaps use a method like this:
/**
* Returns a List ArrayList containing the page source for the supplied web
* page link.<br><br>
*
* #param link (String) The URL address of the web page to process.<br>
*
* #return (List ArrayList) A List ArrayList containing the page source for
* the supplied web page link.
*/
public static List<String> getWebPageSource(String link) {
if (link.equals("")) { return null; }
try {
URL url = new URL(link);
URLConnection yc = null;
//If url is a SSL Endpoint (using a Secure Socket Layer such as https)...
if (link.startsWith("https:")) {
yc = new URL(link).openConnection();
//send request for page data...
yc.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
yc.connect();
}
//and if not a SLL Endpoint (just http)...
else { yc = url.openConnection(); }
BufferedReader in = new BufferedReader(new InputStreamReader(yc.getInputStream()));
String inputLine;
List<String> sourceText = new ArrayList<>();
while ((inputLine = in.readLine()) != null) {
sourceText.add(inputLine);
}
in.close();
return sourceText;
}
catch (MalformedURLException ex) {
// Do whatever you want with exception.
ex.printStackTrace();
}
catch (IOException ex) {
// Do whatever you want with exception.
ex.printStackTrace();
}
return null;
}
In order to utilize both the methods supplied here you can try something like this:
String netLink = "http://checkdnd.com/api/check_dnd_no_api.php?mobiles=9999999999";
String protocol = isHttpOrHttps(netLink);
String netLinkProtocol = netLink.substring(0, netLink.indexOf(":"));
if (!netLinkProtocol.equals(protocol)) {
netLink = protocol + netLink.substring(netLink.indexOf(":"));
}
List<String> list = getWebPageSource(netLink);
for (int i = 0; i < list.size(); i++) {
System.out.println(list.get(i));
}
And the console output will display:
{"msg":"success","msg_text":{"9999999999":"N"}}
I have an interesting scenario here. I have a proxy server address that is supposed to provide me a new exit IP every time I make an HTTP request to it. I have noticed that the exit IP will change only after I restart the program, as opposed to every loop iteration. Below is my source.
Loop calls getHTML every iteration:
String result = getHTML("https://wtfismyip.com/text");
public static String getHTML(String urlToRead) throws Exception {
InetSocketAddress addy = new InetSocketAddress("example.proxy.com", 1234);
Proxy proxy = new Proxy(Proxy.Type.HTTP, addy);
StringBuilder result = new StringBuilder();
URL url = new URL(urlToRead);
HttpURLConnection conn = (HttpURLConnection) url.openConnection(proxy);
conn.setRequestMethod("GET");
BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line;
while ((line = rd.readLine()) != null) {
result.append(line);
}
rd.close();
conn.disconnect();
return result.toString();
}
Result will continue to be the same IP every time, until I restart the program. I feel like some stream or socket hasn't been closed yet and it keeps the connection alive.
Found the answer to my problem. The TCP socket was being kept alive, and would allow it to keep tunneling to the proxy without reconnecting.
I needed to add this statement somewhere in my code, I put it at the beginning of this class's initialization.
System.setProperty("http.keepAlive", "false");
I am new to android and Java. And I am trying to learn android app development from UDACITY. I was trying to run this code and I am expecting a SocketTimeOutExcepetion but what I am getting is UnknownHostException.
try {
final String BASE_URL = "http://api.openweathermap.org/data/2.5/forecast/daily?";
final String ZIP = "zip";
final String MODE = "mode";
final String UNITS = "units";
final String COUNT = "cnt";
final String APP_ID = "appid";
Uri builtUri = Uri.parse(BASE_URL).buildUpon()
.appendQueryParameter(ZIP, params[0] + ",in")
.appendQueryParameter(MODE,format)
.appendQueryParameter(UNITS, units)
.appendQueryParameter(COUNT, Integer.toString(numDays))
.appendQueryParameter(APP_ID, BuildConfig.OPEN_WEATHER_MAP_API_KEY)
.build();
String str = java.net.URLDecoder.decode(builtUri.toString());
URL url = new URL(str);
urlConnection = (HttpURLConnection) url.openConnection();
urlConnection.setRequestMethod("GET");
urlConnection.setConnectTimeout(5000);
urlConnection.setReadTimeout(5000);
urlConnection.connect();
InputStream inputStream = urlConnection.getInputStream();
StringBuffer buffer = new StringBuffer();
if (inputStream == null) {
return null;
}
reader = new BufferedReader(new InputStreamReader(inputStream));
String line;
while ((line = reader.readLine()) != null)
buffer.append(line + "/n");
if (buffer.length() == 0)
return null;
forecastJsonStr = buffer.toString();
Log.v(LOG_TAG,"JSON forcast string:" +forecastJsonStr);
}catch(SocketTimeoutException e) {
startActivity(new Intent(getActivity(),CheckNet.class));
} catch (IOException e) {
Log.e("FetchWeatherTask", "Error:" + e.toString());
return null;
}
I tested it on my phone running Android version 4.0.4. And while testing I had my mobile data and wifi off
When your mobile data and wifi are turned off, the socket layer is unable to resolve internet addresses (e.g. "openweathermap.org") into an IP address. This is why you get an UnknownHostException.
Whereas, when you're on a network, and it's able to resolve IP addresses, and the server fails to reply, you will get a SocketTimeoutException.
If you want to simulate the exception do the following:
Disconnect your data and connect your Wi-Fi
Edit your setting of your Wi-Fi connection
Change to static IP and put 169.254.0.50 for IP, 255.255.0.0 for subnet and 169.254.0.1 for gateway
Change the BASE_URL = "192.241.169.168/data/2.5/forecast/daily?"
Run your app
I'm trying to invoke a webservice call and get a response. When I tried it first time it worked perfectly and printed the response. But after that one run, how many ever times I run it, i throws me
Exception in thread "main" java.lang.IllegalStateException: Already connected
at sun.net.www.protocol.http.HttpURLConnection.setRequestProperty(Unknown Source)
at SOAPClient4XG.main(SOAPClient4XG.java:72)
I have tried various solutions provided for similar problem (like connect / disconnect) but nothing seems to make it work. I understand that it tries to perform an operation on already existing connection, but not sure how to fix. I'm fairly new to all this and I need help.
Below is my code
import java.io.*;
import java.net.*;
public class SOAPClient4XG
{
private static HttpURLConnection httpConn;
public static void main(String[] args) throws Exception {
String SOAPUrl = args[0];
String xmlFile2Send = args[1];*/
String SOAPUrl = "http://10.153.219.88:8011/celg-svcs-soap/business/ApplicantEligibility";
String xmlFile2Send =
"C:\\Users\\dkrishnamoorthy\\workspace\\SOAPUI_Automation\\src\\ApplicantElligibilty.xml";
String SOAPAction = "";
if (args.length > 2)
SOAPAction = args[2];
// Create the connection where we're going to send the file.
URL url = new URL(SOAPUrl);
URLConnection connection = url.openConnection();
//URLConnection connection = new URLConnection(url);
httpConn = (HttpURLConnection) connection;
if(httpConn.getResponseCode()==500)
{
System.out.println("Error Stream for 500 : "+httpConn.getErrorStream());
}
// Open the input file. After we copy it to a byte array, we can see
// how big it is so that we can set the HTTP Cotent-Length
// property. (See complete e-mail below for more on this.)
FileInputStream fin = new FileInputStream(xmlFile2Send);
ByteArrayOutputStream bout = new ByteArrayOutputStream();
// Copy the SOAP file to the open connection.
copy(fin,bout);
fin.close();
byte[] b = bout.toByteArray();
// Set the appropriate HTTP parameters.
httpConn.setRequestProperty( "Content-Length",
String.valueOf( b.length ) );
httpConn.setRequestProperty("Content-Type","text/xml; charset=utf-8");
httpConn.setRequestProperty("SOAPAction",SOAPAction);
httpConn.setRequestMethod( "POST" );
httpConn.setDoOutput(true);
httpConn.setDoInput(true);
// httpConn.connect();
// Everything's set up; send the XML that was read in to b.
OutputStream out = httpConn.getOutputStream();
out.write( b );
out.close();
// Read the response and write it to standard out.
InputStreamReader isr =
new InputStreamReader(httpConn.getInputStream());
BufferedReader in = new BufferedReader(isr);
String inputLine;
System.out.println("Printing the Response ");
while ((inputLine = in.readLine()) != null)
System.out.println(inputLine);
in.close();
}
public static void copy(InputStream in, OutputStream out)
throws IOException {
synchronized (in) {
synchronized (out) {
byte[] buffer = new byte[256];
while (true) {
int bytesRead = in.read(buffer);
if (bytesRead == -1) break;
out.write(buffer, 0, bytesRead);
}
}
}
}
}
If you use eclipse version just restart it. I met the same issue and I sorted out by doing that .
I solved this because I had a forgotten watch for connection.getResponseCode() in my debugging interface in NetBeans. Hope it might help others making the same mistake.
If you have any watch relative to the response value of the request, such as getResponseCode(), getResponseMessage(), getInputStream() or even just connect(), you will get this error in debugging mode.
All of the previous methods implicitly call connect() and fire the request. So when you reach setDoOutput, the connection is already made.
Some times work throuw the proxy server and reading prom buffer content my program to think so more time... until i close them. How set program code that from some seconds if do not have any answer from server to take another server?
URL url = new URL(linkCar);
String your_proxy_host = new String(proxys.getValueAt(xProxy, 1).toString());
int your_proxy_port = Integer.parseInt(proxys.getValueAt(xProxy, 2).toString());
Proxy proxy = null;
// System.out.println(proxys.getValueAt(xProxy, 3).toString());
// if (proxys.getValueAt(xProxy, 3).toString().indexOf("HTTP") > 0)
// {
proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(your_proxy_host, your_proxy_port));
// } else {
// proxy = new Proxy(Proxy.Type.SOCKS, new InetSocketAddress(your_proxy_host, your_proxy_port));
// }
HttpURLConnection connection = (HttpURLConnection)url.openConnection(proxy);
connection.setConnectTimeout(1000);
connection.connect();
String line = null;
StringBuffer buffer_page = new StringBuffer();
BufferedReader buffer_input = new BufferedReader(new InputStreamReader(connection.getInputStream(),"cp1251"));
int cc = 0;
//this is thinking place!!!
while ((line = buffer_input.readLine()) != null && cc < 7000) {
buffer_page.append(line);
cc++;
}
doc = Jsoup.parse(String.valueOf(buffer_page));
connection.disconnect();
i tried to use counter but it not work... What exception i can use to catch this situation by my control?
You need to use URLConnection.setReadTimeout. From the specification,
Sets the read timeout to a specified timeout, in milliseconds. A non-zero value specifies the timeout when reading from Input stream when a connection is established to a resource. If the timeout expires before there is data available for read, a java.net.SocketTimeoutException is raised. A timeout of zero is interpreted as an infinite timeout.
As you can see, reads that time-out will throw SocketTimeoutException, which you can catch appropriately, e.g.
try (BufferedReader buffer_input = new BufferedReader(
new InputStreamReader(connection.getInputStream(), "cp1251"))) {
String line;
while ((line = buffer_input.readLine()) != null) {
buffer_page.append(line);
}
} catch (SocketTimeoutException ex) {
/* handle time-out */
}
Note that you need to be careful when using readLine as above -- this will strip all \r and \n from the input.