.getResponse Code Throwing IOException on a Valid URL - java

I'm building a webcrawler and have a method to check for bad link. At one point I am trying to get the HTTP response code to determine if it is valid or not. despite handing it a valid URL (opened it in a browser just fine) it still returns that it isn't valid. Here is the code:
public static boolean isBrokenLink(URL baseURL, String theHREF) {
boolean isBroken = false;
if (baseURL == null) {
try {
baseURL = new URL("HTTP", "cs.uwec.edu/~stevende/cs145testpages/", theHREF);
System.out.println(baseURL);
} catch (MalformedURLException e) {
isBroken = true;
//e.printStackTrace();
}
}
try {
URLConnection con = baseURL.openConnection();
HttpURLConnection httpProtocol = (HttpURLConnection) con;
System.out.println(httpProtocol.getResponseCode());
if (httpProtocol.getResponseCode() != 200 && httpProtocol.getResponseCode() == -1) {
isBroken = true;
}
} catch (IOException e) {
isBroken = true;
e.printStackTrace();
}
return isBroken;
}
}
And here is the URL I'm passing it. isBroken is the boolean that is being returned. I passing baseURL as null and theHREF as a relative link (page2.htm). I'm printing out the URL after creating it from the string. Thanks for any help!
Here is the error:
java.net.UnknownHostException: cs.uwec.edu/~stevende/cs145testpages/
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
at sun.net.www.http.HttpClient.New(HttpClient.java:308)
at sun.net.www.http.HttpClient.New(HttpClient.java:326)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1300)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
at edu.uwec.cs.carpenne.webcrawler.Webcrawler.isBrokenLink(Webcrawler.java:106)
at edu.uwec.cs.carpenne.webcrawler.Webcrawler.main(Webcrawler.java:181)

The exception tells us, that it is using the hostname and the local part as the (unknown) host. This looks like you have constructing the URL incorrectly. Maybe you forgot to use http:// prefix or used the wrong getters? You can debug it by calling baseURL.getHost(), baseURL.getPath() and baseURL.getProtocol() to see if it returns cs.uwec.edu and /~steve... and http.
I just noticed you added the baseURL with new URL("HTTP", "cs.uwec.edu/~stevende/cs145testpages/", theHREF) this is wrong, you need to use new URL("http", "cs.uwec.edu", 80, "/~stevende/cs145testpages/#"+theHREF). You can however typically skip the anchor/ref, as it will not transmitted to the server.
You can also use the single argument constructor new URL("http://cs.uwec.edu//~stevende/cs145testpages/").

Related

get lantern VPN running and could get access to website, but use eclipse java app got connection timed out

Use lantern VPN, the chrome browser and eclipse have the same proxy and URL could get response but java code gets connection timed out.
The lantern VPN proxy:
Eclipse proxy:
The result for the url http://api.zb.com/data/v1/markets:
The code for get response:
String callback= "";
try {
// request url
String url = ZBConfig.API_DATA + "/markets";
log.info("markets configuration url: " + url);
// request call back
callback = HttpUtilManager.get(url, "UTF-8");
log.info("-markets configuration url:: " + callback);
} catch (Exception ex) {
ex.printStackTrace();
}
public static String get(String urlAll, String charset) {
BufferedReader reader = null;
String result = null;
StringBuffer sbf = new StringBuffer();
String userAgent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.66 Safari/537.36";// 模拟浏览器
try {
URL url = new URL(urlAll);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("GET");
connection.setReadTimeout(30000);
connection.setConnectTimeout(30000);
connection.setRequestProperty("User-agent", userAgent);
connection.connect();
InputStream is = connection.getInputStream();
reader = new BufferedReader(new InputStreamReader(is, charset));
String strRead = null;
while ((strRead = reader.readLine()) != null) {
sbf.append(strRead);
sbf.append("\r\n");
}
reader.close();
result = sbf.toString();
} catch (Exception e) {
e.printStackTrace();
}
return result;
}
The error message from the console:
12:18:44.508 [main] INFO com.Test - markets configuration url: http://api.zb.com/data/v1/markets
java.net.ConnectException: Connection timed out: connect
12:19:06.095 [main] INFO com.Test - -markets configuration url:: null
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
at sun.net.www.http.HttpClient.New(HttpClient.java:339)
at sun.net.www.http.HttpClient.New(HttpClient.java:357)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
at com.zb.kits.HttpUtilManager.get(HttpUtilManager.java:353)
at com.Test.main(Test.java:201)
I had the same scenario in which I intended to bypass filtering which was applied on Youtube and instead of Lantern, I used to use Psiphon.
After running Psiphon it exposes a local port which you can use as proxy(and I'm sure Lantern is the same just check its log) and since it is applied as a system level proxy chrome can use it without any config, but if you wanna use it with Firefox you have to config it in the settings, and in order to use any proxy in your java codes you have to config jvm to use that proxy.
This is the sample code I used:
System.getProperties().put("http.proxyHost", Config.getHTTPPROXYHOST());
System.getProperties().put("http.proxyPort", Config.getHTTPPROXYPORT());
System.getProperties().put("https.proxyHost", Config.getHTTPSPROXYHOST());
System.getProperties().put("https.proxyPort", Config.getHTTPSPROXYPORT());
Also you can check the code in my github, in addition this post in stackoverflow can be quite useful.
hope it helps.

How to check if MongoDB connection is established with Java?

In my app, MongoDB 3.2.4 runs on a custom port, I want to implement logic where my app will try to reach MongoDB on a custom port and if it fails it will use the default 27018 port.
In order to do that I use the following code:
String mongoClientURI = "mongodb://" + DB_SRV_USR + ":" + DB_SRV_PWD + "#" + DB_URL + ":" + DB_PORT_CUS + "/" + dbName;
MongoClientURI connectionString = new MongoClientURI(mongoClientURI);
// enable SSL connection
MongoClientOptions.builder().sslEnabled(true).build();
if (this.mongoClient == null) {
this.mongoClient = new MongoClient(connectionString);
}
// create database if doesn't exist
MongoDatabase mdb = this.mongoClient.getDatabase(dbName);
try {
this.mongoClient.getAddress();
} catch (com.mongodb.MongoSocketOpenException e) {
System.out.println("Switch to default port");
/*…use default port logic…*/
}
The problem is that this exception is not caught.
Although MongoDB throws the following exception:
com.mongodb.MongoSocketOpenException: Exception opening socket at
com.mongodb.connection.SocketStream.open(SocketStream.java:63) at
com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:114)
at
com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:128)
at java.lang.Thread.run(Thread.java:745) Caused by:
java.net.ConnectException: Connection refused: connect at
java.net.DualStackPlainSocketImpl.waitForConnect(Native Method) at
java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) at
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at
java.net.Socket.connect(Socket.java:589) at
com.mongodb.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:50)
at com.mongodb.connection.SocketStream.open(SocketStream.java:58)
... 3 more
my try-catch expression can't catch this exception.
I tried multiple approaches, such as to catch:
Exception
RuntimeException
MongoSocketOpenException
MongoException
MongoCommandException
none of them doesn't work.
My questions:
How can I check if MongoDB connection is established?
How can catch the exception MongoSocketOpenException?
I use this code to check connection:
try {
mongo.getAddress();
} catch (Exception e) {
System.out.println("Database unavailable!");
mongo.close();
return;
}
Not sure here my guess would be that this.mongoClient.getAddress(); does not throw that exception, but I don't really know
EDIT: I initialized it via:
Builder builder = MongoClientOptions.builder().connectTimeout(3000);
MongoClient mongo = new MongoClient(new ServerAddress("192.168.0.1", 3000), builder.build());

Keep getting java.net.UnknownHostException after connecting to the internet

I'm using the Google HTTP Client Library for Java to develop a client application for a webservice of mine.
The problem is when I try to run some code like this:
private static void send() throws IOException {
HttpTransport httpTransport = new ApacheHttpTransport();
HttpRequestFactory httpRequestFactory = httpTransport
.createRequestFactory();
GenericUrl requestUrl = new GenericUrl("https://www.google.com/");
HttpRequest request = httpRequestFactory.buildGetRequest(requestUrl);
HttpResponse response = request.execute();
System.out.println(response.parseAsString());
}
public static void main(String args[]) throws InterruptedException {
boolean sent = false;
while (!sent) {
try {
send();
sent = true;
} catch (IOException e) {
e.printStackTrace();
Thread.sleep(1000);
}
}
}
I expected the program to keep trying to send the request each second until it's finnaly sent.
If I run the program with internet connected everything works fine.
If I disconnect from the internet and then run the program, I get a
java.net.UnknownHostException: www.google.com
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:333)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:123)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:147)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:108)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:415)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:641)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:576)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:554)
at com.google.api.client.http.apache.ApacheHttpRequest.execute(ApacheHttpRequest.java:67)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:965)
at test.client.MyClass.send(MyClass.java:112)
at test.client.MyClass.main(MyClass.java:124)
as expected.
The problem is that I keep getting this exception even after I reconnect to the internet.
Any solutions?
Thanks
========================== UPDATE ==========================
The problem doesn't seem to be relate to the ApacheLibrary as it also happens when using the NetHttpTransport implementation.
Just replaced new ApacheHttpTransport(); by new NetHttpTransport(); on the code above generating a new looping stack trace:
java.net.UnknownHostException: www.google.com
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:275)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:371)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:93)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:965)
at test.client.MyClass.send(MyClass.java:112)
at test.client.MyClass.main(MyClass.java:124)
Also I'm using a Fedora 20 Linux O.S. Kernel 3.16.6-200.
========================== UPDATE ==========================
Followed RealSkeptic's suggestion and debugged the code to find out where and how the DNS lookup is done.
I'm not familiar with java's low levew connection stuff, but I was able to produce the following almost equivalent code for the send function:
private static void send() throws IOException {
URL connUrl = new URL("https://www.google.com/");
URLConnection conn = connUrl.openConnection();
HttpURLConnection connection = (HttpURLConnection) conn;
connection.setRequestMethod("GET");
connection.setInstanceFollowRedirects(false);
connection.addRequestProperty("Accept-Encoding", "gzip");
connection.addRequestProperty("User-Agent",
"Google-HTTP-Java-Client/1.19.0 (gzip)");
connection.setReadTimeout(20000);
connection.setConnectTimeout(20000);
boolean successfulConnection = false;
try {
connection.connect();
//NetHttpResponse response = new NetHttpResponse(connection);
successfulConnection = true;
} finally {
if (!successfulConnection) {
connection.disconnect();
}
}
}
It's lacking on the response treatment.
With the code above, I keep getting the same result: A looping stacktrace that never ends.
Does anyone knows how to solve that?
Some enviroment information:
$cat /etc/resolv.conf
# Generated by NetworkManager
domain velox.com.br
search velox.com.br
nameserver 208.67.222.222
nameserver 8.8.8.8
$ java -version
java version "1.7.0_71"
OpenJDK Runtime Environment (fedora-2.5.3.0.fc20-x86_64 u71-b14)
OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)

How to check if URL is blocked by my firewall

I am trying to check if the URL is accessible or not. I am using HttpURLConnection for it. This is now I am implementing it.
public static boolean isUrlAccessible(final String urlToValidate)
throws WAGException {
URL url = null;
HttpURLConnection huc = null;
int responseCode = -1;
try {
url = new URL(urlToValidate);
huc = (HttpURLConnection) url.openConnection();
huc.setRequestMethod("HEAD");
huc.connect();
responseCode = huc.getResponseCode();
} catch (final UnknownHostException e) {
e.printStackTrace();
System.out.println(e.getMessage()+" "+e.getLocalizedMessage());
return false;
} catch (final MalformedURLException e){
e.printStackTrace();
System.out.println(e.getMessage()+" "+e.getLocalizedMessage());
return false;
} catch (ProtocolException e) {
e.printStackTrace();
System.out.println(e.getMessage()+" "+e.getLocalizedMessage());
return false;
} catch (IOException e) {
e.printStackTrace();
System.out.println(e.getMessage()+" "+e.getLocalizedMessage());
return false;
} finally {
if (huc != null) {
huc.disconnect();
}
}
return responseCode == 200;
}
When the Internet is down it throws an UnknownHostException, I wanted to know how do I check if a fire wall is blocking a URL and thats why I get an exception and not because that the URL is not accessible. Also, I am just checking for response code 200 to make sure that the URL is accessible. Are there any other checks I need to perform?
When the Internet is down it throws an UnknownHostException
No, it throws that when the DNS is down or the host isn't known to DNS.
I wanted to know how do I check if a fire wall is blocking a URL
You will get a connect timeout. In rare cases with obsolete hardware you may get a connection refusal, but I haven't heard of that this century. But you will also get a connect timeout if the host is down.
I am just checking for response code 200 to make sure that the URL is accessible. Are there any other checks I need to perform?
No. But URLs aren't blocked by firewalls. Ports are blocked by firewalls.
The exception I have usually seen when a firewall is blocking the connection is "java.net.NoRouteToHostException". Try catching that and see if it helps.
As others have said, the answer is "it depends".
Our perimeter firewall for example does a redirect, because we want to show the user a custom screen.
In this case, I would try to look into the HTTP Status code (30x).
I think it's hard to write a generic function for something like this, you need to tailor this to your setting or make it very configurable.
Just make sure to remain as generic as possible.
If your code for example assumes a redirect to a specific URL, this will beak once the infrastructure changes (which happens more often than anticipated).

Connection refused exception when using HttpURLConnection in Play framework

I'm using play 2.0.4, I need to connect to an url to see whether it's alive or not to make redirect decision.
public boolean checkServer(String strUrl){
try {
URL url = new URL(strUrl);
HttpURLConnection urlConn = (HttpURLConnection) url.openConnection();
urlConn.connect();
if(HttpURLConnection.HTTP_OK == urlConn.getResponseCode()){
urlConn.disconnect();
return true;
}else{
urlConn.disconnect();
return false;
}
} catch (IOException e) {
System.err.println("Error creating HTTP connection:" + e.getMessage());
return false;
}
}
In this case, the url that i use is localhost:9001, which is another play application. But then I received this error : Connection refused
Stack trace:
Error creating HTTP connection:Connection refused
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:378)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:473)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:203)
at sun.net.www.http.HttpClient.New(HttpClient.java:290)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:995)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:931)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:849)
at models.Backend.checkServer(Backend.java:19)
at controllers.Application.index(Application.java:22)
at Routes$$anonfun$routes$1$$anonfun$apply$1$$anonfun$apply$2.apply(routes_routing.scala:32)
at Routes$$anonfun$routes$1$$anonfun$apply$1$$anonfun$apply$2.apply(routes_routing.scala:32)
at play.core.Router$HandlerInvoker$$anon$5$$anon$1.invocation(Router.scala:1090)
at play.core.j.JavaAction$$anon$1.call(JavaAction.scala:33)
at play.core.j.JavaAction$class.apply(JavaAction.scala:74)
at play.core.Router$HandlerInvoker$$anon$5$$anon$1.apply(Router.scala:1089)
at play.core.ActionInvoker$$anonfun$receive$1$$anonfun$6.apply(Invoker.scala:126)
at play.core.ActionInvoker$$anonfun$receive$1$$anonfun$6.apply(Invoker.scala:126)
at play.utils.Threads$.withContextClassLoader(Threads.scala:17)
at play.core.ActionInvoker$$anonfun$receive$1.apply(Invoker.scala:125)
at play.core.ActionInvoker$$anonfun$receive$1.apply(Invoker.scala:115)
at akka.actor.Actor$class.apply(Actor.scala:318)
at play.core.ActionInvoker.apply(Invoker.scala:113)
at akka.actor.ActorCell.invoke(ActorCell.scala:626)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:197)
at akka.dispatch.Mailbox.run(Mailbox.scala:179)
at akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec(AbstractDispatcher.scala:516)
at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479)
at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
Can somebody please show me what was wrong here?
Thank you very much.

Categories