I have an interesting scenario here. I have a proxy server address that is supposed to provide me a new exit IP every time I make an HTTP request to it. I have noticed that the exit IP will change only after I restart the program, as opposed to every loop iteration. Below is my source.
Loop calls getHTML every iteration:
String result = getHTML("https://wtfismyip.com/text");
public static String getHTML(String urlToRead) throws Exception {
InetSocketAddress addy = new InetSocketAddress("example.proxy.com", 1234);
Proxy proxy = new Proxy(Proxy.Type.HTTP, addy);
StringBuilder result = new StringBuilder();
URL url = new URL(urlToRead);
HttpURLConnection conn = (HttpURLConnection) url.openConnection(proxy);
conn.setRequestMethod("GET");
BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
String line;
while ((line = rd.readLine()) != null) {
result.append(line);
}
rd.close();
conn.disconnect();
return result.toString();
}
Result will continue to be the same IP every time, until I restart the program. I feel like some stream or socket hasn't been closed yet and it keeps the connection alive.
Found the answer to my problem. The TCP socket was being kept alive, and would allow it to keep tunneling to the proxy without reconnecting.
I needed to add this statement somewhere in my code, I put it at the beginning of this class's initialization.
System.setProperty("http.keepAlive", "false");
Related
Here is my code in jsp:
<c:catch var="feederror">
<%
String feedUrl = (String) pageContext.getAttribute("url");
String feedXml = "";
HttpURLConnection conn = (HttpURLConnection) new URL(feedUrl).openConnection();
BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream(), "UTF-8"));
String line;
while ((line = reader.readLine()) != null) {
feedXml += line;
}
reader.close();
pageContext.setAttribute("feedXml", feedXml.trim().replaceAll("",""));
%>
</c:catch>
The variable feederror returns java.net.ProtocolException: Server reditected too many times(20).
I have tried:
to open browser with disabled cookies - page do not load at all (404);
used CookieHandler.setDefault(new CookieManager(null, CookiePolicy.ACCEPT_ALL)) before openning connection - didn't change anything;
tried CookieHandler.setDefault(new CookieManager()) - didn't change anything;
How to solve the problem?
java.net.ProtocolException: Server reditected too many times(20).
This exception comes from HttpURLConnection conn = (HttpURLConnection) new URL(feedUrl).openConnection();
For 99% you're calling an url, which sends a redirect to itself, or site B, which redirect to site A, which redirects to site B etc. so there is a loop.
Check what URL you're trying to call and verify it via some external tool, so you can check headers and responses from external server.
I think it has nothing to do with cookies
I am doing an desktop application using Java that can send SMS to a multiple phone numbers via browser. To send the SMS, I need to put in parameters that username, password, message to be sent and the recipient's number.
I have used this and it worked but the problem is it opens up a browser to every recipient so it is not advisable especially if its going to be sent to a lot:
Runtime rt = Runtime.getRuntime();
String url = String.format("http://www.companysms.com/RemoteAPI/SendSMS.aspx?username=%s&encoding=url&password=%s&messagedata=%s&receiver=%s&binary=0", txtUsername.getText(),txtPassword.getText(), txtMessage.getText(),tblMessage.getValueAt(i, 2));
rt.exec("rundll32 url.dll,FileProtocolHandler "+ url);
I have tried this one too but it gives me the HTTP response code 400:
HttpURLConnection connection = (HttpURLConnection) myURL.openConnection();
connection.setRequestMethod("GET");
connection.setDoOutput(true);
connection.connect();
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
StringBuilder results = new StringBuilder();
String line;
while ((line = reader.readLine()) != null) {
results.append(line);
}
connection.disconnect();
If I use the URLencoder with "UTF-8", it makes it worse because i cant be sure of what the message the user will send and if they will use any of the special characters. Is there a way to have the first set of codes that I used to not open browser after sending the SMS or is there another way of doing this? Any help is appreciated. Thanks!
I am writing a Java program to compute the http connection time for (lets say) 5 http connection (to different IP).
The first scenario is, without threading, the program connect and testing the http server one by one which mean when finish one server testing then proceed to another. In this scenario, the time taken is very long. Moreover, the timeout is not working properly, for example, I have set the
setConnectTimeout(5 * 1000);
setReadTimeout(5 * 1000);
but the time return by
long starTime = System.currentTimeMillis();
c.connect();
String line;
BufferedReader in = new BufferedReader(new InputStreamReader(uc.getInputStream()));
while ((line = in.readLine()) != null){
page.append(line);
elapseTime = System.currentTimeMillis() - starTime;
can be more than 5 second, some even go up to 30 second (but I set 5 second as timeout only).
So, I make the implementation to be multithreading. But the result is more rediculous. I can't even get one successful connection now.
Now my question is, can we establish multiple connection by using multiple thread? If answer is yes, what I have to notice to avoid the issue above?
Thank.
*Extra info*
1) I am computing the proxy connection speed, so, ya, the connection is proxy connection.
2) The threads that I created is around 100. I think it should be fine right?
How are you setting up your connections? Are you using a socket connection? If so, depending on how you setup your socket, you may find that the connection timeout value may be ignored
Socket sock = new Socket("hostname", port);
sock.setSoTimeout(5000);
sock.connect();
Will actually not set the connect timeout value, as the constructor will already attempt to connect.
SocketAddress sockaddr = new InetSocketAddress(host, port);
Socket sock = new Socket();
sock.connect(sockaddr, 5000);
Will more accurately connect with a timeout value. This may explain why your socket timeouts are not working.
public float getConnectionTime(){
long elapseTime = 0;
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(ipAdd, portNum));
URL url;
StringBuilder page = new StringBuilder();
HttpURLConnection uc = null;
try {
uc = (HttpURLConnection)Main.targetMachine.openConnection(proxy);
// uc = (HttpURLConnection)Main.targetMachine.openConnection();
uc.setConnectTimeout(Main.timeOut);
uc.setReadTimeout(Main.timeOut);
long starTime = System.currentTimeMillis();
uc.connect();
// if (uc.getResponseCode() == HttpURLConnection.HTTP_OK){
// System.out.println("55555");
// }else System.out.println("88888");
String line;
BufferedReader in = new BufferedReader(new InputStreamReader(uc.getInputStream()));
while ((line = in.readLine()) != null){
page.append(line);
elapseTime = System.currentTimeMillis() - starTime;
}
} catch (SocketTimeoutException e) {
System.out.println("time out lo");
// e.printStackTrace();
return 9999; //if time out, use 9999 signal.
} catch (IOException e){
System.out.println("open connection error, connect error or inputstream error");
// e.printStackTrace();
return 9999;
}finally{
if (uc != null)
uc.disconnect();
}
// System.out.println(page);
return (float)elapseTime / 1000;
}
I'm have created an application which sends GET requests to a URL, and then downloads the full content of that page.
The client sends a GET to e.g. stackoverflow.com, and forwards the response to a parser, which has the resposibility to find all the sources from the page that needs to be downloaded with subsequent GET requests.
The method below is used to send those GET requests. It is called many times consecutively, with the URLs returned by the parser. Most of those URLs are located on the same host, and should be able to share the TCP connection.
public static void sendGetRequestToSubObject(String RecUrl)
{
URL url = new URL(recUrl.toString());
URLConnection connection = url.openConnection ();
InputStreamReader isr = new InputStreamReader(connection.getInputStream());
}
Each time this method is called, a new TCP connection is created (with a TCP 3-way handshake) and the GET is then sent on that connection. But I want to reuse the TCP connections, to improve performance.
I guess that since I create a new URL object each time the method is called, this is the way it going to work...
Maybe someone can help me do this in a better way?
Thanks!
HttpURLConnection will reuse connections if it can!
For this to work, several preconditions need to be fulfilled, mostly on the server side. Those preconditions are described in the article linked to above.
Found the problem! I was not reading the input stream properly. This caused the input stream objects to hang, and they could not be reused.
I only defined it, like this:
InputStreamReader isr = new InputStreamReader(connection.getInputStream());
but I never read from it :-)
I changed the read method as well. Instead of a buffered reader I stole this:
InputStream in = null;
String queryResult = "";
try {
URL url = new URL(archiveQuery);
HttpURLConnection urlConn = (HttpURLConnection) url.openConnection();
HttpURLConnection httpConn = (HttpURLConnection) urlConn;
httpConn.setAllowUserInteraction(false);
httpConn.connect();
in = httpConn.getInputStream();
BufferedInputStream bis = new BufferedInputStream(in);
ByteArrayBuffer baf = new ByteArrayBuffer(50);
int read = 0;
int bufSize = 512;
byte[] buffer = new byte[bufSize];
while(true){
read = bis.read(buffer);
if(read==-1){
break;
}
baf.append(buffer, 0, read);
}
queryResult = new String(baf.toByteArray());
} catch (MalformedURLException e) {
// DEBUG
Log.e("DEBUG: ", e.toString());
} catch (IOException e) {
// DEBUG
Log.e("DEBUG: ", e.toString());
}
}
From here: Reading HttpURLConnection InputStream - manual buffer or BufferedInputStream?
I'm making an http GET request. It works in about 70% of my attempts. For some reason, I sometimes get no response string from a successful connection. I just setup a button in my app which keeps firing the code below. One call might fail to reply with a string, the next call works fine:
private onButtonClick() {
try {
doit();
} catch (Exception ex) {
...
}
}
public void doit() throws Exception {
URL url = new URL("http://www.example.com/service");
HttpURLConnection connection = (HttpURLConnection)url.openConnection();
connection.setDoInput(true);
connection.setUseCaches(false);
connection.setAllowUserInteraction(false);
connection.setReadTimeout(30 * 1000);
connection.setRequestProperty("Connection", "Keep-Alive");
connection.setRequestProperty("Authorization",
"Basic " + Base64.encode("username" + ":" + "password"));
BufferedReader in = new BufferedReader(new InputStreamReader(connection.getInputStream()));
String line = "";
StringBuilder sb = new StringBuilder();
while ((line = in.readLine()) != null) {
sb.append(line);
}
in.close();
connection.disconnect();
// Every so often this prints an empty string!
System.out.println(sb.toString());
}
am I doing something wrong here? It seems like maybe I'm not closing the connection properly from the last call somehow and the response gets mangled or something? I am also calling doit() from multiple threads simultaneously, but I thought the contents of the method are thread-safe, same behavior though,
Thanks
Thanks
That method looks fine. It's reentrant, so calls shouldn't interfere with each other. It's probably a server issue, either deliberate throttling or just a bug.
EDIT: You can check the status code with getResponseCode.
For checking ResponseCode:
BufferedReader responseStream;
if (((HttpURLConnection) connection).getResponseCode() == 200) {
responseStream = new BufferedReader(new InputStreamReader(connection.getInputStream(), "UTF-8"));
} else {
responseStream = new BufferedReader(new InputStreamReader(((HttpURLConnection) connection).getErrorStream(), "UTF-8"));
}
For empty content resposneCode is 204. So if u can get empty body just add one more "if" with 204 code.
We also came across with the similar scenario, I came across the following solution for this issue:
- Setting up a user agent string on URLConnection object.
URLConnection conn = url.openConnection();
conn.setRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 4.01; Windows NT)");
more details