We Keep Coding

HttpGet sends request with the wrong encoding

I'm trying to get the text response from the following URL:
http://translate.google.cn/translate_a/single?client=t&sl=zh-CN&tl=en&dt=t&tk=265632.142896&q=%E4%BD%A0%E5%A5%BD
The response is the following:
[[["Hello there","你好",,,1]],,"zh-CN"]
(You can verify this response by entering the address into your browser.)
Here is a simplified version of my code that tries to download this text:
import org.apache.http.client.HttpClient;
import org.apache.http.client.ResponseHandler;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.BasicResponseHandler;
import org.apache.http.impl.client.DefaultHttpClient;
public class Test {
public static String downloadString() {
String url = "http://translate.google.cn/translate_a/single?client=t&sl=zh-CN&tl=en&dt=t&tk=265632.142896&q=%E4%BD%A0%E5%A5%BD";
HttpClient client = new DefaultHttpClient();
HttpGet request = new HttpGet(url);
ResponseHandler<String> handler = new BasicResponseHandler();
try {
return client.execute(request, handler);
} catch (Exception e) {
return "GET request failed.";
}
}
}
When I call Test.downloadString(), I get the following (incorrect) response:
[[["Huan Chai Sunsolt","浣犲ソ",,,0]],,"zh-CN"]
I'm guessing that there is some sort of encoding problem behind the scenes somewhere in the request process (there are six bytes that should be interpreted as two Chinese characters, but are instead interpreted as three Japanese characters), but I can't seem to pinpoint the exact cause. What am I doing wrong in my code?

It's strange, but adding the User-Agent header fixed the problem:
request.addHeader("User-Agent", "Mozilla/5.0 (X11; Linux x86_64; rv:33.0) Gecko/20100101 Firefox/33.0");

Android 6.0 release removes support for the Apache HTTP client. If your app is using this client and targets Android 2.3 (API level 9) or higher, use the HttpURLConnection class instead.
here: http://developer.android.com/about/versions/marshmallow/android-6.0-changes.html#behavior-apache-http-client

Reading over HTTP & (hopefully) automating actions using Java

Is there a way to make get & put calls over HTTP in java ? I also need to automate any user inputs like a button click on the target web-page(any web-page, not just yahoo finance)
I tried using the apache commons library & couldn't quite crack it:
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;
public class Fin {
/**
* #param args
*/
public static void main(String[] args) {
DefaultHttpClient httpclient = new DefaultHttpClient();
HttpGet httpget = new HttpGet("http://finance.yahoo.com");
try {
HttpResponse response = httpclient.execute(httpget);
HttpEntity entity = response.getEntity();
} catch (Exception e) {
e.printStackTrace();
} finally {
httpget.releaseConnection();
}
}
}
I keep getting 'java.net.ConnectException: Connection refused', though i can see it in the browser.

If you really want to automate browser-based interactions, you could go further and use Watij, which runs a browser via the JVM and is driven via a browser-based API (I.e. you identify the button you want to press and it will actually do this)
Otherwise a library like the one you've identified will normally work. You have to watch out for client-side JavaScript interactions driving the requests, and configure proxies etc (I suspect this is your problem in the above)

Server returned HTTP response code: 403 for URL: Iam displaying webpage on JEditorPane

Iam trying to display webpage on JEditorPane, but getting error at
JEditorPane editor = new JEditorPane(url);
Below is code which i workout.
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.Authenticator;
import java.net.HttpURLConnection;
import java.net.InetSocketAddress;
import java.net.PasswordAuthentication;
import java.net.Proxy;
import java.net.URL;
import java.net.URLConnection;
import javax.swing.JEditorPane;
import javax.swing.JFrame;
import javax.swing.JScrollPane;
import com.sun.org.apache.xml.internal.security.utils.Base64;
public class webpageDisplay {
/**
* #param args
* #throws IOException
*/
static class MyAuthenticator extends Authenticator {
public PasswordAuthentication getPasswordAuthentication() {
// I haven't checked getRequestingScheme() here, since for NTLM
// and Negotiate, the usrname and password are all the same.
System.err.println("Feeding username and password for " + getRequestingScheme());
return (new PasswordAuthentication("UserId","Password".toCharArray()));
}
}
public static void main(String[] args) throws IOException {
System.getProperties().put( "proxySet", "true" );
System.setProperty("http.proxyHost", "I given proxy host");
System.setProperty("http.proxyPort", "8080");
Authenticator.setDefault(new MyAuthenticator());
URL url=new URL("http://www.google.com");
HttpURLConnection uc = (HttpURLConnection) url.openConnection ();
uc.addRequestProperty("User-Agent","Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)");
uc.connect();
JEditorPane editor = new JEditorPane(url);
editor.setEditable(false);
JScrollPane pane = new JScrollPane(editor);
JFrame f = new JFrame("HTML Demo");
f.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
f.getContentPane().add(pane);
f.setSize(800, 600);
f.setVisible(true);
}
}
This is the error iam getting
Exception in thread "main" java.io.IOException: Server returned HTTP response code: 403 for URL: http://www.google.com
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:45)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:39)
at java.lang.reflect.Constructor.newInstance(Constructor.java:515)
at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1291)
at java.security.AccessController.doPrivileged(AccessController.java:251)
at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1285)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:939)
at javax.swing.JEditorPane.getStream(JEditorPane.java:823)
at javax.swing.JEditorPane.setPage(JEditorPane.java:429)
at javax.swing.JEditorPane.<init>(JEditorPane.java:256)
at webpageDisplay.main(webpageDisplay.java:48)
Caused by: java.io.IOException: Server returned HTTP response code: 403 for URL: http://www.google.com
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1236)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:384)
at javax.swing.JEditorPane.getStream(JEditorPane.java:788)
... 3 more
Please let me know how can i resolve this issue.

Although I don't know what are you using the proxy for, as a piece of answer I would include its reference directly in the connection opening. i.e. instead of declaring it as system properties to be sure that you are effectively using it. It would give something like that:
SocketAddress proxySocketAdress= new InetSocketAddress("Proxy IP address", 8080);
Proxy proxy=new Proxy(Proxy.Type.HTTP,proxySocketAdress);
HttpURLConnection uc = (HttpURLConnection) url.openConnection(proxy);
I hope this helps,
B.F.

I was able to trick Google into thinking I was a different browser by changing the http.agent property. The 403 was immediately resolved.
You can do so by running the following line before the rest of your code:
System.setProperty("http.agent", "Mozilla/5.0");
I imagine there are other things you could set http.agent to that would work, but this worked for me so I left it alone. I formulated it from the answer to this question:
Setting user agent of a java URLConnection

See HTTP 403.
In the HTTP used on the World Wide Web, 403 Forbidden is an HTTP status code returned by a web server when a user requests a web page or media that the server does not allow them to. In other words, the server can be reached, but the server declined to allow access to the page.
(Moments later..)
URL url=new URL("http://www.google.com");
Gee, what a surprise. [ OK, that was sarcasm. ;) ]
Google is notorious for being the 'example URL' that people cannot connect to. It is largely because they do not offer up the labors of their efforts for use by 'any old application'. There was a (very restricted) Google API for about 5 minutes, but it was withdrawn long ago.
I see the code does some 'fibbing' about what it is. That is apparently not enough to fool Google. (And to be plain, I'm not about to expend effort trying to figure how to get around those protections - if Google does not want their pages served in your app, that is their business.)

Java class for embedded HTTP server in Swing app

I wish to embed a very light HTTP server in my Java Swing app which just accepts requests, performs some actions, and returns the results.
Is there a very light Java class that I can use in my app which listens on a specified port for HTTP requests and lets me handle requests?
Note, that I am not looking for a stand-alone HTTP server, just a small Java class which I can use in my app.

Since Java 6, the JDK contains a simple HTTP server implementation.
Example usage:
import java.io.IOException;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.util.Iterator;
import java.util.List;
import java.util.Set;
import java.util.concurrent.Executors;
import com.sun.net.httpserver.Headers;
import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpHandler;
import com.sun.net.httpserver.HttpServer;
public class HttpServerDemo {
public static void main(String[] args) throws IOException {
InetSocketAddress addr = new InetSocketAddress(8080);
HttpServer server = HttpServer.create(addr, 0);
server.createContext("/", new MyHandler());
server.setExecutor(Executors.newCachedThreadPool());
server.start();
System.out.println("Server is listening on port 8080" );
}
}
class MyHandler implements HttpHandler {
public void handle(HttpExchange exchange) throws IOException {
String requestMethod = exchange.getRequestMethod();
if (requestMethod.equalsIgnoreCase("GET")) {
Headers responseHeaders = exchange.getResponseHeaders();
responseHeaders.set("Content-Type", "text/plain");
exchange.sendResponseHeaders(200, 0);
OutputStream responseBody = exchange.getResponseBody();
Headers requestHeaders = exchange.getRequestHeaders();
Set<String> keySet = requestHeaders.keySet();
Iterator<String> iter = keySet.iterator();
while (iter.hasNext()) {
String key = iter.next();
List values = requestHeaders.get(key);
String s = key + " = " + values.toString() + "\n";
responseBody.write(s.getBytes());
}
responseBody.close();
}
}
}
Or you can use Jetty for that purpose. It’s quite lightweight and perfectly fits this purpose.

You can use jetty as embedded server, its fairly light weight. Other option is check this out for a simple java class to handle http requests http://java.sun.com/developer/technicalArticles/Networking/Webserver/.
Other way is in Java 6 you can use com.sun.net.httpserver.HttpServer

Sun embedded web server is useful, but com.sun.net package could be dropped without notice.
A better alternative are
http://tjws.sourceforge.net/ 100kb very small and jdk 1.6-aware
http://winstone.sourceforge.net/ bigger but a good shot
http://www.eclipse.org/jetty/ Jetty, very good in developement, support SPDY and websocket

If you're not using Java 6, then I would certainly recommend Jetty. That works very well and has a decent programming interface.

You said "very light" twice, so I think JLHTTP might be a good match for you. You can embed it as a single source file or a ~35K/50K jar file, yet it supports most functionality you'd need in an HTTP server out of the box.
Disclaimer: I'm the author. But check it out for yourself and see what you think :-)

URLConnection FileNotFoundException for non-standard HTTP port sources

I was trying to use the Apache Ant Get task to get a list of WSDLs generated by another team in our company. They have them hosted on a weblogic 9.x server on http://....com:7925/services/. I am able to get to the page through a browser, but the get task gives me a FileNotFoundException when trying to copy the page to a local file to parse. I was still able to get (using the ant task) a URL without the non-standard port 80 for HTTP.
I looked through the Ant source code, and narrowed the error down to the URLConnection. It seems as though the URLConnection doesn't recognize the data is HTTP traffic, since it isn't on the standard port, even though the protocol is specified as HTTP. I sniffed the traffic using WireShark and the page loads correctly across the wire, but still gets the FileNotFoundException.
Here's an example where you will see the error (with the URL changed to protect the innocent). The error is thrown on connection.getInputStream();
import java.io.File;
import java.io.InputStream;
import java.net.URL;
import java.net.URLConnection;
public class TestGet {
private static URL source;
public static void main(String[] args) {
doGet();
}
public static void doGet() {
try {
source = new URL("http", "test.com", 7925,
"/services/index.html");
URLConnection connection = source.openConnection();
connection.connect();
InputStream is = connection.getInputStream();
} catch (Exception e) {
System.err.println(e.toString());
}
}
}

The response to my HTTP request returned with a status code 404, which resulted in a FileNotFoundException when I called getInputStream(). I still wanted to read the response body, so I had to use a different method: HttpURLConnection#getErrorStream().
Here's a JavaDoc snippet of getErrorStream():
Returns the error stream if the
connection failed but the server sent
useful data nonetheless. The typical
example is when an HTTP server
responds with a 404, which will cause
a FileNotFoundException to be thrown
in connect, but the server sent an
HTML help page with suggestions as to
what to do.
Usage example:
public static String httpGet(String url) {
HttpURLConnection con = null;
InputStream is = null;
try {
con = (HttpURLConnection) new URL(url).openConnection();
con.connect();
//4xx: client error, 5xx: server error. See: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html.
boolean isError = con.getResponseCode() >= 400;
//In HTTP error cases, HttpURLConnection only gives you the input stream via #getErrorStream().
is = isError ? con.getErrorStream() : con.getInputStream();
String contentEncoding = con.getContentEncoding() != null ? con.getContentEncoding() : "UTF-8";
return IOUtils.toString(is, contentEncoding); //Apache Commons IO
} catch (Exception e) {
throw new IllegalStateException(e);
} finally {
//Note: Closing the InputStream manually may be unnecessary, depending on the implementation of HttpURLConnection#disconnect(). Sun/Oracle's implementation does close it for you in said method.
if (is != null) {
try {
is.close();
} catch (IOException e) {
throw new IllegalStateException(e);
}
}
if (con != null) {
con.disconnect();
}
}
}

This is an old thread, but I had a similar problem and found a solution that is not listed here.
I was receiving the page fine in the browser, but got a 404 when I tried to access it via the HttpURLConnection. The URL I was trying to access contained a port number. When I tried it without the port number I successfully got a dummy page through the HttpURLConnection. So it seemed the non-standard port was the problem.
I started thinking the access was restricted, and in a sense it was. My solution was that I needed to tell the server the User-Agent and I also specify the file types I expect. I am trying to read a .json file, so I thought the file type might be a necessary specification as well.
I added these lines and it finally worked:
httpConnection.setRequestProperty("User-Agent","Mozilla/5.0 ( compatible ) ");
httpConnection.setRequestProperty("Accept","*/*");

check the response code being returned by the server

I know this is an old thread but I found a solution not listed anywhere here.
I was trying to pull data in json format from a J2EE servlet on port 8080 but was receiving the file not found error. I was able to pull this same json data from a php server running on port 80.
It turns out that in the servlet, I needed to change doGet to doPost.
Hope this helps somebody.

You could use OkHttp:
OkHttpClient client = new OkHttpClient();
String run(String url) throws IOException {
Request request = new Request.Builder()
.url(url)
.build();
Response response = client.newCall(request).execute();
return response.body().string();
}

I've tried that locally - using the code provided - and I don't get a FileNotFoundException except when the server returns a status 404 response.
Are you sure that you're connecting to the webserver you intend to be connecting to? Is there any chance you're connecting to a different webserver? (I note that the port number in the code doesn't match the port number in the link)

I have run into a similar issue but the reason seems to be different, here is the exception trace:
java.io.FileNotFoundException: http://myhost1:8081/test/api?wait=1
at sun.reflect.GeneratedConstructorAccessor2.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1491)
at java.security.AccessController.doPrivileged(Native Method)
at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1485)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1139)
at com.doitnext.loadmonger.HttpExecution.getBody(HttpExecution.java:85)
at com.doitnext.loadmonger.HttpExecution.execute(HttpExecution.java:214)
at com.doitnext.loadmonger.ClientWorker.run(ClientWorker.java:126)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.io.FileNotFoundException: http://myhost1:8081/test/api?wait=1
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1434)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379)
at com.doitnext.loadmonger.HttpExecution.execute(HttpExecution.java:166)
... 2 more
So it would seem that just getting the response code will cause the URL connection to callGetInputStream.

I know this is an old thread but just noticed something on this one so thought I will just put it out there.
Like Jessica mentioned, this exception is thrown when using non-standard port.
It only seems to happen when using DNS though. If I use IP number I can specify the port number and everything works fine.