We Keep Coding

FileNotFoundException for URL that works in browser

I am trying to use an API from https://us.mc-api.net/ for a project and I have made this as a test.
public static void main(String[] args){
try {
URL url = new URL("http://us.mc-api.net/v3/uuid/193nonaxishsl/csv/");
BufferedReader in = new BufferedReader(new InputStreamReader(url.openStream()));
String line;
while ((line = in.readLine()) != null) {
System.out.println(line);
}
in.close();
}
catch (MalformedURLException e) {
e.printStackTrace();
}
catch (IOException e) {
e.printStackTrace();
System.out.println("I/O Error");
}
}
}
And this is giving me an IOException error but when ever I open the same page in my web browser I get
false,Unknown-Username
which is what I want to get from the code. I am new and don't really know why it is happening or why.
EDIT: StackTrace
java.io.FileNotFoundException: http://us.mc-api.net/v3/uuid/193nonaxishsl/csv/
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.net.URL.openStream(Unknown Source)
at com.theman1928.Test.Main.main(Main.java:13)

The URL is returning status code 404 and therefore the input stream (mild guess here) is not being created and therefore is null. Sort the status code and you should be OK.
Ran it with this CSV and it is fine: other csv
If the error code is important to you then you can use HttpURLConnection:
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
System.out.println("code:"+conn.getResponseCode());
In that way you can process the response code before proceeding with a quick if-then-else check.

I tried it with the Apache HTTP libraries. The API endpoint seems to return a status code of 404, hence your error. Code I used is below.
public static void main(String[] args) throws URISyntaxException, ClientProtocolException, IOException {
HttpClient httpclient = HttpClients.createDefault();
URIBuilder builder = new URIBuilder("http://us.mc-api.net/v3/uuid/193nonaxishsl/csv/");
URI uri = builder.build();
HttpGet request = new HttpGet(uri);
HttpResponse response = httpclient.execute(request);
System.out.println(response.getStatusLine().getStatusCode()); // 404
}
Switching out the http://us.mc-api.net/v3/uuid/193nonaxishsl/csv/ with www.example.com or whatever returns a status code of 200, which further proves an error with the API endpoint. You can take a look at [Apache HTTP Components] library here.

This has to do with how the wire protocols are working in comparison with the java.net classes and an actual browser. A browser is going to be much more sophisticated than the simple java.net API you are using.
If you want to get the equivalent response value in Java, then you need to use a richer HTTP API.
This code will give you the same response as the browser; however, you need to download the Apache HttpComponents jars
The code:
import java.io.BufferedReader;
import java.io.DataOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.URL;
import javax.net.ssl.HttpsURLConnection;
import org.apache.http.HttpEntity;
import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpUriRequest;
import org.apache.http.impl.client.HttpClients;
public class TestDriver
{
public static void main(String[] args)
{
try
{
String url = "http://us.mc-api.net/v3/uuid/193nonaxishsl/csv";
HttpGet httpGet = new HttpGet(url);
getResponseFromHTTPReq(httpGet, url);
}
catch (Throwable e)
{
e.printStackTrace();
}
}
private static String getResponseFromHTTPReq(HttpUriRequest httpReq, String url)
{
HttpClient httpclient = HttpClients.createDefault();
// Execute and get the response.
HttpResponse response = null;
HttpEntity entity = null;
try
{
response = httpclient.execute(httpReq);
entity = response.getEntity();
}
catch (IOException ioe)
{
throw new RuntimeException(ioe);
}
if (entity == null)
{
String errMsg = "No response entity back from " + url;
throw new RuntimeException(errMsg);
}
String returnRes = null;
InputStream is = null;
BufferedReader buf = null;
try
{
is = entity.getContent();
buf = new BufferedReader(new InputStreamReader(is, "UTF-8"));
System.out.println("Response Code : " + response.getStatusLine().getStatusCode());
StringBuilder sb = new StringBuilder();
String s = null;
while (true)
{
s = buf.readLine();
if (s == null || s.length() == 0)
{
break;
}
sb.append(s);
}
returnRes = sb.toString();
System.out.println("Response: [" + returnRes + "]");
}
catch (UnsupportedOperationException | IOException e)
{
throw new RuntimeException(e);
}
finally
{
if (buf != null)
{
try
{
buf.close();
}
catch (IOException e)
{
}
}
if (is != null)
{
try
{
is.close();
}
catch (IOException e)
{
}
}
}
return returnRes;
}
}
Outputs:
Response Code : 404
Response: [false,Unknown-Username]

HTTP Post request read the response [duplicate]

In Java, this code throws an exception when the HTTP result is 404 range:
URL url = new URL("http://stackoverflow.com/asdf404notfound");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.getInputStream(); // throws!
In my case, I happen to know that the content is 404, but I'd still like to read the body of the response anyway.
(In my actual case the response code is 403, but the body of the response explains the reason for rejection, and I'd like to display that to the user.)
How can I access the response body?

Here is the bug report (close, will not fix, not a bug).
Their advice there is to code like this:
HttpURLConnection httpConn = (HttpURLConnection)_urlConnection;
InputStream _is;
if (httpConn.getResponseCode() < HttpURLConnection.HTTP_BAD_REQUEST) {
_is = httpConn.getInputStream();
} else {
/* error from server */
_is = httpConn.getErrorStream();
}

It's the same problem I was having:
HttpUrlConnection returns FileNotFoundException if you try to read the getInputStream() from the connection.
You should instead use getErrorStream() when the status code is higher than 400.
More than this, please be careful since it's not only 200 to be the success status code, even 201, 204, etc. are often used as success statuses.
Here is an example of how I went to manage it
... connection code code code ...
// Get the response code
int statusCode = connection.getResponseCode();
InputStream is = null;
if (statusCode >= 200 && statusCode < 400) {
// Create an InputStream in order to extract the response object
is = connection.getInputStream();
}
else {
is = connection.getErrorStream();
}
... callback/response to your handler....
In this way, you'll be able to get the needed response in both success and error cases.
Hope this helps!

In .Net you have the Response property of the WebException that gives access to the stream ON an exception. So i guess this is a good way for Java,...
private InputStream dispatch(HttpURLConnection http) throws Exception {
try {
return http.getInputStream();
} catch(Exception ex) {
return http.getErrorStream();
}
}
Or an implementation i used. (Might need changes for encoding or other things. Works in current environment.)
private String dispatch(HttpURLConnection http) throws Exception {
try {
return readStream(http.getInputStream());
} catch(Exception ex) {
readAndThrowError(http);
return null; // <- never gets here, previous statement throws an error
}
}
private void readAndThrowError(HttpURLConnection http) throws Exception {
if (http.getContentLengthLong() > 0 && http.getContentType().contains("application/json")) {
String json = this.readStream(http.getErrorStream());
Object oson = this.mapper.readValue(json, Object.class);
json = this.mapper.writer().withDefaultPrettyPrinter().writeValueAsString(oson);
throw new IllegalStateException(http.getResponseCode() + " " + http.getResponseMessage() + "\n" + json);
} else {
throw new IllegalStateException(http.getResponseCode() + " " + http.getResponseMessage());
}
}
private String readStream(InputStream stream) throws Exception {
StringBuilder builder = new StringBuilder();
try (BufferedReader in = new BufferedReader(new InputStreamReader(stream))) {
String line;
while ((line = in.readLine()) != null) {
builder.append(line); // + "\r\n"(no need, json has no line breaks!)
}
in.close();
}
System.out.println("JSON: " + builder.toString());
return builder.toString();
}

I know that this doesn't answer the question directly, but instead of using the HTTP connection library provided by Sun, you might want to take a look at Commons HttpClient, which (in my opinion) has a far easier API to work with.

First check the response code and then use HttpURLConnection.getErrorStream()

InputStream is = null;
if (httpConn.getResponseCode() !=200) {
is = httpConn.getErrorStream();
} else {
/* error from server */
is = httpConn.getInputStream();
}

My running code.
HttpURLConnection httpConn = (HttpURLConnection) urlConn;
if (httpConn.getResponseCode() < HttpURLConnection.HTTP_BAD_REQUEST) {
in = new InputStreamReader(urlConn.getInputStream());
BufferedReader bufferedReader = new BufferedReader(in);
if (bufferedReader != null) {
int cp;
while ((cp = bufferedReader.read()) != -1) {
sb.append((char) cp);
}
bufferedReader.close();
}
in.close();
} else {
/* error from server */
in = new InputStreamReader(httpConn.getErrorStream());
BufferedReader bufferedReader = new BufferedReader(in);
if (bufferedReader != null) {
int cp;
while ((cp = bufferedReader.read()) != -1) {
sb.append((char) cp);
}
bufferedReader.close();
}
in.close();
}
System.out.println("sb="+sb);

How to read 404 response body in java:
Use Apache library - https://hc.apache.org/httpcomponents-client-4.5.x/httpclient/apidocs/
or
Java 11 - https://docs.oracle.com/en/java/javase/11/docs/api/java.net.http/java/net/http/HttpClient.html
Snippet given below uses Apache:
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.util.EntityUtils;
CloseableHttpClient client = HttpClients.createDefault();
CloseableHttpResponse resp = client.execute(new HttpGet(domainName + "/blablablabla.html"));
String response = EntityUtils.toString(resp.getEntity());

HttpURLConnection to Sharepoint not working on Java 7 but working for Java 5

I have a piece of code to bring data from SharePoint to Unix using HttpURLConnection. The code works fine when I compile it using Java 1.5 on the server. The data is displayed on the console. But the same doesn't work when I try with Java 1.7. The Response code returned is "401". Message is "Unauthorized access".
If it is some authentication issue then why it is not coming up in the java 1.5 compiled class. Is anybody aware of any changes in the HttpURLConnection library specific to this. I only found 1 change added in Java 1.7 for streaming length limit. Any help appreciated !!
Below is snapshot of the code:
public class RestMain {
public static void main(String[] args)
{
Authenticator.setDefault (new Authenticator() {
protected PasswordAuthentication getPasswordAuthentication() {
System.out.println(getRequestingScheme());
return new PasswordAuthentication ("Userid", "Password".toCharArray());
}
});
try{
System.out.println(RestGet("SharePoint URL"));
}
catch(Exception e)
{
System.out.println("error");
e.printStackTrace();
}
}
public static String RestGet(String urlStr) throws IOException {
URL url = new URL(urlStr);
System.out.println("URL " + url.toString());
HttpURLConnection conn =
(HttpURLConnection) url.openConnection();
System.out.println("Response Code" + conn.getResponseCode());
System.out.println("after connect");
if (conn.getResponseCode() != 200) {
throw new IOException(conn.getResponseMessage());
}
conn.getResponseMessage();
System.out.println("after Response");
// Buffer the result into a string
BufferedReader rd = new BufferedReader(
new InputStreamReader(conn.getInputStream()));
StringBuilder sb = new StringBuilder();
String line;
while ((line = rd.readLine()) != null) {
sb.append(line);
}
rd.close();
conn.disconnect();
System.out.println(sb);
return (sb.toString().substring(1,500));
}
}

Why got 405 error when post data to website by using Java?

I just try to post data to google by using the following code,but always got 405 error,can anybody tell me way?
package com.tom.labs;
import java.net.*;
import java.io.*;
public class JavaHttp {
public static void main(String[] args) throws Exception {
File data = new File("D:\\in.txt");
File result = new File("D:\\out.txt");
FileOutputStream out = new FileOutputStream(result);
OutputStreamWriter writer = new OutputStreamWriter(out);
Reader reader = new InputStreamReader(new FileInputStream(data));
postData(reader,new URL("http://google.com"),writer);//Not working
//postData(reader,new URL("http://google.com/search"),writer);//Not working
sendGetRequest("http://google.com/search", "q=Hello");//Works properly
}
public static String sendGetRequest(String endpoint,
String requestParameters) {
String result = null;
if (endpoint.startsWith("http://")) {
// Send a GET request to the servlet
try {
// Send data
String urlStr = endpoint;
if (requestParameters != null && requestParameters.length() > 0) {
urlStr += "?" + requestParameters;
}
URL url = new URL(urlStr);
URLConnection conn = url.openConnection();
// Get the response
BufferedReader rd = new BufferedReader(new InputStreamReader(
conn.getInputStream()));
StringBuffer sb = new StringBuffer();
String line;
while ((line = rd.readLine()) != null) {
sb.append(line);
}
rd.close();
result = sb.toString();
} catch (Exception e) {
e.printStackTrace();
}
}
System.out.println(result);
return result;
}
/**
* Reads data from the data reader and posts it to a server via POST
* request. data - The data you want to send endpoint - The server's address
* output - writes the server's response to output
*
* #throws Exception
*/
public static void postData(Reader data, URL endpoint, Writer output)
throws Exception {
HttpURLConnection urlc = null;
try {
urlc = (HttpURLConnection) endpoint.openConnection();
try {
urlc.setRequestMethod("POST");
} catch (ProtocolException e) {
throw new Exception(
"Shouldn't happen: HttpURLConnection doesn't support POST??",
e);
}
urlc.setDoOutput(true);
urlc.setDoInput(true);
urlc.setUseCaches(false);
urlc.setAllowUserInteraction(false);
urlc.setRequestProperty("Content-type", "text/xml; charset=UTF-8");
OutputStream out = urlc.getOutputStream();
try {
Writer writer = new OutputStreamWriter(out, "UTF-8");
pipe(data, writer);
writer.close();
} catch (IOException e) {
throw new Exception("IOException while posting data", e);
} finally {
if (out != null)
out.close();
}
InputStream in = urlc.getInputStream();
try {
Reader reader = new InputStreamReader(in);
pipe(reader, output);
reader.close();
} catch (IOException e) {
throw new Exception("IOException while reading response", e);
} finally {
if (in != null)
in.close();
}
} catch (IOException e) {
e.printStackTrace();
throw new Exception("Connection error (is server running at "
+ endpoint + " ?): " + e);
} finally {
if (urlc != null)
urlc.disconnect();
}
}
/**
* Pipes everything from the reader to the writer via a buffer
*/
private static void pipe(Reader reader, Writer writer) throws IOException {
char[] buf = new char[1024];
int read = 0;
while ((read = reader.read(buf)) >= 0) {
writer.write(buf, 0, read);
}
writer.flush();
}
}

405 means "method not allowed". For example, if you try to POST to a URL that doesn't allow POST, then the server will return a 405 status.
What are you trying to do by making a POST request to Google? I suspect that Google's home page only allows GET, HEAD, and maybe OPTIONS.
Here's the body of a POST request to Google, containing Google's explanation.
405. That’s an error.
The request method POST is inappropriate for the URL /. That’s all we know.

Read url to string in few lines of java code

I'm trying to find Java's equivalent to Groovy's:
String content = "http://www.google.com".toURL().getText();
I want to read content from a URL into string. I don't want to pollute my code with buffered streams and loops for such a simple task. I looked into apache's HttpClient but I also don't see a one or two line implementation.

Now that some time has passed since the original answer was accepted, there's a better approach:
String out = new Scanner(new URL("http://www.google.com").openStream(), "UTF-8").useDelimiter("\\A").next();
If you want a slightly fuller implementation, which is not a single line, do this:
public static String readStringFromURL(String requestURL) throws IOException
{
try (Scanner scanner = new Scanner(new URL(requestURL).openStream(),
StandardCharsets.UTF_8.toString()))
{
scanner.useDelimiter("\\A");
return scanner.hasNext() ? scanner.next() : "";
}
}

This answer refers to an older version of Java. You may want to look at ccleve's answer.
Here is the traditional way to do this:
import java.net.*;
import java.io.*;
public class URLConnectionReader {
public static String getText(String url) throws Exception {
URL website = new URL(url);
URLConnection connection = website.openConnection();
BufferedReader in = new BufferedReader(
new InputStreamReader(
connection.getInputStream()));
StringBuilder response = new StringBuilder();
String inputLine;
while ((inputLine = in.readLine()) != null)
response.append(inputLine);
in.close();
return response.toString();
}
public static void main(String[] args) throws Exception {
String content = URLConnectionReader.getText(args[0]);
System.out.println(content);
}
}
As #extraneon has suggested, ioutils allows you to do this in a very eloquent way that's still in the Java spirit:
InputStream in = new URL( "http://jakarta.apache.org" ).openStream();
try {
System.out.println( IOUtils.toString( in ) );
} finally {
IOUtils.closeQuietly(in);
}

Or just use Apache Commons IOUtils.toString(URL url), or the variant that also accepts an encoding parameter.

There's an even better way as of Java 9:
URL u = new URL("http://www.example.com/");
try (InputStream in = u.openStream()) {
return new String(in.readAllBytes(), StandardCharsets.UTF_8);
}
Like the original groovy example, this assumes that the content is UTF-8 encoded. (If you need something more clever than that, you need to create a URLConnection and use it to figure out the encoding.)

Now that more time has passed, here's a way to do it in Java 8:
URLConnection conn = url.openConnection();
try (BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream(), StandardCharsets.UTF_8))) {
pageText = reader.lines().collect(Collectors.joining("\n"));
}

Additional example using Guava:
URL xmlData = ...
String data = Resources.toString(xmlData, Charsets.UTF_8);

Java 11+:
URI uri = URI.create("http://www.google.com");
HttpRequest request = HttpRequest.newBuilder(uri).build();
String content = HttpClient.newHttpClient().send(request, BodyHandlers.ofString()).body();

If you have the input stream (see Joe's answer) also consider ioutils.toString( inputstream ).
http://commons.apache.org/io/api-1.4/org/apache/commons/io/IOUtils.html#toString(java.io.InputStream)

The following works with Java 7/8, secure urls, and shows how to add a cookie to your request as well. Note this is mostly a direct copy of this other great answer on this page, but added the cookie example, and clarification in that it works with secure urls as well ;-)
If you need to connect to a server with an invalid certificate or self signed certificate, this will throw security errors unless you import the certificate. If you need this functionality, you could consider the approach detailed in this answer to this related question on StackOverflow.
Example
String result = getUrlAsString("https://www.google.com");
System.out.println(result);
outputs
<!doctype html><html itemscope="" .... etc
Code
import java.net.URL;
import java.net.URLConnection;
import java.io.BufferedReader;
import java.io.InputStreamReader;
public static String getUrlAsString(String url)
{
try
{
URL urlObj = new URL(url);
URLConnection con = urlObj.openConnection();
con.setDoOutput(true); // we want the response
con.setRequestProperty("Cookie", "myCookie=test123");
con.connect();
BufferedReader in = new BufferedReader(new InputStreamReader(con.getInputStream()));
StringBuilder response = new StringBuilder();
String inputLine;
String newLine = System.getProperty("line.separator");
while ((inputLine = in.readLine()) != null)
{
response.append(inputLine + newLine);
}
in.close();
return response.toString();
}
catch (Exception e)
{
throw new RuntimeException(e);
}
}

Here's Jeanne's lovely answer, but wrapped in a tidy function for muppets like me:
private static String getUrl(String aUrl) throws MalformedURLException, IOException
{
String urlData = "";
URL urlObj = new URL(aUrl);
URLConnection conn = urlObj.openConnection();
try (BufferedReader reader = new BufferedReader(new InputStreamReader(conn.getInputStream(), StandardCharsets.UTF_8)))
{
urlData = reader.lines().collect(Collectors.joining("\n"));
}
return urlData;
}

URL to String in pure Java
Example call to get payload from http get call
String str = getStringFromUrl("YourUrl");
Implementation
You can use the method described in this answer, on How to read URL to an InputStream and combine it with this answer on How to read InputStream to String.
The outcome will be something like
public String getStringFromUrl(URL url) throws IOException {
return inputStreamToString(urlToInputStream(url,null));
}
public String inputStreamToString(InputStream inputStream) throws IOException {
try(ByteArrayOutputStream result = new ByteArrayOutputStream()) {
byte[] buffer = new byte[1024];
int length;
while ((length = inputStream.read(buffer)) != -1) {
result.write(buffer, 0, length);
}
return result.toString(UTF_8);
}
}
private InputStream urlToInputStream(URL url, Map<String, String> args) {
HttpURLConnection con = null;
InputStream inputStream = null;
try {
con = (HttpURLConnection) url.openConnection();
con.setConnectTimeout(15000);
con.setReadTimeout(15000);
if (args != null) {
for (Entry<String, String> e : args.entrySet()) {
con.setRequestProperty(e.getKey(), e.getValue());
}
}
con.connect();
int responseCode = con.getResponseCode();
/* By default the connection will follow redirects. The following
* block is only entered if the implementation of HttpURLConnection
* does not perform the redirect. The exact behavior depends to
* the actual implementation (e.g. sun.net).
* !!! Attention: This block allows the connection to
* switch protocols (e.g. HTTP to HTTPS), which is <b>not</b>
* default behavior. See: https://stackoverflow.com/questions/1884230
* for more info!!!
*/
if (responseCode < 400 && responseCode > 299) {
String redirectUrl = con.getHeaderField("Location");
try {
URL newUrl = new URL(redirectUrl);
return urlToInputStream(newUrl, args);
} catch (MalformedURLException e) {
URL newUrl = new URL(url.getProtocol() + "://" + url.getHost() + redirectUrl);
return urlToInputStream(newUrl, args);
}
}
/*!!!!!*/
inputStream = con.getInputStream();
return inputStream;
} catch (Exception e) {
throw new RuntimeException(e);
}
}
Pros
It is pure java
It can be easily enhanced by adding different headers as a map (instead of passing a null object, like the example above does), authentication, etc.
Handling of protocol switches is supported