I have a home grown protocol which uses HttpURLConnection (from Java 1.6) & Jetty (6.1.26) to POST a block of xml as a request and receive a block of xml as a response. The amounts of xml are approx. 5KB.
When running both sender and receiver on Linux EC2 instances in different parts of the world I'm finding that in about 0.04% of my requests the Jetty handler sees the xml request (the post body) as an empty string. I've checked and the client outputs that it's consistently trying to send the correct (> 0 length) xml request string.
I have also reproduced this by looping my JUnit tests on my local (Win 8) box.
I assume the error must be something like:
Misuse of buffers
An HttpURLConnection bug
A network error
A Jetty bug
A random head slapping stupid thing I've done in the code
The relevant code is below:
CLIENT
connection = (HttpURLConnection) (new URL (url)).openConnection();
connection.setReadTimeout(readTimeoutMS);
connection.setConnectTimeout(connectTimeoutMS);
connection.setRequestMethod("POST");
connection.setAllowUserInteraction(false);
connection.setDoOutput(true);
// Send request
byte[] postBytes = requestXML.getBytes("UTF-8");
connection.setRequestProperty("Content-length", "" + postBytes.length);
OutputStream os = connection.getOutputStream();
os.write(postBytes);
os.flush();
os.close();
// Read response
InputStream is = connection.getInputStream();
StringWriter writer = new StringWriter();
IOUtils.copy(is, writer, "UTF-8");
is.close();
connection.disconnect();
return writer.toString();
SERVER (Jetty handler)
public void handle(java.lang.String target, javax.servlet.http.HttpServletRequest request, javax.servlet.http.HttpServletResponse response, int dispatch) {
InputStream is = request.getInputStream();
StringWriter writer = new StringWriter();
IOUtils.copy(is, writer, "UTF-8");
is.close();
String requestXML = writer.toString();
// requestXML is 0 length string about 0.04% of time
Can anyone think of why I'd randomly get the request as an empty string?
Thanks!
EDIT
I introduced some more trace and getContentLength() returns -1 when the error occurs, but the client output still shows it's sending the right amount of bytes.
I can't think of why you are getting a empty string. Code looks correct. If you update you code to check for empty string and if found report the content-length and transfer-encoding of the request, that would be helpful to identify the culprit. A wireshark trace of the network data would also be good.
But the bad new is that jetty-6 is really end of life, and we are unlikely to be updating it. If you are writing the code today, then you really should be using jetty-7 or 8. Perhaps even jetty-9 milestone release if you are brave. If you find such and error in jetty-9, I'd be all over it like a rash trying to fix it for you!
Make sure you set connection.setRequestProperty("Content-Type", "application/xml"); It's possible POST data may be discarded without some Content-type. This was the case when I replicated your problem locally (against a Grails embedded Tomcat instance), and supplying this fixed it.
Related
I am sending commands to a server using http, and I currently need to parse a response that the server sends back (I am sending the command via the command line, and the servers response appears in my browser).
There are a lot of resources such as this: Saving a web page to a file in Java, that clearly illustrate how to scrape a page such as cnn.com. However, since this is a response page that is only generated when the camera receives a specific command, my attempts to use the method described by Mike Deck (in the link above) have met with failure. (Specifically, when my program requests the page again the server returns a 401 error.)
The response from the server opens a new tab in my browser. Essentially, I need to know how to save the current web page using java, since reading in a file is probably the most simple way to approach this. Do any of you know how to do this?
TL;DR How do you save the current webpage to a webpage.html or webpage.txt file using java?
EDIT: I used Base64 from the Apache commons codec, which solved my 401 authentication issue. However, I am still getting a 400 error when I attempt to connect my InputStream (see below). Does this mean a connection isn't being established in the first place?
URL url = new URL ("http://"+ipAddress+"/axis-cgi/record/record.cgi?diskid=SD_DISK");
byte[] encodedBytes = Base64.encodeBase64("root:pass".getBytes());
String encoding = new String (encodedBytes);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("POST");
connection.setDoInput (true);
connection.setRequestProperty ("Authorization", "Basic " + encoding);
connection.connect();
InputStream content = (InputStream)connection.getInputStream();
BufferedReader in = new BufferedReader (new InputStreamReader (content));
String line;
while ((line = in.readLine()) != null) {
System.out.println(line);
}
EDIT 2: Changing the request to a GET resolved the issue.
So while scrutinizing my code above, I decided to change
connection.setRequestMethod("POST");
to
connection.setRequestMethod("GET");
This solved my problem. In hindsight, I think the server was not recognizing the HTTP because it is not set up to handle the various trappings that come along with post.
Iam creating automated system to post the data to the form for registering into the web site
URL url = new URL("https://www.walmart.com/subflow/YourAccountLoginContext/1471476370/sub_generic_login/create_account.do");
String postData = "firstName="+xlsDataList.get(0)+"&lastName="+xlsDataList.get(1)+"&userName="+xlsDataList.get(2)+"&userNameConfirm="+xlsDataList.get(3)+"&pwd="+xlsDataList.get(5)+"&pwdConfirm="+xlsDataList.get(6);
HttpsURLConnection uc = (HttpsURLConnection) url.openConnection();
uc.setDoInput(true);
uc.setDoOutput(true);
uc.setRequestMethod("POST");
uc.setRequestProperty("Accept", "*/*");
uc.setRequestProperty("Content-Length", Integer.toString(postData.getBytes().length));
uc.setRequestProperty("Content-Type", "text/html; charset=utf-8");
OutputStreamWriter outputWriter = new OutputStreamWriter(uc.getOutputStream());
outputWriter.write(postData);
outputWriter.flush();
outputWriter.close();
I thought that those above postdata are just request attributes , and coded accordingly. But after closely checking the view source, i came to know that those are form attributes.
I dnt have access to that form. Now how can i post the data to the form, so that the user get registered by the site?
i have to set the values to formbean.
Please provide your suggesions.
Your are using the wrong Content-Type in your POST: you need to use application/x-www-form-urlencoded. Once you change that, the server will interpret your request body as request parameters and (likely) your "formbean" will be filled with the data.
The above code may be a test case, but you really ought to take care to properly encode all of your data that you are trying to POST. Otherwise, you run the risk of either having a syntactically invalid request (in which case, the server will either reject the request, or ignore important parameters) or introducing a security vulnerability where a user can inject arbitrary request parameters into your POST. I highly recommend code that looks like this:
import java.net.URLEncoder;
String charset = "UTF-8"; // Change this if you want some other encoding
StringBuilder postData = new StringBuilder();
postData.append(URLEncoder.encode("firstName", charset));
postData.append("=");
postData.append(URLEncoder.encode(xlsDataList.get(0)), charset);
postData.append("&");
postData.append(URLEncoder.encode("lastName", charset));
postData.append("=");
postData.append(URLEncoder.encode(xlsDataList.get(1), charset));
postData.append("&");
postData.append(URLEncoder.encode("userName", charset));
postData.append("=");
postData.append(URLEncoder.encode(xlsDataList.get(2), charset));
postData.append("&");
postData.append(URLEncoder.encode("userNameConfirm", charset));
postData.append("=");
postData.append(URLEncoder.encode(xlsDataList.get(3), charset));
postData.append("&");
postData.append(URLEncoder.encode("pwd", charset));
postData.append("=");
postData.append(URLEncoder.encode(xlsDataList.get(5), charset));
postData.append("&");
postData.append(URLEncoder.encode("pwdConfirm", charset));
postData.append("=");
postData.append(xlsDataList.get(6), charset));
It seems silly to encode the static strings like "userNameConfirm", but if you get into that habit, you'll end up using it all the time and your code will be a lot safer.
Also, you need to make sure that the data you send through the OutputStream has the right Content-Length: you are computing the content-length properly, but then you aren't using the bytes you used for the computation to send to the client. You want your code to look more like this:
byte[] postDataBytes = postData.getBytes(charset);
uc.setRequestProperty("Content-Length", Integer.toString(postDataBytes.length));
uc.setRequestProperty("Content-Type", "application/x-www-form-urlencoded");
OutputStream outputStream = uc.getOutputStream();
outputStream.write(postDataBytes);
outputStream.flush();
outputStream.close();
You can find a very comprehensive HTTPUrlConnection tutorial in the community wiki: Using java.net.URLConnection to fire and handle HTTP requests
I recommend to use Apache HttpClient. its faster and easier to implement.
PostMethod post = new PostMethod("https://www.walmart.com/subflow/YourAccountLoginContext/1471476370/sub_generic_login/create_account.do");
NameValuePair[] data = {
new NameValuePair("firstName", "joe"),
new NameValuePair("lastName", "bloggs")
};
post.setRequestBody(data);
InputStream in = post.getResponseBodyAsStream();
// handle response.
For details you can refer http://hc.apache.org/
If your project uses Spring 3.x or later I would recommend using the Spring RestTemplate its pretty handy for doing http, code below will log do a form post.
public String login(String username, String password)
{
MultiValueMap<String, String> form = new LinkedMultiValueMap<>();
form.add(usernameInputFieldName, username);
form.add(passwordInputFieldName, password);
RestTemplate template = new RestTemplate();
URI location = template.postForLocation(loginUrl(), form);
return location.toString();
}
The "HTTP Error 500" you've described in your comment is an "Internal server error".
This means that the server either can't use your request (GET/POST) or there's a problem specific to the server you are trying to call.
Taking a look at the URL you're calling, I immediately the same Error 500.
Same happens for both GET and POST requests at httqs://www.walmart.com/subflow/YourAccountLoginContext/1471476370/sub_generic_login/create_account.do (Live link deactivated; replace "q" with "p" to make it work.)
In short: the generally returned "HTTP Error 500" from WallMart's servers prevents your call to succeed.
By the way:
It's not uncommon to get an error 500 instead of a 403 if they are locking your access down.
As you probably don't own the WallMart website and since you're trying to access levels of their websites that are worth to be protected from 3rd party acces, this might well be the case. ;)
PS: I'm not sure if it's wise to show the AccountLogin number in public like this. After all, it's the client ID of a specific WallMart account holder. But hey, that's your choice, not mine.
Also, double check the parameters you are sending. There may be some validations on input data the server is doing. Eg, some fields are mandatory, some are numbers only, etc.
Try spoofing as a browser by modifying the User Agent. WalMart may have a security mechanism that detects that you are doing this in an automated way.
(If you have problems setting the user agent see this post: Setting user agent of a java URLConnection)
String url = "http://maps.googleapis.com/maps/api/directions/xml?origin=Chicago,IL&destination=Los+Angeles,CA&waypoints=Joplin,MO|Oklahoma+City,OK&sensor=false";
URL google = new URL(url);
HttpURLConnection con = (HttpURLConnection) google.openConnection();
and I use BufferedReader to print the content I get 403 error
The same URL works fine in the browser. Could any one suggest.
The reason it works in a browser but not in java code is that the browser adds some HTTP headers which you lack in your Java code, and the server requires those headers. I've been in the same situation - and the URL worked both in Chrome and the Chrome plugin "Simple REST Client", yet didn't work in Java. Adding this line before the getInputStream() solved the problem:
connection.addRequestProperty("User-Agent", "Mozilla/4.0");
..even though I have never used Mozilla. Your situation might require a different header. It might be related to cookies ... I was getting text in the error stream advising me to enable cookies.
Note that you might get more information by looking at the error text. Here's my code:
try {
HttpURLConnection connection = ((HttpURLConnection)url.openConnection());
connection.addRequestProperty("User-Agent", "Mozilla/4.0");
InputStream input;
if (connection.getResponseCode() == 200) // this must be called before 'getErrorStream()' works
input = connection.getInputStream();
else input = connection.getErrorStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(input));
String msg;
while ((msg =reader.readLine()) != null)
System.out.println(msg);
} catch (IOException e) {
System.err.println(e);
}
HTTP 403 is a Forbidden status code. You would have to read the HttpURLConnection.getErrorStream() to see the response from the server (which can tell you why you have been given a HTTP 403), if any.
This code should work fine. If you have been making a number of requests, it is possible that Google is just throttling you. I have seen Google do this before. You can try using a proxy to verify.
Most browsers automatically encode URLs when you enter them, but the Java URL function doesn't.
You should Encode the URL with URLEncoder URL Encoder
I know this is a bit late, but the easiest way to get the contents of a URL is to use the Apache HttpComponents HttpClient project: http://hc.apache.org/httpcomponents-client-ga/index.html
you original page (with link) and the targeted linked page are not the same domain.
original-domain and target-domain.
I found the difference is in request header:
with 403 forbidden error,
request header have one line:
Referer: http://original-domain/json2tree/ipfs/ipfsList.html
when I enter url, no 403 forbidden,
the request header does NOT have above line referer: original-domain
I finally figure out how to fix this error!!!
on your original-domain web page, you have to add
<meta name="referrer" content="no-referrer" />
it will remove or prevent sending the Referer in header, works both for links and for Ajax requests made
I have started a small project in Java.
I have to create a client which will send xml to a url as a HTTP POST request.
I try it using java.net.* package (Following is the piece of code) but I am getting error as follows:
java.io.IOException: Server returned HTTP response code: 500 for URL: "target url"
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
at newExample.main(newExample.java:36)
My code is as follows:
try {
URL url = new URL("target url");
URLConnection connection = url.openConnection();
if( connection instanceof HttpURLConnection )
((HttpURLConnection)connection).setRequestMethod("POST");
connection.setRequestProperty("Content-Length", Integer.toString(requestXml.length()) );
connection.setRequestProperty("Content-Type","text/xml; charset:ISO-8859-1;");
connection.setDoOutput(true);
connection.connect();
// Create a writer to the url
PrintWriter writer = new PrintWriter(new
OutputStreamWriter(connection.getOutputStream()));
// Get a reader from the url
BufferedReader reader = new BufferedReader(new
InputStreamReader(connection.getInputStream()));
writer.println();
writer.println(requestXml);
writer.println();
writer.flush();
String line = reader.readLine();
while( line != null ) {
System.out.println( line );
line = reader.readLine();
}
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
Please help with suitable examples or any other ways of doing this.
Point errors/mistakes in above code or other possibilities.
My Web Service is in spring framework
xml to send is in the string format: requestXml
The problem lies in below code
// Get a reader from the url
BufferedReader reader = new BufferedReader(new
InputStreamReader(connection.getInputStream()));
As the service might not always return you the proper response... as you are calling a service through http, it can be possible that the server itself is not available or the service is not available. So you should always check for the response code before reading response from streams, based on the response code you've to decide whether to read it from inputStream for success response or from errorStream for failure or exception condition.
BufferedReader reader = null;
if(connection.getResponseCode() == 200)
{
reader = new BufferedReader(new
InputStreamReader(connection.getInputStream()));
}
else
{
reader = new BufferedReader(new
InputStreamReader(connection.getErrorStream()));
}
This would resolve the problem
The problem is inside your server code or the server configuration:
10.5.1 500 Internal Server Error
The server encountered an unexpected condition which prevented it from fulfilling the request.
(w3c.org/Protocols)
If the server is under your control (should be, if I look at the URL [before the edit]), then have a look at the server logs.
Well, you should close your streams and connections. Automatic resource maangement from Java 7 or http://projectlombok.org/ can help. However, this is probably not the main problem.
The main problem is that the server-side fails. HTTP code 500 means server-side error. I can't tell you the reason, because I don't know the server side part. Maybe you should look at the log of the server.
I think that your problem is that you are opening the input stream before you have written and closed the output stream. Certainly, the Sun Tutorial does it that way.
If you open the input stream too soon, it is possible that the output stream will be closed automatically, causing the server to see an empty POST request. This could be sufficient to cause it to get confused and send a 500 response.
Even if this is not what is causing the 500 errors, it is a good idea to do things in the order set out in the tutorial. For a start, if you accidentally read the response before you've finished writing the request, you are likely to (at least temporarily) lock up the connection. (In fact, it looks like your code is doing this because you are not closing the writer before reading from the reader.)
A separate issue is that your code does not close the connection in all circumstances, and is therefore liable to leak network connections. If it does this repeatedly, it is likely to lead to more IOExceptions.
If you are calling an External Webservice and passing a JSON in the REST call, check the datatype of the values passed.
Example:
{ "originalReference":"8535064088443985",
"modificationAmount":
{ "amount":"16.0",
"currency":"AUD"
},
"reference":"20170928113425183949",
"merchantAccount":"MOM1"
}
In this example, the value of amount was sent as a string and the webservice call failed with Server returned HTTP response code: 500.
But when the amount: 16.0 was sent, i.e an Integer was passed, the call went through. Though you have referred API documentation while calling such external APIs, small details like this could be missed.
I'am writing a HTTP proxy that is part of a test/verification
system. The proxy filters all requests coming from the client device
and directs them towards various systems under test.
The proxy is implemented as a servlet where each request is forwarded
to the target system, it handles both GET and POST. Somtimes the
response from the target system is altered to fit various test
conditions, but that is not the part of the problem.
When forwarding a request, all headers are copied except for those
that is part of the actual HTTP transfer such as Content-Length and
Connection headers.
If the request is a HTTP POST, then the entity body of the request is
forwarded as well and here is where it doesnt work sometimes.
The code reading the entity body from the servlet request is the following:
URL url = new URL(targetURL);
HttpURLConnection conn = (HttpURLConnection)url.openConnection();
String method = request.getMethod();
java.util.Enumeration headers = request.getHeaderNames();
while(headers.hasMoreElements()) {
String headerName = (String)headers.nextElement();
String headerValue = request.getHeader(headerName);
if (...) { // do various adaptive stuff based on header
}
conn.setRequestProperty(headerName, headerValue);
}
// here is the part that fails
char postBody[] = new char[1024];
int len;
if(method.equals("POST")) {
logger.debug("guiProxy, handle post, read request body");
conn.setDoOutput(true);
BufferedReader br = request.getReader();
BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(conn.getOutputStream()));
do {
logger.debug("Read request into buffer of size: " + postBody.length);
len = br.read(postBody, 0, postBody.length);
logger.debug("guiProxy, send request body, got " + len + " bytes from request");
if(len != -1) {
bw.write(postBody, 0, len);
}
} while(len != -1);
bw.close();
}
So what happends is that the first time a POST is received, -1
characters are read from the request reader, a wireshark trace shows
that the entity body containing URL encoded post parameters are there
and it is in one TCP segment so there is no network related
differences.
The second time, br.read successfully returns the 232 bytes in the
POST request entity body and every forthcoming request works as well.
The only difference between the first and forthcoming POST requests is
that in the first one, no cookies are present, but in the second one,
a cookie is present that maps to the JSESSION.
Can it be a side effect of entity body not being available since the
request processing in the servlet container allready has read the POST
parameters, but why does it work on forthcoming requests.
I believe that the solution is of course to ignore the entity body on
POST requests containing URL encoded data and fetch all parameters
from the servlet request instead using getParameter and reinsert them
int the outgoing request.
Allthough that is tricky since the POST request could contain GET
parameters, not in our application right now, but implementing it
correctly is some work.
So my question is basically: why do the reader from
request.getReader() return -1 when reading and an entity body is
present in the request, if the entity body is not available for
reading, then getReader should throw an illegal state exception. I
have also tried with InputStream using getInputStream() with the same
results.
All of this is tested on apache-tomcat-6.0.18.
So my question is basically: why do the reader from request.getReader() return -1 when reading.
It will return -1 when there is no body or when it has already been read. You cannot read it twice. Make sure that nothing before in the request/response chain has read it.
and an entity body is present in the request, if the entity body is not available for reading, then getReader should throw an illegal state exception.
It will only throw that when you have already called getInputStream() on the request before, not when it is not available.
I have also tried with InputStream using getInputStream() with the same results.
After all, I'd prefer streaming bytes than characters because you then don't need to take character encoding into account (which you aren't doing as far now, this may lead to future problems when you will get this all to work).
Seems, that moving
BufferedReader br = request.getReader()
before all operations, that read request (like request.getHeader() ), works for me well .