I have a rest http URL from which I have to extract the XML response. When I browse the URL using a browser, it returns html content. My code also sees the same html content instead of XML content.
Is there a way to get the XML content instead of html content? In the below code, I am getting only the html response. But if I check with postman plugin in chrome it shows a nice XML response. How do I get the same response using my code.
public static void sendURL(String urlValue)throws Exception{
URL oracle = new URL("https://whois.arin.net/rest/asn/AS2639");
URLConnection yc = oracle.openConnection();
yc.setRequestProperty("content-type", "application/xml");
BufferedReader in = new BufferedReader(new InputStreamReader(
yc.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null)
System.out.println(inputLine);
in.close();
}
Try to replace this:
yc.setRequestProperty("content-type", "application/xml");
with this:
yc.setRequestProperty("Accept", "application/xml");
Indeed the main purpose is totally different, Content-Type describes what you have in the body of your request while Accept indicates to the server what kind of content the client can manage which is what you want to do.
Content-Type:
The MIME type of the body of the request (used with POST and PUT
requests)
Accept:
Content-Types that are acceptable for the response.
So you already have a stream. What you need to do next is to pass that stream to a library that can decode and parse XML. Try https://docs.oracle.com/javase/7/docs/api/javax/xml/parsers/DocumentBuilder.html#parse(org.xml.sax.InputSource)
UPDATE
Sorry, your initial question was not very clear. If your java invocation of the HTTP request is yielding HTML and the one you want is an XML response, there must be some difference between the HTTP requests you make through the browser and through Java. You can use a tool like TCPMON to sit between your backend and your Java program to capture the raw HTTP request and then compare that with the one you make through the browser.
Since HTTP is a request/response pair, equivalent HTTP requests should always send back the same response.
I found the answer. Updated code. We just need to accept only the xml response.
public static void sendURL(String urlValue)throws Exception{
URL oracle = new URL("https://whois.arin.net/rest/asn/AS2639");
URLConnection yc = oracle.openConnection();
yc.setRequestProperty("accept", "application/xml");
BufferedReader in = new BufferedReader(new InputStreamReader(
yc.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null)
System.out.println(inputLine);
in.close();
}
Related
I have a utility used for integrating data and have run into an issue when special characters are used such as "Ã". Below is the method in question where the issue comes in. The response is from an API and is in xml format.
protected String getStringHttpContent(URI url, Map<String,String> headerParameters) throws IOException
{
HttpGet request = new HttpGet(url);
for(String parameter : headerParameters.keySet())
request.setHeader(parameter, headerParameters.get(parameter));
CloseableHttpResponse response = getClient().execute(request);
dumpHeaders(response);
BufferedReader br = new BufferedReader(new InputStreamReader(response.getEntity().getContent(), "UTF-8"));
StringBuffer sb = new StringBuffer();
String output;
while ((output = br.readLine()) != null) {
sb.append(output);
}
response.close();
return sb.toString();
}
The result of njÃmientill in the response string is njämientill. I've tried changing the encoding, but result remains the same. Any advice would be appreciated.
Make sure that you are using UTF-8 encoding end-to-end (through the whole chain). This includes you web pages and user input if it comes from a html form (for example), setting UTF-8 on pages, web services (web.xml, sun-web.xml or so). Also Inbound HttpRequest should include the header attribute "charset", eg. "Content-Type: text/html; charset=utf-8 ". The way your configure server-side and client-side depends on the technologies you use (which I don't know).
EDIT: regarding your comment, even if you are the client you should set the content-type to define which type of content you expect from the server (as this one may be able to serve different contents at the same URL).
Please try configure your HttpGet with:
request.setHeader(HttpHeaders.CONTENT_TYPE, "application/xml; charset=utf-8");
or (if the server is quite old):
request.setHeader(HttpHeaders.CONTENT_TYPE, "text/xml; charset=utf-8");
Better, maybe specify the accept header together with the accepted charset:
request.setHeader("Accept-Charset", "utf-8");
request.setHeader("Accept", "application/xml");
If none of these works I suggest you show your Postman query here or do a Wireshark capture to see the actual request and response, plus also list the content of the headerParameters map. Otherwise we cannot help you more (as the rest of your code looks good, to my opinion).
While trying to update a database parameters using a HTTP POST request I get the following:
Exception in thread "main" java.io.IOException: Server returned HTTP response code: 405 for URL:
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1840)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
at istsos.IstSOS.ExecuteRequest(IstSOS.java:210)
at istsos.Database.setDatabase(Database.java:77)
at istsos.Database.main(Database.java:105)
I removed the URL.
In any case, I checked with the REST responses and it works fine while in the application.
In the IstSOS class, the ExecuteRequest method it seems to have an issue with getting the response:
httpCon.setDoOutput(true);
httpCon.setRequestMethod(httpMethod);
DataOutputStream wr = new DataOutputStream(httpCon.getOutputStream());
wr.write(postDataBytes);
wr.flush();
wr.close();
BufferedReader in = new BufferedReader(new InputStreamReader(httpCon.getInputStream()));
//assigning response to JSON
String inputLine;
JSONObject jsonResponse = new JSONObject();
while((inputLine = in.readLine()) != null)
//System.out.println(inputLine);
jsonResponse = new JSONObject(inputLine);
Basically in the end, I am returning a JSONObject using JSON in Java library for parsing the response.
So far, I've tried using try/catch and sort of System.out.print to see what's wrong but this time I can't really see what's going on.
Also, this sort of request is possible, the server doesn't restrict it as far as I know.
In the end, I solved the issue. It was quite obvious, plain obvious that I couldn't update the DB through this method...
I am sending commands to a server using http, and I currently need to parse a response that the server sends back (I am sending the command via the command line, and the servers response appears in my browser).
There are a lot of resources such as this: Saving a web page to a file in Java, that clearly illustrate how to scrape a page such as cnn.com. However, since this is a response page that is only generated when the camera receives a specific command, my attempts to use the method described by Mike Deck (in the link above) have met with failure. (Specifically, when my program requests the page again the server returns a 401 error.)
The response from the server opens a new tab in my browser. Essentially, I need to know how to save the current web page using java, since reading in a file is probably the most simple way to approach this. Do any of you know how to do this?
TL;DR How do you save the current webpage to a webpage.html or webpage.txt file using java?
EDIT: I used Base64 from the Apache commons codec, which solved my 401 authentication issue. However, I am still getting a 400 error when I attempt to connect my InputStream (see below). Does this mean a connection isn't being established in the first place?
URL url = new URL ("http://"+ipAddress+"/axis-cgi/record/record.cgi?diskid=SD_DISK");
byte[] encodedBytes = Base64.encodeBase64("root:pass".getBytes());
String encoding = new String (encodedBytes);
HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setRequestMethod("POST");
connection.setDoInput (true);
connection.setRequestProperty ("Authorization", "Basic " + encoding);
connection.connect();
InputStream content = (InputStream)connection.getInputStream();
BufferedReader in = new BufferedReader (new InputStreamReader (content));
String line;
while ((line = in.readLine()) != null) {
System.out.println(line);
}
EDIT 2: Changing the request to a GET resolved the issue.
So while scrutinizing my code above, I decided to change
connection.setRequestMethod("POST");
to
connection.setRequestMethod("GET");
This solved my problem. In hindsight, I think the server was not recognizing the HTTP because it is not set up to handle the various trappings that come along with post.
I need to write a simple java function that takes a URL and processes the response which is in JavaScript, I tried using HttpUrlConnection, but it could not. Is there any java library for handling javascript response?
thanks.
EDIT: My code:
Url url = new url("https://login.live.com/oauth20_authorize.srf");
HttpURLConnection con = (HttpURLConnection) url.openConnection();
BufferedReader reader = new BufferedReader( new InputStreamReader(url.openStream()));
while(reader.readLine()!=null){
System.out.println(reader.readLine());
Response:
<html dir="..... Windows Live ID requires JavaScript to sign in. This web browser either does not support JavaScript, or scripts are being blocked......<body onload="evt_LoginHostMobile_onload(event);">
But I want to read those javascript response. Is it possible in java?
I found the way, HtmlUnit does this, it can handle javascript response
Thanks all those negative raters .....
I am trying to use core Java to read HTTP request data from an inputstream, using the following code:
BufferedReader in = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null)
System.out.println(inputLine);
in.close();
I receive the header fine, but then the client just hangs forever because the server never finds "EOF" of the request. How do I handle this? I've seen this question asked quite a bit, and most solutions involve something like the above, however it's not working for me. I've tried using both curl and a web browser as the client, just sending a get request
Thanks for any ideas
An HTTP request ends with a blank line (optionally followed by request data such as form data or a file upload), not an EOF. You want something like this:
BufferedReader in = new BufferedReader(new InputStreamReader(clientSocket.getInputStream()));
String inputLine;
while (!(inputLine = in.readLine()).equals(""))
System.out.println(inputLine);
in.close();
In addition to the answer above (as I am not able to post comments yet), I'd like to add that some browsers like Opera (I guess it was what did it, or it was my ssl setup, I don't know) send an EOF. Even if not the case, you would like to prevent that in order for your server not to crash because of a NullPointerException.
To avoid that, just add the null test to your condition, like this:
while ((inputLine = in.readLine()) != null && !inputLine.equals(""));