Problem fetching XML file in Java - java

I am trying to use Google's unofficial weather API in an Android Application.
I use this code:
//get the text from the edit text
userZip = zipCode.getText().toString();
//create a link using the zip code
//TODO sanitize input
System.out.println(userZip);
link = "http://www.google.com/ig/api?weather=" + userZip;
System.out.println(link);
//connect to the link
URL googleWeatherService = null;
try {
googleWeatherService = new URL(link);
} catch (MalformedURLException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
SAXBuilder parser = new SAXBuilder();
try {
doc = parser.build(googleWeatherService);
} catch (JDOMException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
But I get the error java.io.IOException Couldn't open http://www.google.com/ig/api?weather=08003 (just using 08003 as an example).
If you go to the link in FF you get a nice XML file of weather, so what am I doing wrong?

I think you need to open the connection with the url and get the inputstream for this to work. I would try this:
URL googleWeatherService = null;
URLConnection conn = null;
try {
googleWeatherService = new URL(link);
conn = googleWeatherService.openConnection();
} catch (MalformedURLException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
SAXBuilder parser = new SAXBuilder();
try {
doc = parser.build(conn.getInputStream());
Hopefully this does the trick for you!
Otherwise if this fails, it sounds like you're having to deal with URL redirects, which is a problem i used to have. You would need to do the following in that case:
URL googleWeatherService = null;
URLConnection conn = null;
try {
googleWeatherService = new URL(link);
HttpURLConnection ucon = (HttpURLConnection) googleWeatherService.openConnection();
ucon.setInstanceFollowRedirects(false);
URL secondURL = new URL(ucon.getHeaderField("Location"));
conn = secondURL.openConnection();
} catch (MalformedURLException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
SAXBuilder parser = new SAXBuilder();
try {
doc = parser.build(conn.getInputStream());
Hope this solves it!

This worked perfectly for me:
package weather;
import org.dom4j.Document;
import org.dom4j.io.OutputFormat;
import org.dom4j.io.SAXReader;
import org.dom4j.io.XMLWriter;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.URL;
/**
* GoogleWeather
* #author Michael
* #since 2/12/11
*/
public class GoogleWeather
{
public static void main(String[] args)
{
for (String userZip : args)
{
BufferedReader br = null;
try
{
String link = "http://www.google.com/ig/api?weather=" + userZip;
System.out.println(link);
URL googleWeatherService = new URL(link);
br = new BufferedReader(new InputStreamReader(googleWeatherService.openStream()));
SAXReader reader = new SAXReader();
Document document = reader.read(googleWeatherService);
OutputFormat format = OutputFormat.createPrettyPrint();
XMLWriter writer = new XMLWriter(System.out, format);
writer.write(document);
}
catch (Exception e)
{
e.printStackTrace();
}
finally
{
close(br);
}
}
}
private static void close(BufferedReader br)
{
try
{
if (br != null)
{
br.close();
}
}
catch (Exception e)
{
e.printStackTrace();
}
}
}
Here's the result it brought back:
<?xml version="1.0" encoding="UTF-8"?>
<xml_api_reply version="1">
<weather module_id="0" tab_id="0" mobile_row="0" mobile_zipped="1" row="0" section="0">
<forecast_information>
<city data="Hebron, CT"/>
<postal_code data="06248"/>
<latitude_e6 data=""/>
<longitude_e6 data=""/>
<forecast_date data="2011-02-12"/>
<current_date_time data="2011-02-13 03:00:47 +0000"/>
<unit_system data="US"/>
</forecast_information>
<current_conditions>
<condition data="Partly Cloudy"/>
<temp_f data="28"/>
<temp_c data="-2"/>
<humidity data="Humidity: 45%"/>
<icon data="/ig/images/weather/partly_cloudy.gif"/>
<wind_condition data="Wind: NW at 14 mph"/>
</current_conditions>
<forecast_conditions>
<day_of_week data="Sat"/>
<low data="16"/>
<high data="36"/>
<icon data="/ig/images/weather/partly_cloudy.gif"/>
<condition data="Partly Cloudy"/>
</forecast_conditions>
<forecast_conditions>
<day_of_week data="Sun"/>
<low data="30"/>
<high data="38"/>
<icon data="/ig/images/weather/snow.gif"/>
<condition data="Snow Showers"/>
</forecast_conditions>
<forecast_conditions>
<day_of_week data="Mon"/>
<low data="23"/>
<high data="46"/>
<icon data="/ig/images/weather/cloudy.gif"/>
<condition data="Cloudy"/>
</forecast_conditions>
<forecast_conditions>
<day_of_week data="Tue"/>
<low data="12"/>
<high data="29"/>
<icon data="/ig/images/weather/cloudy.gif"/>
<condition data="Windy"/>
</forecast_conditions>
</weather>
</xml_api_reply>

Are you able to retrieve other URIs successfully?
You could be hitting problems with the JVM configuration. In most environments I've come across, if your machine is configured so that the web browser can make HTTP requests successfully, then Java will also be able to make them successfully. But I've heard of special JVM configuration being needed when you're behind a proxy server, and I've no idea whether anything similar might be needed in an Android environment.

I've ran into this problem with JDOM2 and it was really a NetworkOnMainThreadException in disguise. Threw it off the main thread and everything worked.
new Thread(new Runnable() {
#Override
public void run() {
<your code>
}
}).start();

Related

java.lang.IllegalArgumentException: Unsupported element: rss

I am trying to 'GET' a rss feed.
public RssFeed(String url) {
_url = url;
String res = this.api.get(url);
ByteArrayInputStream bis = new ByteArrayInputStream(res.getBytes());
try {
bis.close();
} catch (IOException e) {
e.printStackTrace();
}
XMLDecoder decoder = new XMLDecoder(bis);
try {
Object xml = decoder.readObject();
_response = xml.toString();
} catch(Exception e) {
e.printStackTrace();
} finally {
decoder.close();
}
}
When I check what's inside of 'res'. It appears to get this entire XML.
But then, I am trying to decode it and I get:
java.lang.IllegalArgumentException: Unsupported element: rss
Can someone help me with that? I am new to Java.
Thanks!
XMLDecoder is meant to be used on elements created by XMLEncoder. Since you're scraping this XML from the web, the elements in this XML may not be valid according to these classes. Use a more generic XML parser, such as DocumentBuilder::parse() to handle this.
DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
DocumentBuilder builder = factory.newDocumentBuilder();
try {
builder.parse(url);
} catch (IOException e) {
e.printStackTrace();
} catch (SAXParseException e) {
e.printStackTrace();
} catch (IllegalArgumentException e) {
e.printStackTrace();
}

API Connections only working once

I am creating an application that makes calls to the Hitbox API. I am trying to get the game name (listed as category_name from a list.
Thus far, I have managed to get the game name one time during the programs running stage, however when I change where to get the game name from, the program doesn't do anything. I am at a loss as to what could cause it not to send another request to the server.
public void apiConnect(){
String channel = text.getText();
HttpClient client = new DefaultHttpClient();
HttpGet request = new HttpGet("http://api.hitbox.tv/media/live/" + channel);
HttpResponse response = null;
try {
response = client.execute(request);
} catch (IOException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
// Get the response
BufferedReader rd = null;
try {
rd = new BufferedReader
(new InputStreamReader(response.getEntity().getContent()));
} catch (UnsupportedOperationException | IOException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
String line = "";
try {
while ((line = rd.readLine()) != null) {
hitbox.append(line);
}
} catch (IOException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
try {
FileUtils.writeStringToFile(new File("hitbox.json"), hitbox.getText());
} catch (IOException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
String game = null;
FileInputStream fileHitbox = null;
try {
fileHitbox = new FileInputStream(new File("hitbox.json"));
} catch (FileNotFoundException e2) {
// TODO Auto-generated catch block
e2.printStackTrace();
}
String strHitbox = null;
try {
strHitbox = IOUtils.toString(fileHitbox, "UTF-8");
} catch (IOException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
JSONObject obj = new JSONObject(strHitbox);
JSONArray ar = obj.getJSONArray("livestream");
for (int i = 0; i < ar.length(); i++)
{
game = ar.getJSONObject(i).getString("category_name");
nameOf.setText("Game Name: " + game);
}
File hb = new File("hitbox.json");
if(hb.exists()){
hb.delete();
}
}
The above sample is the defined function, and the Get Game Name button code is below:
btnGetGameName.addSelectionListener(new SelectionAdapter() {
#Override
public void widgetSelected(SelectionEvent e) {
apiConnect();
}
});
Could anyone suggest what is causing it to not work after the first request, and if possible suggest a solution?
EDIT: I have found the issue. The reading of the data from the API is appended to the hitbox variable. I have thus added a snippet that clears what "hitbox" variable has when the button is pressed, thus meaning the code works without issues.
Try to consume your response after your read it to release the resource :
rd = new BufferedReader
(new InputStreamReader(response.getEntity().getContent()));
response.getEntity().consumeContent();
//Or if you have EntityUtils
EntityUtils.consume(response.getEntity());
source

Using jcifs for copying files to network drive

I was trying to implement the functionality of transferring local files to a network drive using jcifs library but upon running the program on the command line I was receiving following exception:
Exception in thread "main" java.lang.NullPointerException
jcifs.smb.ServerMessageBlock.writeString(ServerMessageBlock.java: 202)
To understand the error I tried debugging the code on eclipse and while doing so at line:
NtlmPasswordAuthentication authentication = new NtlmPasswordAuthentication("xxxxxx.xx.com",username,password);
I received an exception stating ClassNotFoundException. But I have the jcifs.jar in the build path.
A quick google search for 'ntlmpasswordauthentication' classnotfoundexception landed me on two threads with same issue but no resolution.
Please let me know how can I resolve this.
Thank you
Here is the whole function, just in case needed:
private static void TransferFiles()
{
File transfer_files = new File (sourcepath);
File[] files = transfer_files.listFiles();
String username = properties.getProperty("user");
String password = properties.getProperty("password");
String source = sourcepath;
SmbFileOutputStream outputStream = null;
FileInputStream inputStream = null;
SmbFile copyFile = null;
byte[] buffer = new byte[16*1024*1024];
int length = 0;
jcifs.smb.NtlmPasswordAuthentication authentication = new NtlmPasswordAuthentication("xxxxxxx",username,password);
try
{
copyFile = new SmbFile(destinationpath,authentication);
}
catch (MalformedURLException e)
{
e.printStackTrace();
}
try
{
outputStream = new SmbFileOutputStream(copyFile);
}
catch (SmbException | MalformedURLException | UnknownHostException e)
{
e.printStackTrace();
}
try
{
inputStream = new FileInputStream(sourceFile);
}
catch (FileNotFoundException e1)
{
e1.printStackTrace();
}
try
{
while((length = inputStream.read(buffer))>0)
{
outputStream.write(buffer, 0, length);
}
}
catch (IOException e)
{
e.printStackTrace();
}
try
{
inputStream.close();
outputStream.close();
}
catch (IOException e)
{
e.printStackTrace();
}

Can't download html

I'm trying to download this html
I'm using this code:
Document doc = null;
try {
doc =Jsoup.connect(link).userAgent("Mozilla").get();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
Log.i ("html", doc.toString());
UPDATED:
ASLO tried to use it:
HttpClient client = new DefaultHttpClient();
HttpGet request = new HttpGet(link);
HttpResponse response = null;
try {
response = client.execute(request);
} catch (ClientProtocolException e1) {
//
e1.printStackTrace();
} catch (IOException e1) {
//
e1.printStackTrace();
}
InputStream in = null;
try {
in = response.getEntity().getContent();
} catch (IllegalStateException e1) {
//
e1.printStackTrace();
} catch (IOException e1) {
//
e1.printStackTrace();
}
BufferedReader reader = null;
try {
reader = new BufferedReader(new InputStreamReader(in, "UTF-8"));
} catch (UnsupportedEncodingException e) {
//
e.printStackTrace();
}
StringBuilder str = new StringBuilder();
String line = null;
try {
while((line = reader.readLine()) != null)
{
str.append(line);
}
} catch (IOException e1) {
//
e1.printStackTrace();
}
try {
in.close();
} catch (IOException e1) {
//
e1.printStackTrace();
}
String html = str.toString();
Log.e("html", html);
again responce like this one:
<html>
<body>
<script>document.cookie="BPC=f563534535121d5a1ba5bd1e153b";
document.location.href="http://...link.../all?attempt=1";</script>
</body>
</html>
I can't find any solution... Page can not be downloaded maybe because haven't cookie ... or what?
In the script tag, you have this statement :
document.location.href="....link..../all?attempt=1";
Normally it forces the browser to reload the page with the location. I think it's the page "....link...?attempt=1" that you want to download in fact.
It is not sure that it will work anyway if you don't use the cookie defined in the script but it deserves a try.

Need help in getting HTML of a website in Java

I got some code from java httpurlconnection cutting off html and I am pretty much the same code to fetch html from websites in Java.
Except for one particular website that I am unable to make this code work with:
I am trying to get HTML from this website:
http://www.geni.com/genealogy/people/William-Jefferson-Blythe-Clinton/6000000001961474289
But I keep getting junk characters. Although it works very well with any other website like http://www.google.com.
And this is the code that I am using:
public static String PrintHTML(){
URL url = null;
try {
url = new URL("http://www.geni.com/genealogy/people/William-Jefferson-Blythe-Clinton/6000000001961474289");
} catch (MalformedURLException e1) {
// TODO Auto-generated catch block
e1.printStackTrace();
}
HttpURLConnection connection = null;
try {
connection = (HttpURLConnection) url.openConnection();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6");
try {
System.out.println(connection.getResponseCode());
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
String line;
StringBuilder builder = new StringBuilder();
BufferedReader reader = null;
try {
reader = new BufferedReader(new InputStreamReader(connection.getInputStream()));
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
try {
while ((line = reader.readLine()) != null) {
builder.append(line);
builder.append("\n");
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
String html = builder.toString();
System.out.println("HTML " + html);
return html;
}
I don't understand why it doesn't work with the URL that I mentioned above.
Any help will be appreciated.
That site is incorrectly gzipping the response regardless of the client's capabilities. Normally a server should only gzip the response whenever the client supports it (by Accept-Encoding: gzip). You need to ungzip it using GZIPInputStream.
reader = new BufferedReader(new InputStreamReader(new GZIPInputStream(connection.getInputStream()), "UTF-8"));
Note that I also added the right charset to the InputStreamReader constructor. Normally you'd like to extract it from the Content-Type header of the response.
For more hints, see also How to use URLConnection to fire and handle HTTP requests? If all what you after all want is parsing/extracting information from the HTML, then I strongly recommend to use a HTML parser like Jsoup instead.

Categories