Java reading information from a website using Jsoup - java

I've gone through multiple posts about parsing and such. Most of the responses that I saw were recommending the person to use a Library or something else. My problem right now is creating an algorithm that will fetch the exact information I want. My purpose for this is to fetch 2 statuses from the Weather website for school closings. I started using Jsoup as someone recommended but I need help with it.
Webpage: Click here
Image: Click here
Example of webpage source: click here
I could probably figure out how to get a certain line of text within the webpage since I already know the name of the school im looking for, but 2 lines down is the status which is what I need. would be easy if each school had a certain status but they are all either Closed or Two-hour Delay so I can't just make a search for that. I want some ideas or answers on how i can apporach this. I am going to do this 2 times because I am wanting to look up 2 schools. I already have the names which I can use to look them up I just need the status.
Here is an example of what I want to do. (sudo code)
Document doc = connect(to url);
Element schoolName1 = doc.lookForText(htmlLineHere/schoolname);
String status1 = schoolName.getNext().text();//suppose this gets the line right after which should be my status and then cleans off the Html.
This is what I have right now
public static SchoolClosing lookupDebug() throws IOException {
final ArrayList<String> Status = new ArrayList<String>();
try {
//connects to my wanted website
Document doc = Jsoup.connect("http://www.10tv.com/content/sections/weather/closings.html").get();
//selects/fetches the line of code I want
Element schoolName = doc.html("<td valign="+"top"+">Athens City Schools</td>");
//an array of Strings where I am going to add the text I need when I get it
final ArrayList<String> temp = new ArrayList<String>();
//checking if its fetching the text
System.out.println(schoolName.text());
//add the text to the array
temp.add(schoolName.text());
for (int i = 0; i <= 1; i++) {
final String[] tempStatus = temp.get(i).split(" ");
Status.add(tempStatus[0]);
}
} catch (final IOException e) {
throw new IOException("There was a problem loading School Closing Status");
}
return new SchoolClosing(Status);
}


Document doc = Jsoup.connect(
"http://www.10tv.com/content/sections/weather/closings.html")
.get();
for (Element tr : doc.select("#closings tr")) {
Element tds = tr.select("td").first();
if (tds != null) {
String county = tr.select("td:eq(0)").text();
String schoolName = tr.select("td:eq(1)").text();
String status = tr.select("td:eq(2)").text();
System.out.println(String.format(
"county: %s, schoolName: %s, status: %s", county,
schoolName, status));
}
}
output:
county: Athens, schoolName: Beacon School, status: Two-hour Delay
county: Franklin, schoolName: City of Grandview Heights, status: Snow Emergency through 8pm Thursday
county: Franklin, schoolName: Electrical Trades Center, status: All Evening Activities Cancelled
county: Franklin, schoolName: Hilock Fellowship Church, status: PM Services Cancelled
county: Franklin, schoolName: International Christian Center, status: All Evening Activities Cancelled
county: Franklin, schoolName: Maranatha Baptist Church, status: PM Services Cancelled
county: Franklin, schoolName: Masters Commission New Covenant Church, status: Bible Study Cancelled
county: Franklin, schoolName: New Life Christian Fellowship, status: All Activities Cancelled
county: Franklin, schoolName: The Epilepsy Foundation of Central Ohio, status: All Evening Activities Cancelled
county: Franklin, schoolName: Washington Ave United Methodist Church, status: All Evening Activities Cancelled
or in a loop:
for (Element tr : doc.select("#closings tr")) {
System.out.println("----------------------");
for (Element td : tr.select("td")) {
System.out.println(td.text());
}
}
that gives:
----------------------
Athens
Beacon School
Two-hour Delay
----------------------
Franklin
City of Grandview Heights
Snow Emergency through 8pm Thursday
----------------------
Franklin
Electrical Trades Center
All Evening Activities Cancelled
...

Related

How to understand MTA's real-time gtfs data feeds

So far I've filled out the MTA online registration form for a Developer's API Key. Then in my android project, I implemented the gtfs realtime bindings dependecy from one of Google's github repositories,
compile group: 'com.google.transit', name: 'gtfs-realtime-bindings', version: '0.0.4'
Using their Java class, I tried this following code to print out all the gtfs data from the link given by MTA,
try {
String urlString = "http://datamine.mta.info/mta_esi.php?key=insertmykeyhere";
URL url = new URL(urlString.toString());
GtfsRealtime.FeedMessage feed = GtfsRealtime.FeedMessage.parseFrom(url.openStream());
for (GtfsRealtime.FeedEntity entity : feed.getEntityList()) {
if (entity.hasTripUpdate()) {
Log.d("stuff", entity.getTripUpdate().toString());
}
}
} catch (IOException e) {
e.printStackTrace();
}
However, I'm now having trouble interpreting the data printed out. I understand that there are static data feeds from http://web.mta.info/developers/developer-data-terms.html, which I used to interpret some of the data. Here is one of the trip updates printed out,
stuff: trip {
trip_id: "036000_GS.N01R"
start_date: "20170218"
route_id: "GS"
1001: "\n\0200S 0600 GCS/TSS\020\001\030\001"
}
stop_time_update {
departure {
time: 1487415600
}
stop_id: "901N"
1001: "\n\0011\022\0011"
}
stop_time_update {
arrival {
time: 1487415690
}
stop_id: "902N"
1001: "\n\0011"
}
I understand some parts such as trip_id, start_date, and stop_id. But parts such as trip_id, time, and 1001 I'm still unsure about it and the text files from the static feed don't do the best job of explaining them.

The MTA Subway GTFS-RT feeds are a little different than most others.
Typically, GTFS-RT refers directly back to a static GTFS' via trip_id/stop_id/etc. Since in NYC there is usually a deviation from normal service ("2 Train via the 5 Line"), the RT feed retains the option to create new trips that do not exist in the static feed.
To answer your immediate questions, you need to add the realtime extensions. That should solve the empty 1001: field.
To do so, either compile the proto file, or just import the onebusaway-gtfs-realtime-api library, which has the extensions pre-compiled in:
ExtensionRegistry registry = ExtensionRegistry.newInstance();
registry.add(GtfsRealtimeNYCT.nyctFeedHeader);
registry.add(GtfsRealtimeNYCT.nyctStopTimeUpdate);
registry.add(GtfsRealtimeNYCT.nyctTripDescriptor);
GtfsRealtime.FeedMessage feed = GtfsRealtime.FeedMessage.parseFrom(url.openStream(), registry);
This gives a result like:
trip {
trip_id: "B20170217WKD_132800B..S"
start_date: "2017-02-17 22:08:00"
route_id: "B"
[transit_realtime.nyct_trip_descriptor] {
train_id: "1B 2208 145/BBC"
is_assigned: true
direction: SOUTH
}
}
stop_time_update {
arrival {
time: 1487390920
}
departure {
time: 1487390920
}
stop_id: "D39"
schedule_relationship: SCHEDULED
[transit_realtime.nyct_stop_time_update] {
scheduled_track: "A3"
actual_track: "A3"
}
}
stop_time_update {
arrival {
time: 1487391130
}
departure {
time: 1487391130
}
stop_id: "D40"
schedule_relationship: SCHEDULED
[transit_realtime.nyct_stop_time_update] {
scheduled_track: "A3"
actual_track: "A3"
}
}

script to search address on google maps

I have names of 1100 hospitals from NY region. I need to find the address of these hospitals from google. I am looking for some script which I can use to supply all these hospital name and it could return me with an address. The script could return a simple google search result.
Input format:
Hospital Name
Center for Ambulatory Surgery
Genetic Diagnostic Labs Inc
Desired output format:
Hospital Name Hospital Address
Center for Ambulatory Surgery 3112 Sheridan Dr, Amherst, NY 14226
Genetic Diagnostic Labs Inc 490 Delaware Ave, Buffalo, NY 14202

A solution with google Places API, but the results may be not very accurate:
http://codepen.io/anon/pen/JogeyV?editors=101
var NY_latlng = new google.maps.LatLng(40.828624, -73.898605);
map = new google.maps.Map(document.getElementById('map-canvas'), {
center: NY_latlng,
zoom: 15
});
var hospitals = [];
var hospitals_names = ["Center for Ambulatory Surgery","Genetic Diagnostic Labs Inc"];//insert your full list here
var service = new google.maps.places.PlacesService(map);
hospitals_names.forEach( function(name ){
service.textSearch(
{
query: name,
location: NY_latlng,
radius: 50000, //in meter
},function(results,status){
if (status == google.maps.places.PlacesServiceStatus.OK){
var hospital= { name: name, addresses: []};
$('#address-list').append("<h2>"+name+"</h2><ul></ul>");
for (var i = 0; i < results.length; i++) {
hospital.addresses.push( results[i].formatted_address );
$('#address-list > ul').append("<li>"+results[i].formatted_address);
}
hospitals.push( hospital );
}
});

You can do this in R with the ggmap package, though perhaps not reliably enough to produce the results you want. For instance this attempt to geocode fails:
geocode("Genetic Diagnostic Labs Inc")
Warning message:
geocode failed with status ZERO_RESULTS, location = "Genetic Diagnostic Labs Inc"
So to illustrate a solution, I appended " NY" to the Google searches:
library(ggmap)
hospital_names <- c("Center for Ambulatory Surgery", "Genetic Diagnostic Labs Inc")
address_vec <- lapply(hospital_names, function(x) revgeocode(as.numeric(geocode(paste(x,", NY")))))
result <- data.frame(name = hospital_names, address = unlist(address_vec))
Result:
result
name address
1 Center for Ambulatory Surgery 426 Union Road, West Seneca, NY 14224, USA
2 Genetic Diagnostic Labs Inc City Hall Park Path, New York, NY 10007, USA
But these are not the addresses you specified - you may need to refine your inputs.

Error refreshing inventory. In-app Billing

I am setting up and testing in-app billing. I managed to purchase the android.test.purchased, and it did what it should. But now I need to consume it to continue my testing. The problem is that I can't reach the inventory.
When this is called I get the result.isFaliure() is called and I can't get the inventory.
IabHelper.QueryInventoryFinishedListener _gotInventoryListener = new IabHelper.QueryInventoryFinishedListener() {
#Override
public void onQueryInventoryFinished(IabResult result, Inventory inventory) {
if (_iabHelper == null) return;
if (result.isFailure()) {
Log.d(TAG, "Failed to query inventory: " + result);
return;
}
Log.d(TAG, "Query inventory was successful.");
Purchase premiumPurchase = inventory.getPurchase(SKU_PREMIUM);
_isPremium = (premiumPurchase != null && verifyDeveloperPayload(premiumPurchase));
Log.d(TAG, "User is " + (_isPremium ? "PREMIUM" : "NOT PREMIUM"));
update();
}
};
It logs the error message
Failed to query inventory: IabResult: Error refreshing inventory
(querying owned items). (response: -1003:Purchase signature
verification failed)
The android.test.purchased is still owned - it won't let me buy it again. My phone has network connection so it's not that.
I have NOT uploaded a signed APK to Google Play, does that matter even if I test with googles static ID's?

Solved it...
It seems there are problems with the static purchase ID's.
Here's a sollution I found in THIS thread:
If you have used the android.test.purchased then one way to get rid of the error is to do the following:-
1. Edit Security.java and change the "return false" line in the
verifyPurchase to "return true" - this is temporary, we'll be
putting it back in a minute.
2. In your QueryInventoryFinishedListener, after the "if
(result.isFailure()) {...}" lines add the following to consume and
get rid of your never ending android.test.purchased item:-
if (inventory.hasPurchase(SKU_ANDROID_TEST_PURCHASE_GOOD)) {
mHelper.consumeAsync(inventory.getPurchase(SKU_ANDROID_TEST_PURCHASE_GOOD),null);
}
3. Run your app so the consunmeAsync happens, this gets rid of the
"android.test.purchased" item on the server.
4. Remove the consumeAsync code (or comment it out). Back in the
Security.java, change the "return true" back to "return false".

I found the answer here:
"Here's a recommendation: Make sure that your Billing Key (base64EncodedPublicKey) is properly saved. That was my problem, after all that..."
base64EncodedPublicKey was from another aplication...
It was solution for me.

How to Insert time in mysql using java

I'm doing a project to catering item hiring service. They want to save the date and time customer hire their items also save the date and time customer return them. Of course any of those not gona be actual at the movement date or time. So first of all I'm trying to find to save any date and time. Other activities will concern later. So I came up with little table. like given below.
ID | Name | Value | Date | Time
(int) | (Varchar) |( Double) | (DateTime) | (DateTime)
And I used this cord to save data.
private void jButton2ActionPerformed(java.awt.event.ActionEvent evt)
{
String date=new SimpleDateFormat("yyyy-MM-dd").format(dat.getDate());
String time = String.valueOf(dateSpinner.getValue()).split(" ")[3];
try {
new JDBC().putData("INSERT INTO work (name, balance, date) VALUES ('"+txtName.getText()+"',"+txtValue.getText()+"', '"+date+"', '"+time+"') ");
JOptionPane.showMessageDialog(null, "value saved");
} catch (Exception e) {
JOptionPane.showMessageDialog(null, this.getClass().getName() +" "+e);
}
}
When I'm entering data I'm having
Exception in thread "AWT-EventQueue-0" java.lang.ArrayIndexOutOfBoundsException: 3" .
Help me to find the error of this cord and save any time. Actually I can save any date using String Date variable. I saved some demo data before I added time field to table. But I'm having this problem when I'm trying to save time. I am using date Spinner to save time. But I can't format it as HH-mm--ss. If you know another time saving feature please let me know. Please help me.

This line:
String time = String.valueOf(dateSpinner.getValue()).split(" ")[3];
Is the same as this, just done on a single line
String[] splitArr = String.valueOf(dateSpinner.getValue()).split(" ");
String time = splitArr[3];
The error you are getting is saying that there is not 4 elements in the split array. Put in the below piece of code to debug what you are getting so you know what is happening in that single line
String[] splitArr = String.valueOf(dateSpinner.getValue()).split(" ");
for(int i = 0; i < splitArr.length; i++)
{
System.out.println("Element number " + i + " is \"" + splitArr[i] + "\"");
}
Then you can update your insert appropriately. Sorry I cant be of more help but we do not know what datespinner is or what datespinner.getValue() returns so do the above to help debug your code.

How to parse nested json arrays in a single object in android

I have a json which has a number of nested JSONARRAY. here is the snippet.
{
"location": [
{
"name": "Perth",
"conferencelocation": [
{
"locationname":"Stage 1" ,
"guests": [
{
"guestid":"4074513426041094",
"guestname":"Keegan Connor Tracy",
"time":"9am",
"largeimg":"http://ozcomiccon.com/images/banner/Keegan.jpg",
"smallimg":"http://ozcomiccon.com/images/banner/Keeganresized.jpg",
"biotext": "Born in Windsor, Ontario, Canada, Tracy attended Wilfrid Laurier University in Waterloo, Ontario, Canada, where she orginally studied business. She later switched to psychology. She spent a year in Europe working in Dublin, Paris and Nice, where she was supposed to be completing her 4th year of study. She later returned to WLU to finish her degree. She moved to Vancouver, B.C. where she has had all of her acting jobs.",
"autographs":"$30 each (an 8x10 photograph will be included with each signature, or you may choose to have an appropriate personal item signed).",
"photographs":"Photo with Keegan $40.",
"genre":"acting",
"links": [
{"linkname":"Twitter", "url":"https://twitter.com/keegolicious" },
{"linkname":"IMDB", "url":"http://www.imdb.com/name/nm0870535/" }
]
},
{
"guestid":"1054713366041913",
"guestname":"Matthew Clark",
"time":"10am",
"largeimg":"http://ozcomiccon.com/images/banner/matthewclark.jpg",
"smallimg":"http://ozcomiccon.com/images/banner/matthewclarkresized.jpg",
"biotext": "Matthew Clark is a comic book artist living and working in Portland, Oregon. Having worked on art of some of the most well-known DC characters in comic book history, he recently did pencils for DC’s latest version of the Doom Patrol. Matthew is currently working on Ghost Rider for Marvel Comics, with whom he has signed an exclusive two-year deal.<br /><br />Matthew has lived in Portland for the past 13 years, just above an art gallery in the heart of downtown. He loves to walk around this fair city (mainly because he sold his car and needed to force himself to exercise). He’s a native Oregonian and that’s saying something. For the past 11 years he’s met Greg Rucka for coffee every Wednesday pretty much without fail at the same place (we pretty much carved our names in the table). Matthew was also one of the original founders of Mercury Studio (now Periscope Studio); he got his start at Studiosaurus.<br /><br />He still reads comics, loves what he does, and works very hard. Loves to meet new people, but is really quiet (and has great hair).",
"autographs":"Complimentary at guest's discretion.(Twitter)",
"photographs":"",
"genre":"comic",
"links": [
{"linkname":"Website", "url":"http://www.matthewclarkartist.com/" },
{"linkname":"Twitter", "url":"https://twitter.com/emceeartist" }
]
}
]
},
{
"locationname":"Stage 2" ,
"guests": [
{
"guestid":"106146633306036834",
"guestname":"Sean Williams",
"time":"8am",
"largeimg":"http://ozcomiccon.com/images/banner/seanwilliams.jpg",
"smallimg":"http://ozcomiccon.com/images/banner/seanwilliamsresized.jpg",
"biotext": "Sean Williams was born in the dry, flat lands of South Australia, where he still lives with his wife and family. He has been called many things in his time, including “the premier Australian speculative fiction writer of the age†(Aurealis), the “Emperor of Sci-Fi†(Adelaide Advertiser), and the “King of Chameleons†(Australian Book Review) for the diversity of his output. That award-winning output includes thirty-five novels for readers all ages, seventy-five short stories across numerous genres, the odd published poem, and even a sci-fi musical.<br /><br />He is a multiple recipient of the Aurealis and Ditmar Awards in multiple categories and has been nominated for the Philip K. Dick Award, the Seiun Award, and the William Atheling Jr. Award for criticism. He received the “SA Great†Literature Award in 2000 and the Peter McNamara Award for contributions to Australian speculative fiction in 2008.<br /><br />On the sci-fi front, he is best-known internationally for his original award-winning space opera series as well as several novels set in the Star Wars universe, many co-written with fellow-Adelaidean Shane Dix. These include the Astropolis, Evergence, Orphans and Geodesica series, and the computer game tie-in The Force Unleashed–the first such adaptation ever to debut at #1 on the New York Times bestseller list. A series for young readers, The Fixers, pitted an increasingly lost protagonist against zombies, cyborgs, and vampires across numerous universes. His most recent releases in the Star Wars universe are The Old Republic: Fatal Alliance and The Force Unleashed II.",
"autographs":"Complimentary at guest's discretion.",
"photographs":"",
"genre":"comic",
"links": [
{"linkname":"website", "url":"http://www.seanwilliams.com" }
]
},
{
"guestid":"17148603639603876",
"guestname":"Richard Dean Anderson",
"time":"10am",
"largeimg":"http://ozcomiccon.com/images/banner/richarddeananderson.jpg",
"smallimg":"http://ozcomiccon.com/images/banner/richarddeanandersonresized.jpg",
"biotext": "Richard Dean Anderson is probably best known as MacGyver, the clever and inventive nonviolent hero who solved problems in his own unique way for seven successful seasons on ABC. In his roles before and since, this gifted actor has continued to demonstrate his remarkable talent and versatility.<br /><br />Richard was born on January 23, 1950 in Minneapolis, Minnesota. His father, Stuart Anderson, taught English, drama, and humanities at a local high school, and is an accomplished jazz bassist. His mother, Jocelyn, is an artist, talented in both painting and sculpture. Richard is the eldest of four sons. He and his brothers, Jeffrey Scott, Thomas John, and James Stuart, grew up in the Minneapolis suburb of Roseville, where Richard developed early interests in sports, the arts, music, and acting.<br /><br />Like many boys growing up in Minnesota, Richard dreamed of becoming a professional hockey player. However, at the age of 16, he broke both arms, in separate accidents three weeks apart, while playing in high school hockey games. He put aside his dreams of playing professionally, though he still harbors a deep love for the sport. Richard talks of his restlessness growing up, his early desire to explore, and his adventures hitchhiking and hopping freight trains. At the age of 17, he took a 5641 mile bicycle trip from his home in Minnesota through Canada and Alaska, an experience which was sparked by his sense of adventure and discovery, but which also gave him a more centered sense of direction.",
"autographs":"Autograph from Richard $50.",
"photographs":"Photograph with Richard $80. SG1 double shot with Richard and Teryl Rothery $150. SG1 triple shot with Richard, Teryl Rothery and Corin Nemec $200.",
"genre":"acting",
"links": [
{"linkname":"Website", "url":"http://www.rdanderson.com/" },
{"linkname":"Twitter", "url":"https://twitter.com/andersonrdean" }
]
}
]
}
]
}
]
I have object class named Name, and I want to put all the data of the name(perth) jsonobject as a single object of the Name class. But it is not adding data in the arraylist correctly, more clearly the guestid, guestname etc tag values are being saved but overwrites the previous one. Here is my parsing code:
public ArrayList<Guest> parseInitiator() throws ClientProtocolException,
IOException, JSONException {
InputStream is = new FileInputStream(new File(Url));
int size = is.available();
byte[] buffer = new byte[size];
is.read(buffer);
is.close();
String mResponse = new String(buffer);
JSONObject first = new JSONObject(mResponse);
JSONArray firstarray = first.getJSONArray("location");
ArrayList<Guest> parsedData = new ArrayList<Guest>();
for (int i = 0; i < firstarray.length(); i++) {
JSONObject jonj = firstarray.getJSONObject(i);
name = jonj.getString("name");
JSONArray confarray = jonj.getJSONArray("conferencelocation");
locationname = new String[confarray.length()];
for (int j = 0; j < confarray.length(); j++) {
JSONObject conobbj = confarray.getJSONObject(j);
locationname[j] = conobbj.getString("locationname");
JSONArray guestarray = conobbj.getJSONArray("guests");
guestid= new String[guestarray.length()];
guestname= new String[guestarray.length()];
time= new String[guestarray.length()];
largeimg= new String[guestarray.length()];
smallimg= new String[guestarray.length()];
biotext= new String[guestarray.length()];
autographs= new String[guestarray.length()];
photographs= new String[guestarray.length()];
genre= new String[guestarray.length()];
for (int k = 0; k < guestarray.length(); k++) {
JSONObject guestobbj = guestarray.getJSONObject(k);
guestid[k] = guestobbj.getString("guestid");
guestname[k] = guestobbj.getString("guestname");
time[k] = guestobbj.getString("time");
largeimg[k] = guestobbj.getString("largeimg");
smallimg[k] = guestobbj.getString("smallimg");
biotext[k] = guestobbj.getString("biotext");
autographs[k] = guestobbj.getString("autographs");
photographs[k] = guestobbj.getString("photographs");
genre[k] = guestobbj.getString("genre");
JSONArray linkar = guestobbj.getJSONArray("links");
arlink= new String[linkar.length()];
arurl= new String[linkar.length()];
for (int l = 0; l < linkar.length(); l++) {
JSONObject linkobj = linkar.getJSONObject(l);
arlink[l] = linkobj.getString("linkname");
arurl[l] = linkobj.getString("url");
}
}
}
parsedData.add(new Guest(name,locationname, guestid, guestname, time, largeimg, smallimg, biotext,
autographs, photographs, genre,arlink, arurl));
}
return parsedData;
}
Can anyone make me clear about the mistake I have made?

So here is my solution:
JSONObject first = new JSONObject(mResponse);
JSONArray locArr = first.getJSONArray("location"); // contains one object
JSONObject locArrObj = locArr.getJSONObject(0); // cotains one "out" array
JSONArray conferenceLocArr = locArrObj.getJSONArray("conferencelocation");
// this array has two objects and each object has array
JSONObject o = null;
JSONArray arr = null;
for (int i = 0; i < conferenceLocArr.length(); i++) {
o = conferenceLocArr.getJSONObject(i); // it has one array
arr = o.getJSONArray("guests");
// do your work with Stage 1 and guests
// and for second object for Stage 2 and guests.
}

Categories