Fetch JSON Data from URL and Repeatedly Update SQLite Database - java

In my app, I create a SQLite database. Then I populate it with JSON data fetched from a URL using an instance of the HttpAsyncTask class in my main activity. That works fine, but I also want to update the database. New data (one row in the database) is added to the URL page once per day, and I want to implement a "synchronize" button in the app that updates the database with only the new information. Could I get some advice on how to do this? My HttpAsyncTask is below, if that helps - I'm thinking I might need an if/else clause in the onPostExecute() method that adds all the rows only if the database is getting created for the first time. I thought about trying to put an HttpAsyncTask class in my DatabaseHelper, but that doesn't really make sense.
private class HttpAsyncTask extends AsyncTask<String, Void, String> {
#Override
protected String doInBackground(String...urls) {
return GET(urls[0]);
}
#Override
protected void onPostExecute(String result) {
try {
JSONObject main = new JSONObject(result);
JSONObject body = main.getJSONObject("body");
JSONArray measuregrps = body.getJSONArray("measuregrps");
// get measurements for date, unit, and value (weight)
for (int i = 0; i < measuregrps.length(); i++) {
JSONObject row = measuregrps.getJSONObject(i);
// a lot of getting & parsing data happens
db.addEntry(new Entry(date, weight, null, null));
//adds all the lines every time this is run, but I only want to add all
//the lines once and then add new rows one by one from there
}
} catch (JSONException e) {
e.printStackTrace();
}
}
}
public static String GET(String url) {
InputStream is = null;
String result = "";
try {
HttpClient client = new DefaultHttpClient();
HttpGet get = new HttpGet(url);
HttpResponse httpResponse = client.execute(get);
is = httpResponse.getEntity().getContent();
if (is != null)
result = convertInputStream(is);
else
result = "Did not work!";
} catch (Exception e) {
Log.d("input stream", e.getLocalizedMessage());
}
return result;
}
private static String convertInputStream(InputStream is) throws IOException {
StringBuilder builder = new StringBuilder();
BufferedReader reader = new BufferedReader(new InputStreamReader(is));
String line = "";
while((line = reader.readLine()) != null)
builder.append(line);
is.close();
return builder.toString();
}


Your implementation totally depends on the project requirements.
If there are continuously changes over the server, the right way to implement the synchronization process is:
1.
Implement the sync process, which works totally in background. This sync will be customized to call specific API calls/Service classes which will be required to sync.
2.
Server will prompt the mobile client for the data change.
3.
To get server updates, A continuously running service/Sync at some predefined intervals will be run and checks for the updates or implements the GCM.
Sync Adapter would be the best for the sync services.
Ohh, also don't forget to apply the content provider, as database calls would be concurrent from UI and background both.
Hope it may help to decide.


You have to check there is similar data available in the table if yes update the table and if no insert new data to table

Related

Best way to get Amazon page and product information

I want to get Amazon page and product information from their website so I work on a future project. I have no experience with APIs but also saw that I would need to pay in order to use Amazon's. My current plan was to use a WebRequest class which basically takes down the page's raw text and then parse through it to get what I need. It pulls down HTML from all the websites I have tried except amazon. When I try and use it for amazon I get text like this...
??èv~-1?½d!Yä90û?¡òk6??ªó?l}L??A?{í??j?ì??ñF Oü?ª[D ú7W¢!?É?L?]â  v??ÇJ???t?ñ?j?^,Y£>O?|?I`OöN??Q?»bÇJPy1·¬Ç??RtâU??Q%vB??^íè|??ª?
Can someone explain to me why this happens? Or even better if you could point me towards a better way of doing this? Any help is appreciated.
This is the class I mentioned...
public class WebRequest {
protected String url;
protected ArrayList<String> pageText;
public WebRequest() {
url = "";
pageText = new ArrayList<String>();
}
public WebRequest(String url) {
this.url = url;
pageText = new ArrayList<String>();
load();
}
public boolean load() {
boolean returnValue = true;
try {
URL thisURL = new URL(url);
BufferedReader reader = new BufferedReader(new InputStreamReader(thisURL.openStream()));
String line;
while ((line = reader.readLine()) != null) {
pageText.add(line);
}
reader.close();
}
catch (Exception e) {
returnValue = false;
System.out.println("peepee");
}
return returnValue;
}
public boolean load(String url) {
this.url = url;
return load();
}
public String toString() {
String returnString = "";
for (String s : pageText) {
returnString += s + "\n";
}
return returnString;
}
}

It could be that the page is returned using a different character encoding than your platform default. If that's the case, you should specify the appropriate encoding, e.g:
new InputStreamReader(thisURL.openStream(), "UTF-8")
But that data doesn't look like character data at all to me. It's too random. It looks like binary data. Are you sure you're not downloading an image by mistake?
If you want to make more sophisticated HTTP requests, there are quite a few Java libraries, e.g. OkHttp and AsyncHttpClient.
But it's worth bearing in mind that Amazon probably doesn't like people scraping its site, and will have built in detection of malicious or unwanted activity. It might be sending you gibberish on purpose to deter you from continuing. You should be careful because some big sites may block your IP temporarily or permanently.
My advice would be to learn how to use the Amazon APIs. They're pretty powerful—and you won't get yourself banned.

Android HTTP Requests Working In Simulator But Not On Wear Device

I am making a simple Android Wear app to control my thermostats, and I'm sending POST requests with Volley to control them. Everything works great in the Android Wear simulator (the request works), but, while the app does load on my Moto 360, the volley request gets called but invariably times out.
Why could my volley request be failing on my watch but working on the simulator? Other apps' requests succeed on my watch (for example, the built-in weather app can load up weather data in about 3 seconds). And, the weirdest part: I had the app working (successfully making volley requests) on my watch, and, about a day after I installed it to my watch from Android Studio, it suddenly stopped loading data for no apparent reason.
What I've tried so far:
I have requested the Internet permission in my manifest.xml.
I have increased the timeout to 30 seconds (see my code below), which didn't change anything.
I have tried tethering my computer and the simulator to my phone's connection via Bluetooth (to replicate the Bluetooth connection my physical watch has to my phone), and the simulator made the request successfully still (albeit with a two-second delay), ruling out the possibility of Bluetooth being too slow.
I made sure the API level is low enough for my Marshmallow-running watch (my watch and the app are both API level 23).
I tried doing a quick test request to Google before the request to the company's servers with my thermostat data, and while the Google request returns the site's HTML code in the simulator, it times out on my watch (thirty seconds after the request is initiated).
I tried putting some dummy data into the recycler view data should be loaded into, and the dummy data indeed showed up, ruling out that the recycler view is broken.
I deleted the app from my watch and reinstalled it, and deleted the companion from my phone, reinstalled it, and deleted it again, all to no avail.
A lengthy chat with Google Support did not produce anything meaningful.
Here's my code (from my main view's adapter):
public void refreshThermostatsRecyclerView(RequestQueue queue) {
String url = "https://mobile.skyport.io:9090/login"; // login call to the thermostats server Skyport
Log.w("myApp", "Starting /login call to Skyport"); // this gets called on simulator and watch
// Request a string response from the provided URL.
StringRequest stringRequest = new StringRequest(Request.Method.POST, url,
Response.Listener<String>() {
#Override
public void onResponse(String response) {
// Display the response string.
Log.w("myApp", "Response is: " + response); // this gets called on the simulator but not the watch
try {
// there's some code to parse the data.
} catch (JSONException e) {
Log.w("myApp", "catching an error parsing the json."); // never gets called.
e.printStackTrace();
}
}
}, new Response.ErrorListener() {
#Override
public void onErrorResponse(VolleyError error) {
Log.w("myApp", "Skyport request didn't work! " + error); // this always gets called on the watch, with the error being a timeout error (com.Android.Volley.timeouterror) but never gets called in the simulator
}
}) {
#Override
public Map<String, String> getHeaders() throws AuthFailureError {
Map<String, String> m = new HashMap<>();
m.put("Referer", "app:/VenstarCloud.swf");
// here I put some more headers
return m;
}
#Override
protected Map<String, String> getParams() throws AuthFailureError {
Map<String, String> m = new HashMap<>();
m.put("version", "3.0.5");
m.put("email", userEmail);
m.put("password", userToken);
return m;
}
};
// Add the request to the RequestQueue.
int socketTimeout1 = 30000; // times out 30 seconds after the request starts on the watch
RetryPolicy policy1 = new DefaultRetryPolicy(socketTimeout1, DefaultRetryPolicy.DEFAULT_MAX_RETRIES, DefaultRetryPolicy.DEFAULT_BACKOFF_MULT);
stringRequest.setRetryPolicy(policy1);
queue.add(stringRequest);
}
Which is called from the onCreate() method in my Main Activity with this code:
RequestQueue queue = Volley.newRequestQueue(this);
refreshThermostatsRecyclerView(queue);
If you'd like to view the logs created by running this in the simulator and on the watch, they're on Google Drive here.
Edit 1: A reboot of my watch fixes the issue temporarily and allows the watch to make HTTP Requests again, but it breaks again once the watch disconnects from Bluetooth, connects to WiFi, disconnects from WiFi, and reconnects to Bluetooth (so it breaks every time I go across my apartment without my phone and then return).
Edit 2: I switched the volley requests all over to HTTPURLConnection Requests in an Async thread, and the same issues occur as with volley.
tl;dr: My app's Volley requests are working in the simulator but not on my Android Wear watch anymore (though Play Store-downloaded apps' similar requests work), how can I get a volley request to work again on my app on the watch?

As per these two conversations below, it seems that WiFi connectivity only allows Android Wear to connect to a phone over WiFi and not directly to the Internet. However, Android Wear 2.0 lets you use regular network APIs.
Direct internet connection on Android Wear?
Does Android Wear support direct access to the Internet?
So, for Android Wear 2.0+ Volley requests from wearable app should work.
If you want to use Android Wear <2.0, then:
On Wearable, in onCreate() add a key that indicates whether the phone should start collecting data.
PutDataMapRequest putDataMapReq = PutDataMapRequest.create("/shouldStart");
putDataMapReq.getDataMap().putBoolean(SHOULD_START_KEY, true);
PutDataRequest putDataReq = putDataMapReq.asPutDataRequest();
PendingResult pendingResult = Wearable.DataApi.putDataItem(mGoogleApiClient, putDataReq);
On phone, in onDataChanged, check if wearable wants to start collecting data. If yes, start Volley request.
for (DataEvent event : dataEvents) {
if (event.getType() == DataEvent.TYPE_CHANGED) {
// DataItem changed
DataItem item = event.getDataItem();
if (item.getUri().getPath().compareTo("/shouldStart") == 0) {
DataMap dataMap = DataMapItem.fromDataItem(item).getDataMap();
boolean shouldStart = dataMap.getBoolean(SHOULD_START_KEY));
if(shouldStart) {
Volley.newRequestQueue(this).add(request);
}
}
} else if (event.getType() == DataEvent.TYPE_DELETED) {
// DataItem deleted
}
}
Then, your Volley request's onResponse should pass data back to Wearable.
public void onResponse(String response) {
PutDataMapRequest putDataMapReq = PutDataMapRequest.create("/data");
putDataMapReq.getDataMap().putString(DATA_KEY, true);
PutDataRequest putDataReq = putDataMapReq.asPutDataRequest();
PendingResult pendingResult = Wearable.DataApi.putDataItem(mGoogleApiClient, putDataReq);
}
Finally, you can access data in your Wearable using onDataChanged and store it in your model for passing it onto adapter:
for (DataEvent event : dataEvents) {
if (event.getType() == DataEvent.TYPE_CHANGED) {
// DataItem changed
DataItem item = event.getDataItem();
if (item.getUri().getPath().compareTo("/data") == 0) {
DataMap dataMap = DataMapItem.fromDataItem(item).getDataMap();
parseAndpassToAdapter(dataMap.getString(DATA_KEY));
}
} else if (event.getType() == DataEvent.TYPE_DELETED) {
// DataItem deleted
}
}
You'll need Wearable.API to implement this and your class should implement DataApi.DataListener. For more information getting started, refer to Accessing the Wearable Data Layer and Syncing Data Items
Hope this helps.

I am also using volley on an Android wear app I built and I am running it on a Moto 360, I have run into the same problem a couple o times. Try restarting the device. Go to Settings > Restart. It sounds silly but it has worked for me.

You could try an alternative to volley if you can rule out the connection as the problem:
compile 'com.android.support:appcompat-v7:23.1.1'
compile 'com.android.support:support-v4:23.1.0'
compile 'com.android.support:design:23.1.0'
compile 'com.google.code.gson:gson:2.2.4'
compile 'com.google.api-client:google-api-client:1.20.0'
The versions are important.
Then to your request:
Map<String, String> contentParams = new HashMap<>();
InputStream is = null;
NetHttpTransport transport = null;
HttpRequest request = null;
HttpResponse resp = null;
HttpHeaders headers = new HttpHeaders();
JSONObject json = null;
try {
transport = new NetHttpTransport();
HttpRequestFactory factory = transport.createRequestFactory();
request = factory.buildPostRequest(new GenericUrl(url), null);
contentParams = getContentParameters();
headers.putAll(getHeaderParameters());
request.setHeaders(headers);
request.getUrl().putAll(contentParams);
resp = request.execute();
is = resp.getContent();
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
if (is != null) {
string = getJSONFromInputStream(is);
json = new JSONObject(string);
}
} catch (Exception e) {
e.printStackTrace();
}
}
transport.shutdown();
protected Map<String, String> getContentParameters() {
Map<String, String> m = new HashMap<>();
m.put("version", "3.0.5");
m.put("email", userEmail);
m.put("password", userToken);
return m;
}
protected Map<String, String> getHeaderParameters() {
Map<String, String> m = new HashMap<>();
m.put("Referer", "app:/VenstarCloud.swf");
return m;
}
protected String getJSONFromInputStream(InputStream is) {
if (is == null)
throw new NullPointerException();
//instantiates a reader with max size
BufferedReader reader = new BufferedReader(new InputStreamReader(is), 8 * 1024);
StringBuilder sb = new StringBuilder();
try {
//reads the response line by line (and separates by a line-break)
String line;
while ((line = reader.readLine()) != null) {
sb.append(line + "\n");
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
//closes the inputStream
is.close();
} catch (IOException e) {
e.printStackTrace();
}
}
return sb.toString();
}
Then just execute your code from a thread/asynctask/have it delay your frontend slightly
Edit:
Just in case there is a problem with appending a map:
for (Entry<String, String> entry : getHeaderParameters()) {
headers.put(entry.getKey(), entry.getValue());
}
for (Entry<String, String> entry : getContentParameters()) {
request.getUrl().put(entry.getKey(), entry.getValue());
}
Also as another note, make sure to change the return type from void on both those methods to Map

Is this not just the case of when the watch is connected to the phone via bluetooth the internet will not work, as wifi is turned off. If the watch is using wifi to connect to the phone then it will work.
I'm working on wear 2.0 app and just turn blueooth off on my phone for my watch to get internet connection.

webservice client implementaion in axis2-1.6.2 in java

I have implemented webservice client in axis2-1.6.2 in java and I get response when I call first time and for subsequent second time I get below error
java.lang.NullPointerException
at org.apache.axis2.client.OperationClient.prepareMessageContext(OperationClient.java:293)
at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:180)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:165)
at org.apache.axis2.ccws.CustomerCareServiceStub.subscriberRetrieveLite(CustomerCareServiceStub.java:2380)
at Prepost.SubscriberRetrieveBalance.subscriberRetrieveLite(SubscriberRetrieveBalance.java:111)
at Prepost.CheckUser.doGet(CheckUser.java:149)
here is the API implementation class constructor which sets unique parameter which is same for all requests
public SubscriberRetrieveBalance(String url, String strCON_TimeOut, String strSO_TimeOut) {
try {
this.url = url;
stub = new CustomerCareServiceStub(url);
ServiceClient sClient = stub._getServiceClient();
Options options = sClient.getOptions();
options.setProperty(HTTPConstants.REUSE_HTTP_CLIENT, Constants.VALUE_TRUE);
options.setProperty(AddressingConstants.WS_ADDRESSING_VERSION, AddressingConstants.Submission.WSA_NAMESPACE);
//options.setTimeOutInMilliSeconds(2000);
TransportInDescription transportIn = new TransportInDescription("HTTP");
options.setTransportIn(transportIn);
options.setProperty(HTTPConstants.SO_TIMEOUT, Integer.parseInt(strSO_TimeOut));
options.setProperty(HTTPConstants.CONNECTION_TIMEOUT, Integer.parseInt(strCON_TimeOut));
sClient.setOptions(options);
} catch (Exception e) {
if (e.getMessage().equals("Can not find the external id")) {
System.out.println("Exception ::" + e.getMessage());
}
}
}
and it is called in a servlet and for performance issue I make object of this class for different-2 states(urls) and saved these object to hashmap when first request comes for respective states then make new object and use that object for subsequent requests for that state
SubscriberRetrieveBalance objBal = null;
BalanceBean bal = new BalanceBean();
if (mapObj.isEmpty() || (mapObj.get(strIP) == null)) {
objBal = new SubscriberRetrieveBalance(url, strCON_TimeOut, strSO_TimeOut);
mapObj.put(strIP, objBal);
} else {
objBal = mapObj.get(strIP);
}
bal = objBal.subscriberRetrieveLite(strMsisdn, userId, token, strCircleId, strCircleName, strSessionId, strDlgId);
first time it gives response and then gives nullpointer exception and above error for all requests that belongs to that state
This code is working fine with axis2-1.5
Is there any change in axis2-1.6.2 version that every time it needs new object of API implemented class
Please suggest.

Text not being saved to String variable

I have been trying to get an app to work that allows me to read from a URL and then use the text that I get from the URL for other purposes in the app.
The problem I'm having is that the text isn't being "saved".
I know for a fact that my #getText method works because I ran a basic command line application in IntelliJ:
String textFromUrl;
public static void main(String[] args) {
textFromUrl = Vars.getEngUrlText();
System.out.println(textFromUrl);
}
and the result was it printing the exact text it should. And this was written inside of the main activity's class in my Android project, I just ran the main method as a normal Java application instead of running the actual apk from my USB device.
Now, I try to do the same in the Android device by doing
In Vars class:
String textFromUrl;
In #onCreate of the first activity:
Vars.textFromUrl = Vars.getEngUrlText();
TextView tx = (TextView) findViewById(R.id.titletext);
tx.setText(Vars.textFromUrl);
and it just displays blank, no text, no nothing. Rest of the layout is fine though, everything else shows and no errors. I'm assuming the value of textFromUrl is null and I don't know why.
Yes I do have the proper permissions to access the web in my AndroidManifest because I'm using a WebView and it works fine. I've even tried running threads that give it some time to wait (about 5 seconds) before changing the text and it still won't work.
What's going on?
getText and #getEngUrlText below.
getEngUrlText calls getText:
public static String getText(String url) throws Exception {
URL website = new URL(url);
URLConnection connection = website.openConnection();
BufferedReader in = new BufferedReader(
new InputStreamReader(
connection.getInputStream()));
StringBuilder response = new StringBuilder();
String inputLine;
while ((inputLine = in.readLine()) != null)
response.append(inputLine);
in.close();
return response.toString();
}
public static String getEngUrlText() {
try {
textFromUrl = getText("url that is supposed to be here removed");
} catch (Exception e) {
e.printStackTrace();
}
return textFromUrl;
}

Fixed this by targetting a lower SDK. Apparently I didn't see a NetworkOnMainThreadException that the logcat was displaying because I was only looking for errors and not all warnings and such.

NullPointerException on JSONArray initialized in background thread

I'm writing an android app I use a background thread to pull a JSONArray from a web service. I then need to interact with that JSONArray inside the main activity. Here's what I'm doing now:
public class MainActivity extends Activity {
JSONArray stories;
#Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
new getAll().execute();
// try {
System.out.println("stories.length());
// } catch (JSONException e) {
// TODO Auto-generated catch block
// e.printStackTrace();
//}
}
And the background thread:
private class getAll extends AsyncTask <Void, Void, JSONArray> {
private static final String url = "http://10.0.2.2:8080/CalibServer/webresources/storypkg.story/";
#Override
protected JSONArray doInBackground(Void... params) {
//set up client and prepare request object to accept a json object
HttpClient httpclient = new DefaultHttpClient();
HttpGet httpget = new HttpGet(url);
httpget.addHeader("accept", "application/json");
HttpResponse response;
String resprint = new String();
try {
response = httpclient.execute(httpget);
// Get the response entity
HttpEntity entity = response.getEntity();
if (entity != null) {
// get entity contents and convert it to string
InputStream instream = entity.getContent();
String result= convertStreamToString(instream);
resprint = result;
// construct a JSON object with result
stories =new JSONArray(result);
// Closing the input stream will trigger connection release
instream.close();
}
}
catch (ClientProtocolException e) {System.out.println("CPE"); e.printStackTrace();}
catch (IOException e) {System.out.println("IOE"); e.printStackTrace();}
catch (JSONException e) { System.out.println("JSONe"); e.printStackTrace();}
System.out.println("FUCKYEAHBG: " + resprint);
// stories = object;
return stories;
}
My problem is that I'm getting a NullPointerException at the call to
System.out.println("stories.length());
It's acting like a didn't initialize the stories array, but should't that have been taken care of by the background thread (at the line: stories =new JSONArray(result); ) before that call is ever made?
I have a feeling this is because of the threading - perhaps there is another step I have to take to update the main activity after the AsyncTask runs?

You're initializing the variable in a background thread. That means that the line
System.out.println(stories.length());
is executed in parallel with the code initializing the variable. That thus means that there is a huge chance that the background thread has not had the time to initialize the variable yet when this line is executed.
Your code is similar to the following situation: you have an empty cup in front of you, and ask someone to go make some coffee and to fill your cup. And immediately after asking, you start drinking. There will be no coffee inside the cup, obviously.
Re-read the android documentation on how to execute asynchronous tasks.

You can't rely on stories to be initialized when a separate Thread that runs parallel to the UI Thread initializes and updates stories.
perhaps there is another step I have to take to update the main
activity after the AsyncTask runs?
onPostExecute() of the AsyncTask. Do whatever UI updates you need there. Since getAll is already a private inner class, you have full access to the Activity. You already return stories off to that (unoverriden) method, so this should be a minor change.
#Override
protected void onPostExecute (JSONArray stories)
{
//use the now initialized stories
}

Categories