I have used the HSQLDB for storing the data in my app. I make 15 Web service call one by one. and store the data in DB. It's working fine for medimum amount of data. I got the OME(Out of memory error) for large data. I mean, suppose I am having like 20k records for web service method.So we are getting the OME while storing the data into the DB. I tried with profiler feature also but I am unable to find which object was consumed more heap memory. So I struck here. I need the somebody help to solve this.
Code
try {
JSONObject jSONObject = getJSONData(clientId, ApplicationConstants.REST_METHOD_GET_ALL_CATEGORIES_LIST_BY_CLIENT_ID);
if (jSONObject != null) {
JSONArray jSONArray = jSONObject.getJSONArray(ApplicationConstants.JSON_OBJECT_RESPONSE);
Integer length = jSONArray.length();
if (length == 0) {
if (_logger.isInfoEnabled()) {
_logger.info("Categories not found for client {}", clientId);
}
} else {
if (_logger.isInfoEnabled()) {
_logger.info("{} Categories found for client {}", length, clientId);
}
SyncDao synDao = (SyncDao) getDAO(ApplicationConstants.REST_SYNC_DAO);
for (int i = 0; i < length; i++) {
JSONObject jSONObj = jSONArray.getJSONObject(i);
ProjectMenu projectMenu = new ProjectMenu();
projectMenu.setClientId(clientId);
projectMenu.setCategory(ApplicationConstants.PROJECT_MENU_CATEGORY_SUB_MENU);
projectMenu.setCategoryType(ApplicationConstants.CATEGORY_TYPE_CATEGORY);
projectMenu.setCreatedDate(jSONObj.getString(ApplicationConstants.REST_CATEGORY_CREATED_DATE));
projectMenu.setElementId(jSONObj.getString(ApplicationConstants.REST_CATEGORY_ID));
projectMenu.setIsLeaf(Boolean.FALSE);
projectMenu.setLabel(jSONObj.getString(ApplicationConstants.REST_CATEGORY_NUMBER));
projectMenu.setLevel(new Integer(4));
projectMenu.setSortOrder(jSONObj.getInt(ApplicationConstants.REST_CATEGORY_SORT_ORDER));
// Getting parent menu details
ProjectMenu parentProjectMenu = synDao.getProjectMenu(clientId,
jSONObj.getString(ApplicationConstants.REST_CATEGORY_PROJECT_COMPONENT_ID), ApplicationConstants.CATEGORY_TYPE_COMPONENT);
if (parentProjectMenu != null) {
if (ApplicationConstants.REST_NO_COMPONENT_LABEL.equalsIgnoreCase(parentProjectMenu.getLabel())) {
projectMenu.setParentMenuId(parentProjectMenu.getParentMenuId());
} else {
projectMenu.setParentMenuId(parentProjectMenu.getId());
}
projectMenu.setProjectId(parentProjectMenu.getProjectId());
}
if (_logger.isDebugEnabled()) {
_logger.debug("Persisting Category: {}", projectMenu.toString());
}
synDao.persist(projectMenu);
}
}
}
} catch (JSONException jsonex) {
if (_logger.isErrorEnabled()) {
_logger.error("Exception while synchronizing Categories.", jsonex);
}
}
Use of MEMORY tables is often the cause for OOM errors when the amount of data in the database gets larger.
Convert the tables that hold large amounts of data to CACHED tables and there won't be any more OOM errors.
SET TABLE t TYPE CACHED
Related
We have an application that loads all contacts stored in an account using the Microsoft Graph API. The initial call we issue is https://graph.microsoft.com/v1.0/users/{userPrincipalName}/contacts$count=true&$orderBy=displayName%20ASC&$top=100, but we use the Java JDK to do that. Then we iterate over all pages and store all loaded contacts in a Set (local cache).
We do this every 5 minutes using an account with over 3000 contacts and sometimes, the count of contacts we received due to using $count does not match the number of contacts we loaded and stored in the local cache.
Verifying the numbers manually we can say, that the count was always correct, but there are contacts missing.
We use the following code to achieve this.
public List<Contact> loadContacts() {
Set<Contact> contacts = new TreeSet<>((contact1, contact2) -> StringUtils.compare(contact1.id, contact2.id));
List<QueryOption> requestOptions = List.of(
new QueryOption("$count", true),
new QueryOption("$orderBy", "displayName ASC"),
new QueryOption("$top", 100)
);
ContactCollectionRequestBuilder pageRequestBuilder = null;
ContactCollectionRequest pageRequest;
boolean hasNextPage = true;
while (hasNextPage) {
// initialize page request
if (pageRequestBuilder == null) {
pageRequestBuilder = graphClient.users(userId).contacts();
pageRequest = pageRequestBuilder.buildRequest(requestOptions);
} else {
pageRequest = pageRequestBuilder.buildRequest();
}
// load
ContactCollectionPage contactsPage = pageRequest.get();
if (contactsPage == null) {
throw new IllegalStateException("request returned a null page");
} else {
contacts.addAll(contactsPage.getCurrentPage());
}
// handle next page
hasNextPage = contactsPage.getNextPage() != null;
if (hasNextPage) {
pageRequestBuilder = contactsPage.getNextPage();
} else if (contactsPage.getCount() != null && !Objects.equals(contactsPage.getCount(), (long) contacts.size())) {
throw new IllegalStateException(String.format("loaded %d contacts but response indicated %d contacts", contacts.size(), contactsPage.getCount()));
} else {
// done
}
}
log.info("{} contacts loaded using graph API", contacts.size());
return new ArrayList<>(contacts);
}
Initially, we did not put the loaded contacts in a Set by ID but just in a List. With the List we very often got more contacts than $count. My idea was, that there is some caching going on and some pages get fetched multiple times. Using the Set we can make sure, that we only have unique contacts in our local cache.
But using the Set, we sometimes have less contacts than $count, meaning some pages got skipped and we end up in the condition that throws the IllegalStateException.
Currently, we use microsoft-graph 5.8.0 and azure-identiy 1.4.2.
Have you experienced similar issues and can help us solve this problem?
Or do you have any idea what could be causing these inconsistent results?
Your help is very much appreciated!
I'm continuing a project that has been coming for a few years at my university. One of the activities this project does is to collect some web pages using the google bot.
Due to a problem that I cannot understand, the project is not getting through this part. Already research a lot about what may be happening, if it is some part of the code that is outdated.
The code is in Java and uses Maven for project management.
I've tried to update some information from maven's "pom".
I already tried to change the part of the code that uses the bot, but nothing works.
I'm posting the part of code that isn't working as it should:
private List<JSONObject> querySearch(int numSeeds, String query) {
List<JSONObject> result = new ArrayList<>();
start=0;
do {
String url = SEARCH_URL + query.replaceAll(" ", "+") + FILE_TYPE + "html" + START + start;);
Connection conn = Jsoup.connect(url).userAgent("Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)").timeout(5000);
try {
Document doc = conn.get();
result.addAll(formatter(doc);
} catch (IOException e) {
System.err.println("Could not search for seed pages in IO.");
System.err.println(e);
} catch (ParseException e) {
System.err.println("Could not search for seed pages in Parse.");
System.err.println(e);
}
start += 10;
} while (result.size() < numSeeds);
return result;
}
what some variables do:
private static final String SEARCH_URL = "https://www.google.com/search?q=";
private static final String FILE_TYPE = "&fileType=";
private static final String START = "&start=";
private QueryBuilder queryBuilder;
public GoogleAjaxSearch() {
this.queryBuilder = new QueryBuilder();
}
Until this part is ok, it connect with the bot and can get a html from google. The problem is to separate what found and take only the link, that should be between ("h3.r> a").
That it does in this part with the result.addAll(formatter(doc)
public List<JSONObject> formatter(Document doc) throws ParseException {
List<JSONObject> entries = new ArrayList<>();
Elements results = doc.select("h3.r > a");
for (Element result : results) {
//System.out.println(result.toString());
JSONObject entry = new JSONObject();
entry.put("url", (result.attr("href").substring(6, result.attr("href").indexOf("&")).substring(1)));
entry.put("anchor", result.text());
So when it gets to this part: Elements results = doc.select ("h3.r> a"), find, probably, no h3 and can't increment the "results" list by not entering the for loop. Then goes back to the querysearch function and try again, without increment the results list. And with that, entering in a infinite loop trying to get the requested data and never finding.
If anyone here can help me, I've been trying for a while and I don't know what else to do. Thanks in advance.
Could someone help me out ow to get data from a table in the database by sending a list.
`
List<CampaignStructure> campIdList = new ArrayList<>(); try {
JSONArray clientjson = new JSONArray(campaignids);
Set<Long> list = new HashSet<>();
for(int i = 0; i < clientjson.length(); i++){
list.add(Long.parseLong(clientjson.getJSONObject(i).getString("id")));
}
for(Long id: list){
CampaignStructure c = new CampaignStructure();
c.setCampaignId(id);
campIdList.add(c);
}
} catch (JSONException e) {
}
Here campaignids contains list of ids (can contain duplicates).
campIdList contains distinct ids. i want to send these ids and get the data from database.
findByCampaignIdIn(List campIdList) should work.
You can find all list of JPA repository keywords can be found in the current documentation listing.
It shows that IsIn is equivalent – if you prefer the verb for readability – and that JPA also supports NotIn and IsNotIn.
My android app connects to Firebase and pulls "Alert Objects" that are sent there by my server.
When I export the data from Firebase, I get a beautifully formated JSON representation of the data.
Problem:
When I pull the data to my android device using a DataSnapshot, the data has '=' (equals signs) instead of ':' (semicolons). Also the quotations are not there.
When I try to do something like JSONObject alert = new JSONObject(data.getValue().toString()); I get errors for obvious reasons. I say obvious because if you look at what my code prints to the console you can see that it is no longer in valid JSON format.
A friend mentioned that I need to do something with encoding but we didn't have time to discuss it.
How can I iterate through these (kinda weird) Alert Objects that I have created and turn them into JSON objects within my Java so that I can access their properties like alert.date and alert.message.
I thought screenshots would help you see what I am doing. The firebase is not secured at all so you can feel free to take a look at it. It won't really do much and when I go to production I will be moving it anyways.
I am sure this is a super easy question to answer, I am just not too well versed with JSON and encoding as a whole.
Thanks!
you can it using gson library
public void onChildAdded(DataSnapshot dataSnapshot, String s) {
/*JSONObject jsonObject = null;
try {
jsonObject=new JSONObject();
} catch (JSONException e) {
e.printStackTrace();
}*/
Gson gson = new Gson();
String s1 = gson.toJson(dataSnapshot.getValue());
JSONArray object = null;
try {
object = new JSONArray(s1);
} catch (JSONException e) {
e.printStackTrace();
}
JSONArray jsonArray = object;
Log.e("string", s1);
}
You cannot access JSON natively in Java.
But the Firebase DataSnapshot class provides everything you need.
If you have a DataSnapshot of the data at the fbAlerts in your screenshot, you can print the date+message and recipients for each:
for (DataSnapshot alert: alerts.getChildren()) {
System.out.println(alert.child("date").getValue();
System.out.println(alert.child("message").getValue();
for (DataSnapshot recipient: alert.child("recipients").getChildren()) {
System.out.println(recipient.child("name").getValue();
}
}
Alternatively, you can build a Java class that represents an alert. See the Firebase guide on reading data in Android for examples of that.
Use this way to convert the jsonObject form the dataSnapshot
Map<String, String> value = (Map<String, String>) dataSnapshot.getValue();
Log.i("dataSnapshot", "dataSnapshot" + new JSONObject(value));
From the above code extract, it looks like you have a JSONArray rather than a JSONObject.
In which case, you need to do something like the following:
// Find the right array object
JSONArray jsonArray = response.getJSONArray("fbAlerts");
// Loop through the array
for (int i=0; i< jsonArray.length(); i++) {
JSONObject myObj = jsonArray.getJSONObject(i);
strMsg = myObj.getString("message");
}
In the example - when you have repeating groups, this would seem to indicate an array and therefore needs an iterator to access the object contents.
I'm using google maps to plot markers on a map. I can save the data for ALL these points (it's over 17000 rows with 3 columns: shopId,shopName,lat,long).
I can also send JSON queries specifying my lat/long and the radius at what shops around I want data about. Then I'll receive the data back. This works, but when I create the markers (with AsyncTask) freezing occurs in the app (and it is noticeable).
This is the code I'm using to generate the custom markers on Google maps:
#Override
protected void onPostExecute(String s) {
super.onPostExecute(s);
try {
JSONArray jsonArray = new JSONArray(result);
String finalReturn[] = result.split("\\r?\\n");
if(jsonArray.get(0).toString().equals("4")) {
for (int i = 1; i < finalReturn.length; i++) {
jsonArray = new JSONArray(finalReturn[i]);
IconGenerator iconGenerator = new IconGenerator(getApplicationContext());
iconGenerator.setStyle(IconGenerator.STYLE_RED);
iconGenerator.setRotation(90);
iconGenerator.setContentRotation(-90);
Bitmap iconBitmap = iconGenerator.makeIcon(jsonArray.get(5).toString());
Marker marker = mMap.addMarker(new MarkerOptions()
.position(new LatLng(jsonArray.getDouble(6), jsonArray.getDouble(7)))
.icon(BitmapDescriptorFactory.fromBitmap(iconBitmap)));
marker.setTitle(jsonArray.getString(1));
marker.setSnippet(jsonArray.getString(2) + " " + jsonArray.getString(8));
}
}
} catch (JSONException e) {
}
My question is, what is the best solution here, store the points in a MySQL server and generate nearest shops from that area (SQlite Getting nearest locations (with latitude and longitude) something like this), or always query the server for the data. Or maybe a hybrid of both (query the server, then save the data in an SQLite db.)
I'm only a beginner in Android so sorry if this question is simple.
The fastest way should be to save the data in an SQLite db and query it from there, but if you only need the few shops that are near the user, it should be fine to simply call the web service every time.
Other than that, the freezing that occurs in your app is most likely due to the onPostExecute Method being called in the UI-Thread and you doing heavy work in this method.
You should not parse your JSON there, but rather in the doInBackground method and for each parsed element call publishProgress that calls the onProgressUpdate Method (which is also executed in the UI-Thread.
Like this, you can handle setting one single marker on the map at a time and that way, the time between the single onProgressUpdate calls can be used by the system to update the UI and so the freezing should no longer occur.
It should look somewhat like this:
protected Void doInBackground(...) {
String result = getResult();
try {
JSONArray jsonArray = new JSONArray(result);
String finalReturn[] = result.split("\\r?\\n");
if(jsonArray.get(0).toString().equals("4")) {
for (int i = 1; i < finalReturn.length; i++) {
jsonArray = new JSONArray(finalReturn[i]);
publishProgress(jsonArray);
}
}
} catch (JSONException e) {
//handle error
}
}
#Override
protected void onProgressUpdate(JSONArray... progress) {
JSONArray array = progress[0];
IconGenerator iconGenerator = new IconGenerator(getApplicationContext());
iconGenerator.setStyle(IconGenerator.STYLE_RED);
iconGenerator.setRotation(90);
iconGenerator.setContentRotation(-90);
Bitmap iconBitmap = iconGenerator.makeIcon(jsonArray.get(5).toString());
Marker marker = mMap.addMarker(new MarkerOptions()
.position(new LatLng(jsonArray.getDouble(6), jsonArray.getDouble(7)))
.icon(BitmapDescriptorFactory.fromBitmap(iconBitmap)));
marker.setTitle(jsonArray.getString(1));
marker.setSnippet(jsonArray.getString(2) + " " + jsonArray.getString(8));
}