We Keep Coding

Loading all contacts using the Microsoft Graph API sometimes looses/skips pages

We have an application that loads all contacts stored in an account using the Microsoft Graph API. The initial call we issue is https://graph.microsoft.com/v1.0/users/{userPrincipalName}/contacts$count=true&$orderBy=displayName%20ASC&$top=100, but we use the Java JDK to do that. Then we iterate over all pages and store all loaded contacts in a Set (local cache).
We do this every 5 minutes using an account with over 3000 contacts and sometimes, the count of contacts we received due to using $count does not match the number of contacts we loaded and stored in the local cache.
Verifying the numbers manually we can say, that the count was always correct, but there are contacts missing.
We use the following code to achieve this.
public List<Contact> loadContacts() {
Set<Contact> contacts = new TreeSet<>((contact1, contact2) -> StringUtils.compare(contact1.id, contact2.id));
List<QueryOption> requestOptions = List.of(
new QueryOption("$count", true),
new QueryOption("$orderBy", "displayName ASC"),
new QueryOption("$top", 100)
);
ContactCollectionRequestBuilder pageRequestBuilder = null;
ContactCollectionRequest pageRequest;
boolean hasNextPage = true;
while (hasNextPage) {
// initialize page request
if (pageRequestBuilder == null) {
pageRequestBuilder = graphClient.users(userId).contacts();
pageRequest = pageRequestBuilder.buildRequest(requestOptions);
} else {
pageRequest = pageRequestBuilder.buildRequest();
}
// load
ContactCollectionPage contactsPage = pageRequest.get();
if (contactsPage == null) {
throw new IllegalStateException("request returned a null page");
} else {
contacts.addAll(contactsPage.getCurrentPage());
}
// handle next page
hasNextPage = contactsPage.getNextPage() != null;
if (hasNextPage) {
pageRequestBuilder = contactsPage.getNextPage();
} else if (contactsPage.getCount() != null && !Objects.equals(contactsPage.getCount(), (long) contacts.size())) {
throw new IllegalStateException(String.format("loaded %d contacts but response indicated %d contacts", contacts.size(), contactsPage.getCount()));
} else {
// done
}
}
log.info("{} contacts loaded using graph API", contacts.size());
return new ArrayList<>(contacts);
}
Initially, we did not put the loaded contacts in a Set by ID but just in a List. With the List we very often got more contacts than $count. My idea was, that there is some caching going on and some pages get fetched multiple times. Using the Set we can make sure, that we only have unique contacts in our local cache.
But using the Set, we sometimes have less contacts than $count, meaning some pages got skipped and we end up in the condition that throws the IllegalStateException.
Currently, we use microsoft-graph 5.8.0 and azure-identiy 1.4.2.
Have you experienced similar issues and can help us solve this problem?
Or do you have any idea what could be causing these inconsistent results?
Your help is very much appreciated!

Ldap search by email value can't find records

I have written a Java application that searches Active directory via LDAP for user information. I have a list of instances of custom Person class that is passed in. In it I have either DN or email defined. I am modifying the search criteria accordingly. Here is the code:
for (Person person : members) {
boolean ready = false;
String filter = getConfig().getUserSearchFilter();
// (&(|(objectclass=user)(objectclass=person)(objectclass=inetOrgPerson)(objectclass=organizationalPerson)))
String base = person.getDistinguishedName();
if (base != null && !base.isEmpty()) {
ready = true;
} else if (person.getEmail() != null) {
base = getConfig().getMemberSearchBase();
// ou=Users,ou=Managed,dc=division,dc=company,dc=com
String mail = person.getEmail();
StringBuilder filterBuilder = new StringBuilder(filter);
int pIdx = filterBuilder.lastIndexOf(")");
filterBuilder.insert(pIdx, "(|(mail=" + mail + ")(x-personalmail=" + mail + "))");
filter = filterBuilder.toString();
LOG.debug("New value of a filter = {}", filter);
ready = true;
}
if (ready) {
try {
NamingEnumeration<SearchResult> search = getContext().search(base, filter, searchControls);
...
} catch (NamingException nex) {
throw new IOException(nex);
}
} else {
LOG.error("Incorrect search criteria for user {} of group {}. Person skipped", person.getName(), this.group.getName());
}
}
Code is working without errors, but when DN is specified it does find a person, but when email is defined it finds nothing.
However, If I copy generated filter string and pass it to ldapsearch command in a form of:
ldapsearch -LLL -x -H ldaps://my.ldap.server.com -D 'svc-acct#corp-dev.company.com' -W -b "ou=Users,ou=Managed,dc=division,dc=company,dc=com" '(&(|(objectclass=user)(objectclass=person)(objectclass=inetOrgPerson)(objectclass=organizationalPerson))(|(mail=person#domain.com)(x-personalmail=person#domain.com)))'
It does find this person perfectly.
Did anyone faced similar problem? Do you see any flaws in my code?
Please, do help me.

I did find the cause of my problem.
In the search control I had scope defined as OBJECT_SCOPE.
It does work when you are specifying DN, but with the search per one of the fields it fails finding the object.
I changed the scope to SUBTREE_SCOPE and everything started working as expected.

Best way to load a large amount of data in android without making the user wait

I'm working in an application that read phone contacts and use them in my application (Call history, Favorite contacts and All contacts).
My UI consist of tab host control and user can swap between them, as I want my data to be shared across all my activities and also to be saved in only one place.
So I have created a singleton class called data controller, and when I open the application I show loading screen until all data loaded.
The problem now that user is complaining because of waiting a lot of time about (1 minute) every time they open the application when he has a very large amount of contacts, so how can I optimize my code in a good way?
EDIT
This is the method that I'm using to get all contacts:
public static ArrayList<ContactInfo> getAllContactWithNumberAndNameAndPhoto(
Context context, boolean starred) {
ArrayList<ContactInfo> retList = new ArrayList<ContactInfo>();
ContentResolver cr = context.getContentResolver();
Cursor cur = null;
if (starred == true) {
cur = cr.query(ContactsContract.Contacts.CONTENT_URI, null,
"starred=?", new String[] { "1" }, null);
} else {
cur = cr.query(ContactsContract.Contacts.CONTENT_URI, null, null,
null, null);
}
if (cur.getCount() > 0) {
while (cur.moveToNext()) {
ContactInfo item = new ContactInfo();
String id = cur.getString(cur
.getColumnIndex(ContactsContract.Contacts._ID));
String name = cur
.getString(cur
.getColumnIndex(ContactsContract.Contacts.DISPLAY_NAME));
Uri photo = PhoneUtils.getPhotoUriFromID(context, id);
String starredValue = cur.getString(cur
.getColumnIndex(ContactsContract.Contacts.STARRED));
boolean isFav = false;
if (starredValue.equals("1"))
isFav = true;
if (Integer
.parseInt(cur.getString(cur
.getColumnIndex(ContactsContract.Contacts.HAS_PHONE_NUMBER))) > 0) {
Cursor pCur = cr.query(
ContactsContract.CommonDataKinds.Phone.CONTENT_URI,
null,
ContactsContract.CommonDataKinds.Phone.CONTACT_ID
+ " = ?", new String[] { id }, null);
while (pCur.moveToNext()) {
String phoneNo = pCur
.getString(pCur
.getColumnIndex(ContactsContract.CommonDataKinds.Phone.NUMBER));
item.addPhone(removeCharactersFromPhoneNumber(phoneNo));
}
pCur.close();
if (photo != null) {
item.setPhoto(photo.toString());
}
item.setName(name);
item.setFavorite(isFav);
item.setRecent(false);
retList.add(item);
}
}
cur.close();
}
return retList;
}
Please let me know if I can optimize this method.

Im surprised it takes that long to load the contacts from the device.
Have you profiled the app to see where the time is actually spent? Something seems wrong here.
If it truly takes that long to load from the system providers (due to the OS) you could cache the results (i.e. put in your own SQL db) so you can load quickly on each app visit (< 1 sec) and refresh from device in the background.

I guess that the bottleneck of your method is photo loading. Try to load everything except photos, and then show your activity, concurrently loading photos.
Also you can try to create your own app table that contains just the data you need. So you'll do less selects while loading contacts. But you would have to synchronize your table. You can do so concurrently.

Crash when executing cursor.getCount() (Android)

this doesn't always happen so i can't understand properly what's going on: my application take and modify a picture, then save it in external storage. If i try to open in the application a new saved pictures FROM A FILE MANAGER AND NOT FROM GALLERY, it crashes when executing cursor.getCount(), in DDMS i read the error:"cursor not closed before finally" this is the piece of code where the problem is, i can post more if necessary, thank you!
ps this code is taken from other answers here in stackoverflow, as you could expect i'm not an expert so please be patient with me, thanks
pps i can't see immediatly images in gallery after saving it, when they appear in gallery this error desappear.
public static int getOrientation(Context context, Uri photoUri) {
/* it's on the external media. */
Cursor cursor = context.getContentResolver().query(photoUri,
new String[] { MediaStore.Images.ImageColumns.ORIENTATION }, null, null, null);
if (cursor.getCount() != 1) { //HERE IS THE PROBLEM
return -1;
}
cursor.moveToFirst();
return cursor.getInt(0);
}

You can use the
if(cursor.moveToFirst())
{
//Your code here
}
instead off
cursor.getCount()
it will return true if cursor size is greater then 0 else it will return false........so you can write like this.........
if (!cursor.moveToFirst())
return -1;
else
return 1;

Use this instead of the return statement. The cursor is getting leaked since you are not closing it
try{
if (cursor.getCount() != 1) { //HERE IS THE PROBLEM
return -1;
}
int i = 0;
i++;
cursor.moveToFirst();
return cursor.getInt(0);
}finally{
if(cursor != null)
cursor.close();
}
Edit:
When you open a file from File manager, the uri will be of the form file:///sdcard/filename but the Mediastore can only understand uri of the format content://media/audio/1. This is the reason you are getting cursor as null.
one way is to query the whole Mediastore and get the MediaStore.Images.Media.DATA column and compare with the path that you get from uri

Controlling the list of URL(s) to be crawled at runtime

In crawler4j we can override a function boolean shouldVisit(WebUrl url) and control whether that particular url should be allowed to be crawled by returning 'true' and 'false'.
But can we add URL(s) at runtime ? if yes , what are ways to do that ?
Currently I can add URL(s) at beginning of program using addSeed(String url) function before the start(BasicCrawler.class, numberOfCrawlers) in CrawlController class and if I try to add new url using addSeed(String url), it gives error. Here is error image .
Any help will be appreciative and please let me know if any more detail about project is required to answer the question .

You can do this.
Use public void schedule(WebURL url) to add URLs to the crawler frontier which is a member of the Frontier.java class. But for this you need to have your url of type WebURL. If you want to make a WebURL out of your string. Please have a look at the addSeed() (below code) which is in the CrawlController.java class to see how it has converted the string (url) into a WebURL.
Also use the existing frontier instance.
Hope this helps..
public void addSeed(String pageUrl, int docId) {
String canonicalUrl = URLCanonicalizer.getCanonicalURL(pageUrl);
if (canonicalUrl == null) {
logger.error("Invalid seed URL: " + pageUrl);
return;
}
if (docId < 0) {
docId = docIdServer.getDocId(canonicalUrl);
if (docId > 0) {
// This URL is already seen.
return;
}
docId = docIdServer.getNewDocID(canonicalUrl);
} else {
try {
docIdServer.addUrlAndDocId(canonicalUrl, docId);
} catch (Exception e) {
logger.error("Could not add seed: " + e.getMessage());
}
}
WebURL webUrl = new WebURL();
webUrl.setURL(canonicalUrl);
webUrl.setDocid(docId);
webUrl.setDepth((short) 0);
if (!robotstxtServer.allows(webUrl)) {
logger.info("Robots.txt does not allow this seed: " + pageUrl);
} else {
frontier.schedule(webUrl); //method that adds URL to the frontier at run time
}
}

Presumably you can implement this function however you like, and have it depend on a list of URLs that should not be crawled. The implementation of shouldVisit is then going to involve asking if a given URL is in your list of forbidden URLs (or permitted URLs), and returning true or false on that basis.