We have an application that loads all contacts stored in an account using the Microsoft Graph API. The initial call we issue is https://graph.microsoft.com/v1.0/users/{userPrincipalName}/contacts$count=true&$orderBy=displayName%20ASC&$top=100, but we use the Java JDK to do that. Then we iterate over all pages and store all loaded contacts in a Set (local cache).
We do this every 5 minutes using an account with over 3000 contacts and sometimes, the count of contacts we received due to using $count does not match the number of contacts we loaded and stored in the local cache.
Verifying the numbers manually we can say, that the count was always correct, but there are contacts missing.
We use the following code to achieve this.
public List<Contact> loadContacts() {
Set<Contact> contacts = new TreeSet<>((contact1, contact2) -> StringUtils.compare(contact1.id, contact2.id));
List<QueryOption> requestOptions = List.of(
new QueryOption("$count", true),
new QueryOption("$orderBy", "displayName ASC"),
new QueryOption("$top", 100)
);
ContactCollectionRequestBuilder pageRequestBuilder = null;
ContactCollectionRequest pageRequest;
boolean hasNextPage = true;
while (hasNextPage) {
// initialize page request
if (pageRequestBuilder == null) {
pageRequestBuilder = graphClient.users(userId).contacts();
pageRequest = pageRequestBuilder.buildRequest(requestOptions);
} else {
pageRequest = pageRequestBuilder.buildRequest();
}
// load
ContactCollectionPage contactsPage = pageRequest.get();
if (contactsPage == null) {
throw new IllegalStateException("request returned a null page");
} else {
contacts.addAll(contactsPage.getCurrentPage());
}
// handle next page
hasNextPage = contactsPage.getNextPage() != null;
if (hasNextPage) {
pageRequestBuilder = contactsPage.getNextPage();
} else if (contactsPage.getCount() != null && !Objects.equals(contactsPage.getCount(), (long) contacts.size())) {
throw new IllegalStateException(String.format("loaded %d contacts but response indicated %d contacts", contacts.size(), contactsPage.getCount()));
} else {
// done
}
}
log.info("{} contacts loaded using graph API", contacts.size());
return new ArrayList<>(contacts);
}
Initially, we did not put the loaded contacts in a Set by ID but just in a List. With the List we very often got more contacts than $count. My idea was, that there is some caching going on and some pages get fetched multiple times. Using the Set we can make sure, that we only have unique contacts in our local cache.
But using the Set, we sometimes have less contacts than $count, meaning some pages got skipped and we end up in the condition that throws the IllegalStateException.
Currently, we use microsoft-graph 5.8.0 and azure-identiy 1.4.2.
Have you experienced similar issues and can help us solve this problem?
Or do you have any idea what could be causing these inconsistent results?
Your help is very much appreciated!
I'm sorry this question header is not 100% correct. Because of that, I'll explain my scenario here.
I created a function to merge 4 data sets into one return format. Because that's the format front-end side needed. So this is working fine now.
public ReturnFormat makeThribleLineChart(List<NameCountModel> totalCount, List<NameCountModel>,p1Count, List<NameCountModel> p2Count, List<NameCountModel> average) {
ReturnFormat returnFormat = new ReturnFormat(null,null);
try {
String[] totalData = new String[totalCount.size()];
String[] p1Data = new String[p1Count.size()];
String[] p2Data = new String[p2Count.size()];
String[] averageData = new String[p2Count.size()];
String[] lableList = new String[totalCount.size()];
for (int x = 0; x < totalCount.size(); x++) {
totalData[x] = totalCount.get(x).getCount();
p1Data[x] = p1Count.get(x).getCount();
p2Data[x] = p2Count.get(x).getCount();
averageData[x] = average.get(x).getCount();
lableList[x] = totalCount.get(x).getName();
}
FormatHelper<String[]> totalFormatHelper= new FormatHelper<String[]>();
totalFormatHelper.setData(totalData);
totalFormatHelper.setType("line");
totalFormatHelper.setLabel("Uudet");
totalFormatHelper.setyAxisID("y-axis-1");
FormatHelper<String[]> p1FormatHelper= new FormatHelper<String[]>();
p1FormatHelper.setData(p1Data);
p1FormatHelper.setType("line");
p1FormatHelper.setLabel("P1 päivystykseen heti");
FormatHelper<String[]> p2FormatHelper= new FormatHelper<String[]>();
p2FormatHelper.setData(p2Data);
p2FormatHelper.setType("line");
p2FormatHelper.setLabel("P2 päivystykseen muttei yöllä");
FormatHelper<String[]> averageFormatHelper= new FormatHelper<String[]>();
averageFormatHelper.setData(averageData);
averageFormatHelper.setType("line");
averageFormatHelper.setLabel("Jonotusaika keskiarvo");
averageFormatHelper.setyAxisID("y-axis-2");
List<FormatHelper<String[]>> formatHelpObj = new ArrayList<FormatHelper<String[]>>();
formatHelpObj.add(totalFormatHelper);
formatHelpObj.add(p1FormatHelper);
formatHelpObj.add(p2FormatHelper);
formatHelpObj.add(averageFormatHelper);
returnFormat.setData(formatHelpObj);
returnFormat.setLabels(lableList);
returnFormat.setMessage(Messages.Success);
returnFormat.setStatus(ReturnFormat.Status.SUCCESS);
} catch (Exception e) {
returnFormat.setData(null);
returnFormat.setMessage(Messages.InternalServerError);
returnFormat.setStatus(ReturnFormat.Status.ERROR);
}
return returnFormat;
}
so, as you can see here, all the formatting is hardcoded. So my question is how to automate this code for list count. Let's assume next time I have to create chart formatting for five datasets. So I have to create another function to it. That's the thing I want to reduce. So I hope you can understand my question.
Thank you.
You're trying to solve the more general problem of composing a result object (in this case ReturnFormat) based on dynamic information. In addition, there's some metadata being setup along with each dataset - the type, label, etc. In the example that you've posted, you've hardcoded the relationship between a dataset and this metadata, but you'd need some way to establish this relationship for data dynamically if you have a variable number of parameters here.
Therefore, you have a couple of options:
Make makeThribleLineChart a varargs method to accept a variable number of parameters representing your data. Now you have the problem of associating metadata with your parameters - best option is probably to wrap the data and metadata together in some new object that is provided as each param of makeThribleLineChart.
So you'll end up with a signature that looks a bit like ReturnFormat makeThribleLineChart(DataMetadataWrapper... allDatasets), where DataMetadataWrapper contains everything required to build one FormatHelper instance.
Use a builder pattern, similar to the collection builders in guava, for example something like so:
class ThribbleLineChartBuilder {
List<FormatHelper<String[]>> formatHelpObj = new ArrayList<>();
ThribbleLineChartBuilder addDataSet(String describeType, String label, String yAxisId, List<NameCountModel> data) {
String[] dataArray = ... ; // build your array of data
FormatHelper<String[]> formatHelper = new FormatHelper<String[]>();
formatHelper.setData(dataArray);
formatHelper.setType(describeType);
... // set any other parameters that the FormatHelper requires here
formatHelpObj.add(formatHelper);
return this;
}
ReturnFormat build() {
ReturnFormat returnFormat = new ReturnFormat(null, null);
returnFormat.setData(this.formatHelpObj);
... // setup any other fields you need in ReturnFormat
return returnFormat;
}
}
// usage:
new ThribbleLineChartBuilder()
.addDataSet("line", "Uudet", "y-axis-1", totalCount)
.addDataSet("line", "P1 päivystykseen heti", null, p1Count)
... // setup your other data sources
.build()
I try to get all the live videos of youtube api at a given time.
However, after the two firsts pages that contains 50 results each, all the others pages are empty.
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/7Mn5oZUWkgMIU4kfKx7cq0D0nDI\"","items":[{THIS ONE IS OK WITH 50 RESULTS}],"kind":"youtube#searchListResponse","nextPageToken":"CDIQAA","pageInfo":{"resultsPerPage":50,"totalResults":1994},"regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/wp4SIiAGHN1uLiJgvLGoMdQoehs\"","items":[{THIS ONE IS OK TOO WITH 50 RESULTS}],"kind":"youtube#searchListResponse","nextPageToken":"CGQQAA","pageInfo":{"resultsPerPage":50,"totalResults":1994},"prevPageToken":"CDIQAQ","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/3zCHC7x--dhww0e7udfLQn3DB5Y\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CJYBEAA","pageInfo":{"resultsPerPage":50,"totalResults":1988},"prevPageToken":"CGQQAQ","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/pGfYwQfcIDdD05OBzqTVFbdg39E\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CMgBEAA","pageInfo":{"resultsPerPage":50,"totalResults":1986},"prevPageToken":"CJYBEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/qtrckZRTf7czKQNL5nQpXRu7X5A\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CPoBEAA","pageInfo":{"resultsPerPage":50,"totalResults":1985},"prevPageToken":"CMgBEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/gTY6oWopjQcy6Cuf1MGQqYGYfog\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CKwCEAA","pageInfo":{"resultsPerPage":50,"totalResults":1993},"prevPageToken":"CPoBEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/umiznrtjy7vmWkIJ5Hsm8wtU0W0\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CN4CEAA","pageInfo":{"resultsPerPage":50,"totalResults":1985},"prevPageToken":"CKwCEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/thJws1H1pJJPtOkVeq6L6VGabn8\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CJADEAA","pageInfo":{"resultsPerPage":50,"totalResults":1988},"prevPageToken":"CN4CEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/TaDYlGeeLP2H1xanLHVTtmt_DNg\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CMIDEAA","pageInfo":{"resultsPerPage":50,"totalResults":1995},"prevPageToken":"CJADEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/w3wj8nt1NHhxyh-fuWLJ1v5ncvU\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CPQDEAA","pageInfo":{"resultsPerPage":50,"totalResults":1989},"prevPageToken":"CMIDEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/r1hbVDG2ANenKRichmZxj0J2dws\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CKYEEAA","pageInfo":{"resultsPerPage":50,"totalResults":1996},"prevPageToken":"CPQDEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/hdDkPLk2aNKrJLZ8tnMnCGqhUy8\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CNgEEAA","pageInfo":{"resultsPerPage":50,"totalResults":1994},"prevPageToken":"CKYEEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/OjWuZOr5rNl_HasCmAEhF5cAN9E\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CIoFEAA","pageInfo":{"resultsPerPage":50,"totalResults":1995},"prevPageToken":"CNgEEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/g8lxb3uVDwSfamhbbxFlyoVKBTg\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CLwFEAA","pageInfo":{"resultsPerPage":50,"totalResults":1986},"prevPageToken":"CIoFEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/wnTWwe1b-0gy--2ochuizsAjD9o\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CO4FEAA","pageInfo":{"resultsPerPage":50,"totalResults":1993},"prevPageToken":"CLwFEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/N1y4dp3ngAItfwukyLLitFQFDPA\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CKAGEAA","pageInfo":{"resultsPerPage":50,"totalResults":1981},"prevPageToken":"CO4FEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/F8oNrDOjlKMvHjzPIF7cJqfK_zo\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CNIGEAA","pageInfo":{"resultsPerPage":50,"totalResults":1986},"prevPageToken":"CKAGEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/JCWNr1NCzgl0GLh66R-SHeokbW8\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CIQHEAA","pageInfo":{"resultsPerPage":50,"totalResults":1992},"prevPageToken":"CNIGEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/uTbvXolYwtv5Zt4mpLQVHCQsbF4\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CLYHEAA","pageInfo":{"resultsPerPage":50,"totalResults":1999},"prevPageToken":"CIQHEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/xxkZZRO_iCVgmWLAsBW4Qk6VdcU\"","items":[],"kind":"youtube#searchListResponse","pageInfo":{"resultsPerPage":50,"totalResults":1995},"prevPageToken":"CLYHEAE","regionCode":"FR"}
Here is the code used :
List<SearchResult> searchItemList = new ArrayList<>();
YouTube.Search.List searchListRequest = youTubeService.search().list("snippet");
searchListRequest.setMaxResults(NUMBER_OF_VIDEOS_RETURNED);
searchListRequest.setEventType("live");
searchListRequest.setType("video");
searchListRequest.setVideoCategoryId(GAMING_VIDEO_CATEGORY);
searchListRequest.setOrder("viewCount");
searchListRequest.setSafeSearch("none");
searchListRequest.setPrettyPrint(true);
String nextToken = "";
do {
if (!nextToken.isEmpty()) {
searchListRequest.setPageToken(nextToken);
}
SearchListResponse searchListResponse = searchListRequest.execute();
System.out.println(searchListResponse);
if (searchListResponse.getItems().size() > 0) {
searchItemList.addAll(searchListResponse.getItems());
}
nextToken = searchListResponse.getNextPageToken();
} while (nextToken != null);
I don't get why the third to last request "items" field is empty. Are there some kind of restrictions from Youtube API?
I am learning Amazon Cloud Search but I couldn't find any code in either C# or Java (though I am creating in C# but if I can get code in Java then I can try converting in C#).
This is just 1 code I found in C#: https://github.com/Sitefinity-SDK/amazon-cloud-search-sample/tree/master/SitefinityWebApp.
This is 1 method i found in this code:
public IResultSet Search(ISearchQuery query)
{
AmazonCloudSearchDomainConfig config = new AmazonCloudSearchDomainConfig();
config.ServiceURL = "http://search-index2-cdduimbipgk3rpnfgny6posyzy.eu-west-1.cloudsearch.amazonaws.com/";
AmazonCloudSearchDomainClient domainClient = new AmazonCloudSearchDomainClient("AKIAJ6MPIX37TLIXW7HQ", "DnrFrw9ZEr7g4Svh0rh6z+s3PxMaypl607eEUehQ", config);
SearchRequest searchRequest = new SearchRequest();
List<string> suggestions = new List<string>();
StringBuilder highlights = new StringBuilder();
highlights.Append("{\'");
if (query == null)
throw new ArgumentNullException("query");
foreach (var field in query.HighlightedFields)
{
if (highlights.Length > 2)
{
highlights.Append(", \'");
}
highlights.Append(field.ToUpperInvariant());
highlights.Append("\':{} ");
SuggestRequest suggestRequest = new SuggestRequest();
Suggester suggester = new Suggester();
suggester.SuggesterName = field.ToUpperInvariant() + "_suggester";
suggestRequest.Suggester = suggester.SuggesterName;
suggestRequest.Size = query.Take;
suggestRequest.Query = query.Text;
SuggestResponse suggestion = domainClient.Suggest(suggestRequest);
foreach (var suggest in suggestion.Suggest.Suggestions)
{
suggestions.Add(suggest.Suggestion);
}
}
highlights.Append("}");
if (query.Filter != null)
{
searchRequest.FilterQuery = this.BuildQueryFilter(query.Filter);
}
if (query.OrderBy != null)
{
searchRequest.Sort = string.Join(",", query.OrderBy);
}
if (query.Take > 0)
{
searchRequest.Size = query.Take;
}
if (query.Skip > 0)
{
searchRequest.Start = query.Skip;
}
searchRequest.Highlight = highlights.ToString();
searchRequest.Query = query.Text;
searchRequest.QueryParser = QueryParser.Simple;
var result = domainClient.Search(searchRequest).SearchResult;
//var result = domainClient.Search(searchRequest).SearchResult;
return new AmazonResultSet(result, suggestions);
}
I have already created domain in Amazon Cloud Search using AWS console and uploaded document using Amazon predefine configuration option that is movie Imdb json file provided by Amazon for demo.
But in this method I am not getting how to use this method, like if I want to search Director name then how do I pass in this method as because this method parameter is of type ISearchQuery?
I'd suggest using the official AWS CloudSearch .NET SDK. The library you were looking at seems fine (although I haven't look at it any detail) but the official version is more likely to expose new CloudSearch features as soon as they're released, will be supported if you need to talk to AWS support, etc, etc.
Specifically, take a look at the SearchRequest class -- all its params are strings so I think that obviates your question about ISearchQuery.
I wasn't able to find an example of a query in .NET but this shows someone uploading docs using the AWS .NET SDK. It's essentially the same procedure as querying: creating and configuring a Request object and passing it to the client.
EDIT:
Since you're still having a hard time, here's an example. Bear in mind that I am unfamiliar with C# and have not attempted to run or even compile this but I think it should at least be close to working. It's based off looking at the docs at http://docs.aws.amazon.com/sdkfornet/v3/apidocs/
// Configure the Client that you'll use to make search requests
string queryUrl = #"http://search-<domainname>-xxxxxxxxxxxxxxxxxxxxxxxxxx.us-east-1.cloudsearch.amazonaws.com";
AmazonCloudSearchDomainClient searchClient = new AmazonCloudSearchDomainClient(queryUrl);
// Configure a search request with your query
SearchRequest searchRequest = new SearchRequest();
searchRequest.Query = "potato";
// TODO Set your other params like parser, suggester, etc
// Submit your request via the client and get back a response containing search results
SearchResponse searchResponse = searchClient.Search(searchRequest);
So this is the case: I have a program that takes two large csv-files, finds the diffs and then sends a array list to a method that is supposed to update the mongodb with the lines from the array. The problem is the updates are taking forever. A test case with 5000 updates takes 36 minutes. Is this normal?
the update(List<String> changes)-method something like this:
mongoClient = new MongoClient(ip);
db = mongoClient.getDB("foo");
collection = db.getCollection("bar");
//for each line of change
for (String s : changes) {
//splits the csv-lines on ;
String[] fields = s.split(";");
//identifies wich document in the database to be updated
long id = Long.parseLong(fields[0]);
BasicDBObject sq = new BasicDBObject().append("organizationNumber",id);
//creates a new unit-object, that is converted to JSON and then inserted into the database.
Unit u = new Unit(fields);
Gson gson = new Gson();
String jsonObj = gson.toJson(u);
DBObject objectToUpdate = collection.findOne(sq);
DBObject newObject = (DBObject) JSON.parse(jsonObj);
if(objectToUpdate != null){
objectToUpdate.putAll(newObject);
collection.save(objectToUpdate);
}
That's because you are taking extra steps to update.
You don't need to parse JSONs manually and you don't have to do the query-then-update when you can just do an update with a "where" clause in a single step.
Something like this:
BasicDBObject query= new BasicDBObject().append("organizationNumber",id);
Unit unit = new Unit(fields);
BasicDBObject unitDB= new BasicDBObject().append("someField",unit.getSomeField()).append("otherField",unit.getOtherField());
collection.update(query,unitDB);
Where query specifies the "where" clause and unitDB specifies the fields that need to be updated.