We have an application that loads all contacts stored in an account using the Microsoft Graph API. The initial call we issue is https://graph.microsoft.com/v1.0/users/{userPrincipalName}/contacts$count=true&$orderBy=displayName%20ASC&$top=100, but we use the Java JDK to do that. Then we iterate over all pages and store all loaded contacts in a Set (local cache).
We do this every 5 minutes using an account with over 3000 contacts and sometimes, the count of contacts we received due to using $count does not match the number of contacts we loaded and stored in the local cache.
Verifying the numbers manually we can say, that the count was always correct, but there are contacts missing.
We use the following code to achieve this.
public List<Contact> loadContacts() {
Set<Contact> contacts = new TreeSet<>((contact1, contact2) -> StringUtils.compare(contact1.id, contact2.id));
List<QueryOption> requestOptions = List.of(
new QueryOption("$count", true),
new QueryOption("$orderBy", "displayName ASC"),
new QueryOption("$top", 100)
);
ContactCollectionRequestBuilder pageRequestBuilder = null;
ContactCollectionRequest pageRequest;
boolean hasNextPage = true;
while (hasNextPage) {
// initialize page request
if (pageRequestBuilder == null) {
pageRequestBuilder = graphClient.users(userId).contacts();
pageRequest = pageRequestBuilder.buildRequest(requestOptions);
} else {
pageRequest = pageRequestBuilder.buildRequest();
}
// load
ContactCollectionPage contactsPage = pageRequest.get();
if (contactsPage == null) {
throw new IllegalStateException("request returned a null page");
} else {
contacts.addAll(contactsPage.getCurrentPage());
}
// handle next page
hasNextPage = contactsPage.getNextPage() != null;
if (hasNextPage) {
pageRequestBuilder = contactsPage.getNextPage();
} else if (contactsPage.getCount() != null && !Objects.equals(contactsPage.getCount(), (long) contacts.size())) {
throw new IllegalStateException(String.format("loaded %d contacts but response indicated %d contacts", contacts.size(), contactsPage.getCount()));
} else {
// done
}
}
log.info("{} contacts loaded using graph API", contacts.size());
return new ArrayList<>(contacts);
}
Initially, we did not put the loaded contacts in a Set by ID but just in a List. With the List we very often got more contacts than $count. My idea was, that there is some caching going on and some pages get fetched multiple times. Using the Set we can make sure, that we only have unique contacts in our local cache.
But using the Set, we sometimes have less contacts than $count, meaning some pages got skipped and we end up in the condition that throws the IllegalStateException.
Currently, we use microsoft-graph 5.8.0 and azure-identiy 1.4.2.
Have you experienced similar issues and can help us solve this problem?
Or do you have any idea what could be causing these inconsistent results?
Your help is very much appreciated!
I try to get all the live videos of youtube api at a given time.
However, after the two firsts pages that contains 50 results each, all the others pages are empty.
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/7Mn5oZUWkgMIU4kfKx7cq0D0nDI\"","items":[{THIS ONE IS OK WITH 50 RESULTS}],"kind":"youtube#searchListResponse","nextPageToken":"CDIQAA","pageInfo":{"resultsPerPage":50,"totalResults":1994},"regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/wp4SIiAGHN1uLiJgvLGoMdQoehs\"","items":[{THIS ONE IS OK TOO WITH 50 RESULTS}],"kind":"youtube#searchListResponse","nextPageToken":"CGQQAA","pageInfo":{"resultsPerPage":50,"totalResults":1994},"prevPageToken":"CDIQAQ","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/3zCHC7x--dhww0e7udfLQn3DB5Y\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CJYBEAA","pageInfo":{"resultsPerPage":50,"totalResults":1988},"prevPageToken":"CGQQAQ","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/pGfYwQfcIDdD05OBzqTVFbdg39E\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CMgBEAA","pageInfo":{"resultsPerPage":50,"totalResults":1986},"prevPageToken":"CJYBEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/qtrckZRTf7czKQNL5nQpXRu7X5A\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CPoBEAA","pageInfo":{"resultsPerPage":50,"totalResults":1985},"prevPageToken":"CMgBEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/gTY6oWopjQcy6Cuf1MGQqYGYfog\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CKwCEAA","pageInfo":{"resultsPerPage":50,"totalResults":1993},"prevPageToken":"CPoBEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/umiznrtjy7vmWkIJ5Hsm8wtU0W0\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CN4CEAA","pageInfo":{"resultsPerPage":50,"totalResults":1985},"prevPageToken":"CKwCEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/thJws1H1pJJPtOkVeq6L6VGabn8\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CJADEAA","pageInfo":{"resultsPerPage":50,"totalResults":1988},"prevPageToken":"CN4CEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/TaDYlGeeLP2H1xanLHVTtmt_DNg\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CMIDEAA","pageInfo":{"resultsPerPage":50,"totalResults":1995},"prevPageToken":"CJADEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/w3wj8nt1NHhxyh-fuWLJ1v5ncvU\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CPQDEAA","pageInfo":{"resultsPerPage":50,"totalResults":1989},"prevPageToken":"CMIDEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/r1hbVDG2ANenKRichmZxj0J2dws\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CKYEEAA","pageInfo":{"resultsPerPage":50,"totalResults":1996},"prevPageToken":"CPQDEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/hdDkPLk2aNKrJLZ8tnMnCGqhUy8\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CNgEEAA","pageInfo":{"resultsPerPage":50,"totalResults":1994},"prevPageToken":"CKYEEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/OjWuZOr5rNl_HasCmAEhF5cAN9E\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CIoFEAA","pageInfo":{"resultsPerPage":50,"totalResults":1995},"prevPageToken":"CNgEEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/g8lxb3uVDwSfamhbbxFlyoVKBTg\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CLwFEAA","pageInfo":{"resultsPerPage":50,"totalResults":1986},"prevPageToken":"CIoFEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/wnTWwe1b-0gy--2ochuizsAjD9o\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CO4FEAA","pageInfo":{"resultsPerPage":50,"totalResults":1993},"prevPageToken":"CLwFEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/N1y4dp3ngAItfwukyLLitFQFDPA\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CKAGEAA","pageInfo":{"resultsPerPage":50,"totalResults":1981},"prevPageToken":"CO4FEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/F8oNrDOjlKMvHjzPIF7cJqfK_zo\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CNIGEAA","pageInfo":{"resultsPerPage":50,"totalResults":1986},"prevPageToken":"CKAGEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/JCWNr1NCzgl0GLh66R-SHeokbW8\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CIQHEAA","pageInfo":{"resultsPerPage":50,"totalResults":1992},"prevPageToken":"CNIGEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/uTbvXolYwtv5Zt4mpLQVHCQsbF4\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CLYHEAA","pageInfo":{"resultsPerPage":50,"totalResults":1999},"prevPageToken":"CIQHEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/xxkZZRO_iCVgmWLAsBW4Qk6VdcU\"","items":[],"kind":"youtube#searchListResponse","pageInfo":{"resultsPerPage":50,"totalResults":1995},"prevPageToken":"CLYHEAE","regionCode":"FR"}
Here is the code used :
List<SearchResult> searchItemList = new ArrayList<>();
YouTube.Search.List searchListRequest = youTubeService.search().list("snippet");
searchListRequest.setMaxResults(NUMBER_OF_VIDEOS_RETURNED);
searchListRequest.setEventType("live");
searchListRequest.setType("video");
searchListRequest.setVideoCategoryId(GAMING_VIDEO_CATEGORY);
searchListRequest.setOrder("viewCount");
searchListRequest.setSafeSearch("none");
searchListRequest.setPrettyPrint(true);
String nextToken = "";
do {
if (!nextToken.isEmpty()) {
searchListRequest.setPageToken(nextToken);
}
SearchListResponse searchListResponse = searchListRequest.execute();
System.out.println(searchListResponse);
if (searchListResponse.getItems().size() > 0) {
searchItemList.addAll(searchListResponse.getItems());
}
nextToken = searchListResponse.getNextPageToken();
} while (nextToken != null);
I don't get why the third to last request "items" field is empty. Are there some kind of restrictions from Youtube API?
I am learning Amazon Cloud Search but I couldn't find any code in either C# or Java (though I am creating in C# but if I can get code in Java then I can try converting in C#).
This is just 1 code I found in C#: https://github.com/Sitefinity-SDK/amazon-cloud-search-sample/tree/master/SitefinityWebApp.
This is 1 method i found in this code:
public IResultSet Search(ISearchQuery query)
{
AmazonCloudSearchDomainConfig config = new AmazonCloudSearchDomainConfig();
config.ServiceURL = "http://search-index2-cdduimbipgk3rpnfgny6posyzy.eu-west-1.cloudsearch.amazonaws.com/";
AmazonCloudSearchDomainClient domainClient = new AmazonCloudSearchDomainClient("AKIAJ6MPIX37TLIXW7HQ", "DnrFrw9ZEr7g4Svh0rh6z+s3PxMaypl607eEUehQ", config);
SearchRequest searchRequest = new SearchRequest();
List<string> suggestions = new List<string>();
StringBuilder highlights = new StringBuilder();
highlights.Append("{\'");
if (query == null)
throw new ArgumentNullException("query");
foreach (var field in query.HighlightedFields)
{
if (highlights.Length > 2)
{
highlights.Append(", \'");
}
highlights.Append(field.ToUpperInvariant());
highlights.Append("\':{} ");
SuggestRequest suggestRequest = new SuggestRequest();
Suggester suggester = new Suggester();
suggester.SuggesterName = field.ToUpperInvariant() + "_suggester";
suggestRequest.Suggester = suggester.SuggesterName;
suggestRequest.Size = query.Take;
suggestRequest.Query = query.Text;
SuggestResponse suggestion = domainClient.Suggest(suggestRequest);
foreach (var suggest in suggestion.Suggest.Suggestions)
{
suggestions.Add(suggest.Suggestion);
}
}
highlights.Append("}");
if (query.Filter != null)
{
searchRequest.FilterQuery = this.BuildQueryFilter(query.Filter);
}
if (query.OrderBy != null)
{
searchRequest.Sort = string.Join(",", query.OrderBy);
}
if (query.Take > 0)
{
searchRequest.Size = query.Take;
}
if (query.Skip > 0)
{
searchRequest.Start = query.Skip;
}
searchRequest.Highlight = highlights.ToString();
searchRequest.Query = query.Text;
searchRequest.QueryParser = QueryParser.Simple;
var result = domainClient.Search(searchRequest).SearchResult;
//var result = domainClient.Search(searchRequest).SearchResult;
return new AmazonResultSet(result, suggestions);
}
I have already created domain in Amazon Cloud Search using AWS console and uploaded document using Amazon predefine configuration option that is movie Imdb json file provided by Amazon for demo.
But in this method I am not getting how to use this method, like if I want to search Director name then how do I pass in this method as because this method parameter is of type ISearchQuery?
I'd suggest using the official AWS CloudSearch .NET SDK. The library you were looking at seems fine (although I haven't look at it any detail) but the official version is more likely to expose new CloudSearch features as soon as they're released, will be supported if you need to talk to AWS support, etc, etc.
Specifically, take a look at the SearchRequest class -- all its params are strings so I think that obviates your question about ISearchQuery.
I wasn't able to find an example of a query in .NET but this shows someone uploading docs using the AWS .NET SDK. It's essentially the same procedure as querying: creating and configuring a Request object and passing it to the client.
EDIT:
Since you're still having a hard time, here's an example. Bear in mind that I am unfamiliar with C# and have not attempted to run or even compile this but I think it should at least be close to working. It's based off looking at the docs at http://docs.aws.amazon.com/sdkfornet/v3/apidocs/
// Configure the Client that you'll use to make search requests
string queryUrl = #"http://search-<domainname>-xxxxxxxxxxxxxxxxxxxxxxxxxx.us-east-1.cloudsearch.amazonaws.com";
AmazonCloudSearchDomainClient searchClient = new AmazonCloudSearchDomainClient(queryUrl);
// Configure a search request with your query
SearchRequest searchRequest = new SearchRequest();
searchRequest.Query = "potato";
// TODO Set your other params like parser, suggester, etc
// Submit your request via the client and get back a response containing search results
SearchResponse searchResponse = searchClient.Search(searchRequest);
There isnt an example online with the Java API to show how to limit the rows that come back from a search with ElasticSearch for all items. I tried the Filter Limit but it just wouldnt work because it would bring back more then the limit. I know its per shard to, but is there no way around this. I cant find the from/size query/filter either in the Java API
SearchQuery searchQuery = startQuery(limit, null).build();
Iterable<Statement> iterableStatements = esSpringDataRepository.search(searchQuery);
if (iterableStatements != null) {
return IteratorUtils.toList(iterableStatements.iterator());
}
private NativeSearchQueryBuilder startQuery(int limit, QueryBuilder query) {
NativeSearchQueryBuilder searchQueryBuilder = new NativeSearchQueryBuilder();
if(query != null) {
searchQueryBuilder = searchQueryBuilder.withQuery(query);
}
if(limit > 0) {
searchQueryBuilder = searchQueryBuilder.withFilter(FilterBuilders.limitFilter(limit));
}
return searchQueryBuilder;
}
Well I got it to work perfectly with this instead of the limit filter:
searchQueryBuilder = searchQueryBuilder.withPageable(new PageRequest(0, limit));
Using a JSP web page and Rserve, I am getting data from a MySQL database and using an R dataframe to store the data. This works fine and plots perfect.
However, if the db query returns nothing the dataframe is then empty and it throws an error when trying to plot.
What I want to do is redirect to another JSP page which will then display the error but I am not sure how to do this.
I have found this R code (what it does was purely for testing purposes) which tells me if the dataframe is empty or not but how can I then include Java (or something else) to redirect the page?
if (nrow(df) != 0) {
df
} else {
df <- "Empty"
df
}
Edit: I have managed to get this far:
c.eval("if(nrow(df) != 0){ print(ggplot(df, aes(x=Date, y=UID))+geom_point(shape=1)) }"
+"else { print(\"Failed\") }");
The 'failed' doesn't print (I didn't really expect it to) but as said above in the else I would like a redirect. Any thoughts about how this would be possible?
A simply try{} catch{} solved the problem! Don't know why I didn't think of that earlier.
So instead of:
c.eval("if(nrow(df) != 0){ print(ggplot(df, aes(x=Date, y=UID))+geom_point(shape=1)) } else { print(\"Failed\") }");
I have used this:
try {
c.eval("print(ggplot(df, aes(x=Date, y=UID)) + geom_point(shape=1))"); // point graph
System.out.print("Success");
} catch (Exception e) {
out.print(e.getMessage());
System.out.print("Failed");
}