Size of RSS feed - java

I'm using ROME to generate a feed from data in my database.
In all the samples I found, the Servlet extracts all the data from the database, and sends it as a feed.
Now, if the database contains thousands of entries, how many entries should I send?
protected void processRequest(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
try {
SyndFeed feed = getFeed(request);
String feedType = request.getParameter("type");
feedType = feedType != null ? feedType : defaultType;
feed.setFeedType(feedType);
response.setContentType("application/xml; charset=UTF-8");
SyndFeedOutput output = new SyndFeedOutput();
output.output(feed, response.getWriter());
} catch (FeedException ex) {
String msg = "Could not generate feed";
log(msg, ex);
response.sendError(HttpServletResponse.SC_INTERNAL_SERVER_ERROR, msg);
}
}
protected SyndFeed getFeed(HttpServletRequest request) {
// **** Here I query the database for posts, but I don't know how many
// I should fetch or where should I stop? ***
List<Post> posts = getPosts();
SyndFeed feed = new SyndFeedImpl();
feed.setTitle("My feed");
feed.setLink("http://myurl");
feed.setDescription("my desc");
// create the feeds.Each tutorial will be a feed entry
List<SyndEntry> entries = new ArrayList<SyndEntry>();
for (Post post : posts) {
SyndEntry entry = new SyndEntryImpl();
SyndContent description;
String title = post.getTitle();
String link = post.getLink();
entry.setTitle(title);
entry.setLink(link);
// Create the description of the feed entry
description = new SyndContentImpl();
description.setType("text/plain");
description.setValue(post.getDesc());
entry.setDescription(description);
entries.add(entry);
}
feed.setEntries(entries);
return feed;
}


There really isn't a single way to do this that all rss clients will support, but i would recommend checking out rfc 5005 appendix B, you'll at least have a referenxe to give to clients. https://www.rfc-editor.org/rfc/rfc5005#appendix-B
so long as your default query always shows the latest (page length you define) items, sorted descending, all clients will appear correct. Clients that need to be able to page can implement this standard.


I suggest a paging system. the user makes a request for page 0 to take 30 items. then the user makes a request for page 1 to take the next 30 items. first request: items 0->29, second request: items 30->59. to model this have a integer variable called skip keep track of what position to start at, for instance:
int skip = page * numItems; // first request: 0 * 30 (starts at 0), sec request: 1 * 30 (starts at 30)
So you will skip so many items and take only the value of numItems. Then the client requests for however many feed items the client wants at once.

Related

Loading all contacts using the Microsoft Graph API sometimes looses/skips pages

We have an application that loads all contacts stored in an account using the Microsoft Graph API. The initial call we issue is https://graph.microsoft.com/v1.0/users/{userPrincipalName}/contacts$count=true&$orderBy=displayName%20ASC&$top=100, but we use the Java JDK to do that. Then we iterate over all pages and store all loaded contacts in a Set (local cache).
We do this every 5 minutes using an account with over 3000 contacts and sometimes, the count of contacts we received due to using $count does not match the number of contacts we loaded and stored in the local cache.
Verifying the numbers manually we can say, that the count was always correct, but there are contacts missing.
We use the following code to achieve this.
public List<Contact> loadContacts() {
Set<Contact> contacts = new TreeSet<>((contact1, contact2) -> StringUtils.compare(contact1.id, contact2.id));
List<QueryOption> requestOptions = List.of(
new QueryOption("$count", true),
new QueryOption("$orderBy", "displayName ASC"),
new QueryOption("$top", 100)
);
ContactCollectionRequestBuilder pageRequestBuilder = null;
ContactCollectionRequest pageRequest;
boolean hasNextPage = true;
while (hasNextPage) {
// initialize page request
if (pageRequestBuilder == null) {
pageRequestBuilder = graphClient.users(userId).contacts();
pageRequest = pageRequestBuilder.buildRequest(requestOptions);
} else {
pageRequest = pageRequestBuilder.buildRequest();
}
// load
ContactCollectionPage contactsPage = pageRequest.get();
if (contactsPage == null) {
throw new IllegalStateException("request returned a null page");
} else {
contacts.addAll(contactsPage.getCurrentPage());
}
// handle next page
hasNextPage = contactsPage.getNextPage() != null;
if (hasNextPage) {
pageRequestBuilder = contactsPage.getNextPage();
} else if (contactsPage.getCount() != null && !Objects.equals(contactsPage.getCount(), (long) contacts.size())) {
throw new IllegalStateException(String.format("loaded %d contacts but response indicated %d contacts", contacts.size(), contactsPage.getCount()));
} else {
// done
}
}
log.info("{} contacts loaded using graph API", contacts.size());
return new ArrayList<>(contacts);
}
Initially, we did not put the loaded contacts in a Set by ID but just in a List. With the List we very often got more contacts than $count. My idea was, that there is some caching going on and some pages get fetched multiple times. Using the Set we can make sure, that we only have unique contacts in our local cache.
But using the Set, we sometimes have less contacts than $count, meaning some pages got skipped and we end up in the condition that throws the IllegalStateException.
Currently, we use microsoft-graph 5.8.0 and azure-identiy 1.4.2.
Have you experienced similar issues and can help us solve this problem?
Or do you have any idea what could be causing these inconsistent results?
Your help is very much appreciated!

Downloading attachments from unseen messages

I work on university project in java. I have to download attachments from new emails using GMAIL API.
I successfully connected to gmail account using OAuth 2.0 authorization.
private static final List<String> SCOPES = Collections.singletonList(GmailScopes.GMAIL_READONLY);
I tried to get unseen mails using
ListMessagesResponse listMessageResponse = service.users().messages().list(user).setQ("is:unseen").execute();
listMessageResponse is not null but when I call method .getResultSizeEstimate() it returns 0
also I tried to convert listMessageResponse to List < Message > (I guess this is more usable) using
List<Message> list = listMessageResponse.getMessages();
But list launches NullPointerException
Then tried to get each attachment with
for(Message m : list) {
List<MessagePart> part = m.getPayload().getParts();
for(MessagePart p: part) {
if(p.getFilename()!=null && p.getFilename().length()>0) {
System.out.println(p.getFilename()); // Just to check attachment filename
}
}
}
Is my approach correct (if not how to fix it) and how should I download those attachments.
EDIT 1:
Fixed q parameter, I mistakenly wrote is:unseen instead of is:unread.
Now app reaches unread mails successfully.
(For example there was two unread mails and both successfully reached, I can get theirs IDs easy).
Now this part trows NullPointerException
List<MessagePart> part = m.getPayload().getParts();
Both messages have attachments and m is not null (I get ID with .getID())
Any ideas how to overcome this and download attachment?
EDIT 2:
Attachments Downloading part
for(MessagePart p : parts) {
if ((p.getFilename() != null && p.getFilename().length() > 0)) {
String filename = p.getFilename();
String attId = p.getBody().getAttachmentId();
MessagePartBody attachPart;
FileOutputStream fileOutFile = null;
try {
attachPart = service.users().messages().attachments().get("me", p.getPartId(), attId).execute();
byte[] fileByteArray = Base64.decodeBase64(attachPart.getData());
fileOutFile = new FileOutputStream(filename); // Or any other dir
fileOutFile.write(fileByteArray);
fileOutFile.close();
}catch (IOException e) {
System.out.println("IO Exception processing attachment: " + filename);
} finally {
if (fileOutFile != null) {
try {
fileOutFile.close();
} catch (IOException e) {
// probably doesn't matter
}
}
}
}
}
Downloading working like charm, tested app with different type of emails.
Only thing left is to change label of unread message (that was reached by app) to read. Any tips how to do it?
And one tiny question:
I want this app to fetch mails on every 10 minutes using TimerTask abstract class. Is there need for manual "closing" of connection with gmail or that's done automatically after run() method iteration ends?
#Override
public void run(){
// Some fancy code
service.close(); // Something like that if even exists
}

I don't think ListMessagesResponse ever becomes null. Even if there are no messages that match your query, at least resultSizeEstimate will get populated in the resulting response: see Users.messages: list > Response.
I think you are using the correct approach, just that there is no message that matches your query. Actually, I never saw is:unseen before. Did you mean is:unread instead?
Update:
When using Users.messages: list only the id and the threadId of each message is populated, so you cannot access the message payload. In order to get the full message resource, you have to use Users.messages: get instead, as you can see in the referenced link:
Note that each message resource contains only an id and a threadId. Additional message details can be fetched using the messages.get method.
So in this case, after getting the list of messages, you have to iterate through the list, and do the following for each message in the list:
Get the message id via m.getId().
Once you have retrieved the message id, use it to call Gmail.Users.Messages.Get and get the full message resource. The retrieved message should have all fields populated, including payload, and you should be able to access the corresponding attachments.
Code sample:
List<Message> list = listMessageResponse.getMessages();
for(Message m : list) {
Message message = service.users().messages().get(user, m.getId()).execute();
List<MessagePart> part = message.getPayload().getParts();
// Rest of code
}
Reference:
Class ListMessagesResponse
Users.messages: list > Response

In Youtube API, I get empty pages after the second page

I try to get all the live videos of youtube api at a given time.
However, after the two firsts pages that contains 50 results each, all the others pages are empty.
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/7Mn5oZUWkgMIU4kfKx7cq0D0nDI\"","items":[{THIS ONE IS OK WITH 50 RESULTS}],"kind":"youtube#searchListResponse","nextPageToken":"CDIQAA","pageInfo":{"resultsPerPage":50,"totalResults":1994},"regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/wp4SIiAGHN1uLiJgvLGoMdQoehs\"","items":[{THIS ONE IS OK TOO WITH 50 RESULTS}],"kind":"youtube#searchListResponse","nextPageToken":"CGQQAA","pageInfo":{"resultsPerPage":50,"totalResults":1994},"prevPageToken":"CDIQAQ","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/3zCHC7x--dhww0e7udfLQn3DB5Y\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CJYBEAA","pageInfo":{"resultsPerPage":50,"totalResults":1988},"prevPageToken":"CGQQAQ","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/pGfYwQfcIDdD05OBzqTVFbdg39E\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CMgBEAA","pageInfo":{"resultsPerPage":50,"totalResults":1986},"prevPageToken":"CJYBEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/qtrckZRTf7czKQNL5nQpXRu7X5A\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CPoBEAA","pageInfo":{"resultsPerPage":50,"totalResults":1985},"prevPageToken":"CMgBEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/gTY6oWopjQcy6Cuf1MGQqYGYfog\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CKwCEAA","pageInfo":{"resultsPerPage":50,"totalResults":1993},"prevPageToken":"CPoBEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/umiznrtjy7vmWkIJ5Hsm8wtU0W0\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CN4CEAA","pageInfo":{"resultsPerPage":50,"totalResults":1985},"prevPageToken":"CKwCEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/thJws1H1pJJPtOkVeq6L6VGabn8\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CJADEAA","pageInfo":{"resultsPerPage":50,"totalResults":1988},"prevPageToken":"CN4CEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/TaDYlGeeLP2H1xanLHVTtmt_DNg\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CMIDEAA","pageInfo":{"resultsPerPage":50,"totalResults":1995},"prevPageToken":"CJADEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/w3wj8nt1NHhxyh-fuWLJ1v5ncvU\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CPQDEAA","pageInfo":{"resultsPerPage":50,"totalResults":1989},"prevPageToken":"CMIDEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/r1hbVDG2ANenKRichmZxj0J2dws\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CKYEEAA","pageInfo":{"resultsPerPage":50,"totalResults":1996},"prevPageToken":"CPQDEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/hdDkPLk2aNKrJLZ8tnMnCGqhUy8\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CNgEEAA","pageInfo":{"resultsPerPage":50,"totalResults":1994},"prevPageToken":"CKYEEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/OjWuZOr5rNl_HasCmAEhF5cAN9E\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CIoFEAA","pageInfo":{"resultsPerPage":50,"totalResults":1995},"prevPageToken":"CNgEEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/g8lxb3uVDwSfamhbbxFlyoVKBTg\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CLwFEAA","pageInfo":{"resultsPerPage":50,"totalResults":1986},"prevPageToken":"CIoFEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/wnTWwe1b-0gy--2ochuizsAjD9o\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CO4FEAA","pageInfo":{"resultsPerPage":50,"totalResults":1993},"prevPageToken":"CLwFEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/N1y4dp3ngAItfwukyLLitFQFDPA\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CKAGEAA","pageInfo":{"resultsPerPage":50,"totalResults":1981},"prevPageToken":"CO4FEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/F8oNrDOjlKMvHjzPIF7cJqfK_zo\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CNIGEAA","pageInfo":{"resultsPerPage":50,"totalResults":1986},"prevPageToken":"CKAGEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/JCWNr1NCzgl0GLh66R-SHeokbW8\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CIQHEAA","pageInfo":{"resultsPerPage":50,"totalResults":1992},"prevPageToken":"CNIGEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/uTbvXolYwtv5Zt4mpLQVHCQsbF4\"","items":[],"kind":"youtube#searchListResponse","nextPageToken":"CLYHEAA","pageInfo":{"resultsPerPage":50,"totalResults":1999},"prevPageToken":"CIQHEAE","regionCode":"FR"}
{"etag":"\"7991kDR-QPaa9r0pePmDjBEa2h8/xxkZZRO_iCVgmWLAsBW4Qk6VdcU\"","items":[],"kind":"youtube#searchListResponse","pageInfo":{"resultsPerPage":50,"totalResults":1995},"prevPageToken":"CLYHEAE","regionCode":"FR"}
Here is the code used :
List<SearchResult> searchItemList = new ArrayList<>();
YouTube.Search.List searchListRequest = youTubeService.search().list("snippet");
searchListRequest.setMaxResults(NUMBER_OF_VIDEOS_RETURNED);
searchListRequest.setEventType("live");
searchListRequest.setType("video");
searchListRequest.setVideoCategoryId(GAMING_VIDEO_CATEGORY);
searchListRequest.setOrder("viewCount");
searchListRequest.setSafeSearch("none");
searchListRequest.setPrettyPrint(true);
String nextToken = "";
do {
if (!nextToken.isEmpty()) {
searchListRequest.setPageToken(nextToken);
}
SearchListResponse searchListResponse = searchListRequest.execute();
System.out.println(searchListResponse);
if (searchListResponse.getItems().size() > 0) {
searchItemList.addAll(searchListResponse.getItems());
}
nextToken = searchListResponse.getNextPageToken();
} while (nextToken != null);
I don't get why the third to last request "items" field is empty. Are there some kind of restrictions from Youtube API?

Getting videos from Youtube official Channels

There are some Youtube official channels like Music, Games, News, etc. I'm trying to get the information of this channels like if it's a normal channel but it's sending me null responses. It might be because the channel videos are not owned by the channel even if it shows you them as "uploaded videos". This is my code which works with "normal" channels: (String ids means the id of each video comma separated)
This method gives the videos of the channel videos and it's working with "normal" channels.
public static ArrayList<VideoYoutube> getVideos (String ids) throws IOException
{
try
{
YouTube.Videos.List videos = youtube.videos().list("id,snippet,statistics");
videos.setKey(API_KEY);
//Setting all the video ids we want to get back
videos.setId(ids);
videos.setMaxResults(RETURN_NUMBER);
//Setting all the fields we want to be called from the api. Its such an
videos.setFields("items(id,snippet/publishedAt,snippet/title,snippet/description," +
"snippet/thumbnails/default/url,snippet/thumbnails/medium/url,snippet/thumbnails/high/url," +
"snippet/channelId,snippet/channelTitle,statistics/viewCount,statistics/likeCount,statistics/dislikeCount," +
"statistics/favoriteCount,statistics/commentCount)");
VideoListResponse videoResponse = videos.execute();
List<Video> videoResultList = videoResponse.getItems();
}
And this is the method that is supposed to retrieve the video ids from the channel I want. It's giving null for this exact channel but it's working with "normal" channels.
public static String getChannelVideosIds(String channelId) throws IOException
{
YouTube.Search.List search = youtube.search().list("id");
//Define the search key
search.setKey(API_KEY);
search.setChannelId(channelId);
//Give the type (video, playlist or channel)
search.setType("video");
//Giving the params we want to be returned comma separated (for example if we had chosen the snipped before, we could choose snipped fields like snipped/publishedAt)
search.setFields("items(id)");
//Set the max return number
search.setMaxResults(RETURN_NUMBER);
//Execute the search query
SearchListResponse searchResponse = search.execute();
List<SearchResult> searchResultList = searchResponse.getItems();
//Check if the result is null
return retrieveIds(searchResultList);
}

authenticate url in java and allow the browser to open secured page

where i will populate a table which has links and each link is pointing to different case id, when click on that link, i need to validate that 3rd party url in my java method and need to allow the browser to open secured page.
any pointers on how to achieve this is very helpful.
Thanks.

Yes, this can be achieved with a simple serlvet.
Suppose to say that you have list of href links in a table
1. Upon clicking on each href link direct it to your servlet.
Ex: < a href="/yourServlet.do?thirdPartyURL=actual3rdPartyURL">actual3rdPartyURL < / a>
Validate this third party URL in your servlet code. If everything is okay
Then redirect it with the SendRedirect method.
Note: it is not a good practice to show the URL in the browser address bar.
as you mentioned that, you are the one who populating this URLs, Use a hashmap to store these URL's and map it with Case ID and redirect it. hope you got the complete info.
Please check the below example and let me know if you need more info
/**
*
*/
public void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException{
PrintWriter out = response.getWriter();
/**
* Assume that this is the map you are getting from third party
* This map holds two value pairs
* Map<CaseID, URL>
*/
Map<String, String> lstURLS = new HashMap<String,String>();
lstURLS.put("CASEID1", "https://www.abc.com/abc1");
lstURLS.put("CASEID2", "https://www.def.com/def");
lstURLS.put("CASEID3", "https://www.egh.com/egh");
/**
* Assume that the request parameter caseID,
* will provide you the case id which was selected by the user
* from the provided table of URLS
*/
String userProvidedCaseID = request.getParameter("caseID");
System.out.println("MySerlvet | caseID | "+ userProvidedCaseID);
/**
* Retrieve the URL from the list of third party URL's
*/
if(null != userProvidedCaseID){
String thirdPartyURL = lstURLS.get("userProvidedCaseID");
if(null != thirdPartyURL){
response.sendRedirect(thirdPartyURL);
}else{
out.print("No Case ID found / Error message");
}
}else{
out.print("No Case ID found / Error message");
}
}

Categories