I'm using twitter4j version 2.2.5. setPage() doesn't seem to work if I use it with setSince() and setUntil(). See the following code:
Twitter twitter = new TwitterFactory().getInstance();
twitter.setOAuthConsumer(CONSUMER_KEY, CONSUMER_SECRET);
AccessToken accessToken = new AccessToken(ACCESS_TOKEN, ACCESS_TOKEN_SECRET);
twitter.setOAuthAccessToken(accessToken);
int page = 1;
while(true) {
Query query = new Query("from:someUser");
query.setSince("2012-01-01");
query.setUntil("2012-07-05");
query.setRpp(100);
query.setPage(page++);
QueryResult qr = twitter.search(query);
List<twitter4j.Tweet> qrTweets = qr.getTweets();
if(qr.getTweets().size() == 0) break;
for(twitter4j.Tweet t : qrTweets) {
System.out.println(t.getId() + " " + t.getText());
}
}
The code inside the loop is only executed once if I use setSince() and setUntil() methods. But without them, setPage() seems to work and I get more tweet results.
Any ideas why this is happening?
Your code appears to be working for me. It only returns tweets from the past nine days, but that's expected (approximately) according to your comment.
You mentioned in the question that the loop only ran once with rpp=100, and you said in a comment that all 80 of the user's tweets were returned when rpp=40. I think that would indicate that the code is working as expected. If the user only has 80 tweets, the loop only should run once when rpp=100, and it should execute twice and display all their tweets when rpp=40. Try displaying the page number as soon as you enter the while loop so you can see how many times the loop runs.
Related
I want to get tweets from certain user timelines using java library twitter4j, currently I have source code which can get ~ 3200 tweets from user time line but I can't get full tweet. I have searched in various sources on the internet but I can't find a solution to my problem. anyone can help me or can anyone provide an alternative to get a full tweet from the user timeline with java programming?
my source code :
public static void main(String[] args) throws SQLException {
ConfigurationBuilder cb = new ConfigurationBuilder();
cb.setDebugEnabled(true)
.setOAuthConsumerKey("aaa")
.setOAuthConsumerSecret("aaa")
.setOAuthAccessToken("aaa")
.setOAuthAccessTokenSecret("aaa");
TwitterFactory tf = new TwitterFactory(cb.build());
Twitter twitter = tf.getInstance();
int pageno = 1;
String user = "indtravel";
List statuses = new ArrayList();
while (true) {
try {
int size = statuses.size();
Paging page = new Paging(pageno++, 100);
statuses.addAll(twitter.getUserTimeline(user, page));
System.out.println("***********************************************");
System.out.println("Gathered " + twitter.getUserTimeline(user, page).size() + " tweets");
//get status dan user
for (Status status: twitter.getUserTimeline(user, page)) {
//System.out.println("*********Place Tweets :**************\npalce country :"+status.getPlace().getCountry()+"\nplace full name :"+status.getPlace().getFullName()+"\nplace name :"+status.getPlace().getName()+"\nplace id :"+status.getPlace().getId()+"\nplace tipe :"+status.getPlace().getPlaceType()+"\nplace addres :"+status.getPlace().getStreetAddress());
System.out.println("["+(no++)+".] "+"Status id : "+status.getId());
System.out.println("id user : "+status.getUser().getId());
System.out.println("Length status : "+status.getText().length());
System.out.println("#" + status.getUser().getScreenName() +" . "+status.getCreatedAt()+ " : "+status.getUser().getName()+"--------"+status.getText());
System.out.println("url :"+status.getUser().getURL());
System.out.println("Lang :"+status.getLang());
}
if (statuses.size() == size)
break;
}catch(TwitterException e) {
e.printStackTrace();
}
}
System.out.println("Total: "+statuses.size());
}
Update :
After the answer given by #AndyPiper
the my problem is every tweet that I get will be truncated or not complete. a tweets that I get will be truncated if the length of tweet more than 140 characters. I found the reference tweet_mode=extended, but I do not know how to use it. if you know something please tell me.
Your Configuration should be like this:
ConfigurationBuilder cb = new ConfigurationBuilder();
cb.setDebugEnabled(true)
.setOAuthConsumerKey("aaa")
.setOAuthConsumerSecret("aaa")
.setOAuthAccessToken("aaa")
.setOAuthAccessTokenSecret("aaa")
.setTweetModeExtended(true);
It is explained well here
The Twitter API limits the timeline history to 3200 Tweets. To get more than that you would need to use the (commercial) premium or enterprise APIs to search for Tweets by a specific user.
if you are streaming tweets
fist: you have to add .setTweetModeExtended(true); into your configurationbuilder
second(here is the code)
TwitterStream twitterStream = new TwitterStreamFactory(cb.build()).getInstance();
StatusListener listener = new StatusListener(){
public void onStatus(Status status) {
System.out.println("-------------------------------");
if(status.isRetweet()){
System.out.println(status.getRetweetedStatus().getText());
}
else{
System.out.println(status.getText());
}`
its totally works for me.
take care yourself :)
If you are implementing twitter api using twitter4j.properties file and getting truncated timeline text value,simply add the below property in it at the end.
tweetModeExtended=TRUE
I am trying to run a Bulk Request through JEST and want to append my data (say "bills") one at a time and then execute all at once, however when i run the following code on 10 bills just the last bill is getting executed, can someone please correct this code to execute all 10 bills (by executing it outside the for loop ie using Bulk Request)?
for(JSONObject bill : bills) {
bulkRequest = new Bulk.Builder()
.addAction(new Index.Builder(bill.toString()).index(index).type(type).id(id).build())
.build();
}
bulkResponse = Client.execute(bulkRequest);
You need to build the Bulk Builder out of the loop and then use it to add all bills:
bulkRequest = new Bulk.Builder()
for(JSONObject bill : bills) {
bulkRequest.addAction(new Index.Builder(bill.toString()).index(index).type(type).id(id).build())
}
bulkResponse = Client.execute(bulkRequest.build());
I know It's an old question, but just in case someone stumbles across this, here is a java 8/(lambdas) way of doing the same thing.
Client.execute( new Bulk.Builder()
.addAction(
bills.stream()
.map(bill ->
new Index.Builder(bill.toString()
)
.index(index).type(type).id(id).build())
.collect(Collectors.toList())
).build());
I'm trying to retrieve tweets for a given set of search terms using twitter4j.
I can successfully retrieve tweets for a query with only one term, or a query using the OR operator, but when I enter an AND condition in the query I get zero results. I have completed the same query on Twitter's search bar and I get hundreds of results.
It looks like this page references a similar issue, but there is no listed solution.
I would like to get results for the query "#treet java"
Here's my method that completes the search...
// queryString = "#treet" - works
// queryString = "java" - works
// queryString = "#treet OR java" - works
// queryString = "#treet java" - doesn't work
// queryString = "#treet AND java" - doesn't work
public static Treet[] loadAndSaveRemote() {
Twitter twitter = TwitterFactory.getSingleton();
Query query = new Query();
query.setQuery(queryString);
query.setCount(100);
QueryResult result = null;
try {
result = twitter.search(query);
} catch (TwitterException e) {
System.out.println("Getting tweet failed");
e.printStackTrace();
}
List<Status> tweets = result.getTweets();
List<Treet> list = new ArrayList<>();
// With queries using "term1 term2" the output is "0 tweets were retrieved" in all cases
System.out.printf("%d tweets were retrieved %n", tweets.size());
for (Status status : tweets) {
list.add( new Treet(status.getUser().getScreenName(),
status.getText(),
status.getCreatedAt()));
}
Treet[] returning = list.toArray(new Treet[list.size()]);
Treets.save(returning);
return returning;
}
I have looked at the Twitter search api and my query syntax seems correct. I have also seen examples online using a similar format.
Any thoughts would be much appreciated.
I'm working on writing a simple program to fetch tweets for an information retrieval class (later I will be learning to index and search through them). For now, I just want to gather data. I'm using the twitter4j API and am having trouble understanding the Paging class and how it works. Below is a snippet of code:
TwitterFactory tf = new TwitterFactory(cb.build());
Twitter twitter = tf.getInstance();
Paging paging = new Paging(2,40);
try{
List<Status> statuses = twitter.getUserTimeline("google", paging);
System.out.println(paging);
for(Status status : statuses)
{
System.out.println(status.getText());
}
System.out.println("\n\n\n");
paging.setPage(2);
statuses = twitter.getUserTimeline("google",paging);
for(Status status : statuses)
{
System.out.println(status.getText());
}
}
catch(TwitterException e){
e.printStackTrace();
}
My hope was that the second call to twitter.getUserTimeline would return the next 40 tweets from Google as this example seems to suggest, but upon inspection, both return the same 40 tweets. Can someone then explain what the Paging class actually does?
Your first call should request the first page Paging paging = new Paging(1, 40);, not the second one.
I am trying to extract comments on some YouTubeVideos using the youtube-api with Java. Everything is going fine except the fact that I am not able to extract all the comments if the video has a large number of comments (it stops at somewhere in between 950 and 999). I am following a simple method of paging through the CommentFeed of the VideoEntry, getting comments on each page and then storing each comment in an ArrayList before writing them in an XML file. Here is my code for retrieving the comments
int commentCount = 0;
CommentFeed commentFeed = service.getFeed(new URL(commentUrl), CommentFeed.class);
do {
//Gets each comment in the current feed and prints to the console
for(CommentEntry comment : commentFeed.getEntries()) {
commentCount++;
System.out.println("Comment " + commentCount + " plain text content: " + comment.getPlainTextContent());
}
//Checks if there is a next page of comment feeds
if (commentFeed.getNextLink() != null) {
commentFeed = service.getFeed(new URL(commentFeed.getNextLink().getHref()), CommentFeed.class);
}
else {
commentFeed = null;
}
}
while (commentFeed != null);
My question is: Is there some limit on the number of comments that I could extract or am I doing something wrong?
use/refer this
String commentUrl = videoEntry.getComments().getFeedLink().getHref();
CommentFeed commentFeed = service.getFeed(new URL(commentUrl), CommentFeed.class);
for(CommentEntry comment : commentFeed.getEntries()) {
System.out.println(comment.getPlainTextContent());
}
source
Max number of results per iteration is 50 (it seems) as mentioned here
and you can use start-index to retrieve multiple result sets as mentioned here
Google Search API and as well as Youtube Comments search limits max. 1000 results and u cant extract more than 1000 results