I am trying to write a java code to get the tweets within a special range of time.
I am using Since and Until functions in twitter4j api. However It only returns a limited number of tweets within last minutes.
is there a way to get it work? did any body have the same experience ?
Related
Requirement: I need to monitor the Twilio account and subaccount usages in near real-time.
Any solution in java, php, python or even curl will do for me.
Twilio provides Usage Records API and allows some subresources which contains Today but that returns all data from the start of that day till the nearest current time. I am unable to find anything in the documents that would retrieve only usages of the last minute or last 10 minutes or even between two time constants. The Usage API accepts two dates but not time.
Hoping someone out there has a solution to offer.
So I am currently trying to gather tweets on a specific location and then analyse what is going on in that location from the tweets gathered. My task basically involves a lot of data mining.
The main problem I have come across however is gathering enough tweets that will allow me to make a judgement.
I have been using the Twitter Streaming API, however this only gives 1% of all the tweets which is far from enough. I mined 100,000 tweets and very little were in English let alone related to the location I was looking for.
I have also noticed that twitter rate limits how often you can call a method via their API. How are sites like trendsmap.com working? Are they somehow accessing a larger data set?
Edit: Ok, so I have tried to use the geolocation feature in the twiiter4j API. Turns out the rate limits can be avoided if you are careful with your implementation. The amount of people however that actually have the geolocation feature turned on when tweeting is very low. This therefore does not represent people in that area. I seem to be getting the same tweets every single time. Twitter does offer a search operator "near" which works great on their website. However they have not included this functionality in their API as far as I can tell.
If you are searching using the Twitter API you can restrict your searches to a specific geolocation using the geocode option.
You can use result_type=recent to ensure you're only getting the most recent tweets.
The maximum count - that is, number of tweets per request - is 100.
The current limit on number of search requests per hour is 450.
So, that's a maximum of 45,000 tweets per hour - is that enough for you?
tl:dr - use the most restrictive set of search parameters to limit the results to those you actually need.
I have a requirement for retrieving all(i mean "all") till a given date or between dates.
But the code i wrote gives me tweets but only for today. I implemented paging but its no help i do get multiple pages and the data is not redundant. But the data is still limited for the current day. I only get like 600-700 tweets. And i used hasNext() and it retrieves false after 6-7 pages.
I'm fairly new to this API and i dont have much idea about the framework so forgive me if i sound really naive.
Heres the code:
Query search=new Query(searchKeyWord);
QueryResult results;
search.setCount(100);
//search.setMaxId(-1);
search.setSince("2013-01-01");
search.lang("en");
// search.setUntil("2013-05-01");
int i=0;
//TwitterFactory.getSingleton().search(search);//
do{
i++;
System.out.println("Page "+i);
results=tweety.search(search);
for(Status stats : results.getTweets()){
Text=stats.getText();
Text=Text.replace("\n", " ");
writer.append(stats.getUser().getScreenName()+";"+Text+";"+stats.getCreatedAt()+";"+"\n");
}
search=results.nextQuery();
} while(search!=null);
The requirement is for text mining on a large amount data so the more tweets retrieved the better. Of course I will restricting the since and until dates. But if i set the dates for an older time interval the tweets are still retrieved only for the last day of that interval.
Am i wrong here somewhere? And is there something I need to add or change to get all the tweets? I'm aware of rate limits. Is this the reason why i receive only limited data?
Thanks in advance.
You should use both search API and Streaming API. I am also working on data mining with twitter data and what I am doing is I just implemented two different apps to collect tweets. You can also do same thing. The streaming API needs only one twitter account for token and authentication stuff.However, you should have more accounts for the search API. If you have more questions let me know.
With https://dev.twitter.com/docs/api/1/get/statuses/user_timeline I can get 3,200 most recent tweets. However, certain sites like http://www.mytweet16.com/ seems to bypass the limit, and my browse through the API documentation could not find anything.
How do they do it, or is there another API that doesn't have the limit?
You can use twitter search page to bypass 3,200 limit. However you have to scroll down many times in the search results page. For example, I searched tweets from #beyinsiz_adam. This is the link of search results:
https://twitter.com/search?q=from%3Abeyinsiz_adam&src=typd&f=realtime
Now in order to scroll down many times, you can use the following javascript code.
var myVar=setInterval(function(){myTimer()},1000);
function myTimer() {
window.scrollTo(0,document.body.scrollHeight);
}
Just run it in the FireBug console. And wait some time to load all tweets.
The only way to see more is to start saving them before the user's tweet count hits 3200. Services which show more than 3200 tweets have saved them in their own dbs. There's currently no way to get more than that through any Twitter API.
http://www.quora.com/Is-there-a-way-to-get-more-than-3200-tweets-from-a-twitter-user-using-Twitters-API-or-scraping
https://dev.twitter.com/discussions/276
Note from that second link: "…the 3,200 limit is for browsing the timeline only. Tweets can always be requested by their ID using the GET statuses/show/:id method."
I've been in this (Twitter) industry for a long time and witnessed lots of changes in Twitter API and documentation. I would like to clarify one thing to you. There is no way to surpass 3200 tweets limit. Twitter doesn't provide this data even in its new premium API.
The only way someone can surpass this limit is by saving the tweets of an individual Twitter user.
There are tools available which claim to have a wide database and provide more than 3200 tweets. Few of them are followersanalysis.com, keyhole.co which I know of.
You can use a tool I wrote that bypasses the limit.
It saves the Tweets in a JSON format.
https://github.com/pauldotknopf/twitter-dump
You can use a Python library snscrape to do it. Or you can use ExportData tool to get all tweets for the user, which returns already preprocessed CSV and spreadsheet files. The first option is free, but has less information and requires more manual work.
I'm trying to get the number of results from google for search term in Java.
eg. For the term "computer" :About 3,070,000,000 results.
Is it possible?
You should be able to do this with Java sample library for custom google search API.
http://code.google.com/p/google-api-java-client/wiki/APIs#CustomSearch_API
Also you can reduce the data transferred during service requests by specifying members that contain data elements that your application needs.
It is not as straightforward as one might think... You can write a Java program that reads the results of a Google search on a term like "computer", but the number of results will not be statically present in the output.
You will have to use the Google Custom Search Engine, or the JSON/Atom Custom Search API.