I am working on a project that I will get all tweets from a country that has tweeted within a certain time period. I will make a data mining on it after that(examining that how many positive thoughts are said for a certain pupil etc.). I want to use Java as programming language. However I don't know how to start this project. I made a search and I know that there is:
Twitter's Search API
Twitter's Streaming API
Twitter4J a twitter API for Java
Something interesting here out of Java : http://dev.datasift.com/discussions/category/csdl-language
Where I can start to get all tweets from a country(if it can be from a given state) within a time period. Some examples are like: you are giving a username and it returns the tweets if it is a public profile. I don't have the list of all public profiles. Should I handle that problem and how?
Any ideas?
If you gonna use Java Twitter4j is your best shot.
But you gonna have to choose a strategy for retrieving the tweets that you want.
You can either get the data from Twitter itself or get it from a Data Provider which has full Firehose Access. DataSift and Gnip are those providers which has full access to Firehose.If you want to use a data provider DataSift is the way to go because of its own query language which is pretty cool.
In case of retrieving the data by yourself.
Firstly if you want to get the Tweets in real time you need to use Twitter Streaming API and Twitter4j makes it really easy to use it.But unfortunately Streaming API doesn't support country or language filtering.You can listen the Streaming API for the search queries that you are registered for.
Your second option is Search API.Twitter4j also makes using Search API pretty easy.Search API supports much more filtering options.But there isn't any way to filter tweets for country.But instead of that filtering tweets depending on the Language is much more useful way to do that. E.g filtering tweets that are en,fr or so on.
Hope this helps.
You want to use the search API. However, the API doesn't allow searching by country, only by geocode.
in Twitter4J
You can get location like this.
tweet.getUser().getLocation()
But it gets user's location input field.
Related
i want to build a weather app and i have some problems.
my big problems is places names! you know, i want that user find his location with two ways. with GPS and by searching. but my problem is that place names. how can find a database from whole places in the world?!
is it good idea that i store them in a database in my server? or there is some services that provide this functionality? if i have to create my own database how i can create a database like that. is there a database with city names and latitude/ longitude and other information? if a new location created how i can add it to the system? by hand?
the second problem is that after catching the city name how i can find that locations latitude/ longitude? it seems google has a Geocoder service but i don't know how it works. please help me. i just want to select a city name and find related coordinates...
The Google Maps API provides you with Geocoding, which allows you to turn strings like 'Santiago, Chile' or 'New York', into proper latitude/longitude coordinates. So in your code, you should perform an HTTP GET request to a URL like this one:
http://maps.googleapis.com/maps/api/geocode/json?address=Santiago,+Chile&sensor=true
And it returns a JSON object with a properly formatted address along with latitude and longitude information of the place you were looking up.
It's explained very thoroughly in the Google Maps API Geocoder documentation, so you should probably take a look at that. I'm no expert in Android development, but there should be some library that allows you to easily access what the Maps API has to offer in a clean way.
Maps API also provides you with a solution for Place Searching, and even input autocompletion, but all the examples I see are on JavaScript/HTML, so I'm not completely sure if there's an alternative in plain Java/Android to what you're trying to do. Nevertheless, you should take a look at basic place searching and place search autocompletion so you get a general sense of how it works.
There's an entire section in the Google Developers website dedicated to the Maps API on Android, so make sure to take a look at that aswell and you might find more useful information - sadly I have no experience with Android whatsoever so I can't really point you in any direction.
Good luck!
You could use the Google Maps Geocoding API. Querying coordinates returns a JSON response containing the current location in multiple levels of detail that you could parse.
E.g. for the position lat:40.714224 long: -73.961452, the query URL would look like the following:
http://maps.googleapis.com/maps/api/geocode/json?latlng=40.714224,-73.961452&sensor=false
The result now contains the city and country, beside much more.
You can find more information on the API here: https://developers.google.com/maps/documentation/geocoding/
You could also try http://openweathermap.org/
I haven't used this extensively but I tried it enough that I know it works. It lets you query by location name or geographic coordinates, and the city name is included in the responses along with plenty of other data.
http://api.openweathermap.org/data/2.5/weather?lat=xx&lon=xx returns a weather object with city name & related data. API key is even optional so you can try it out very quickly.
So I am currently trying to gather tweets on a specific location and then analyse what is going on in that location from the tweets gathered. My task basically involves a lot of data mining.
The main problem I have come across however is gathering enough tweets that will allow me to make a judgement.
I have been using the Twitter Streaming API, however this only gives 1% of all the tweets which is far from enough. I mined 100,000 tweets and very little were in English let alone related to the location I was looking for.
I have also noticed that twitter rate limits how often you can call a method via their API. How are sites like trendsmap.com working? Are they somehow accessing a larger data set?
Edit: Ok, so I have tried to use the geolocation feature in the twiiter4j API. Turns out the rate limits can be avoided if you are careful with your implementation. The amount of people however that actually have the geolocation feature turned on when tweeting is very low. This therefore does not represent people in that area. I seem to be getting the same tweets every single time. Twitter does offer a search operator "near" which works great on their website. However they have not included this functionality in their API as far as I can tell.
If you are searching using the Twitter API you can restrict your searches to a specific geolocation using the geocode option.
You can use result_type=recent to ensure you're only getting the most recent tweets.
The maximum count - that is, number of tweets per request - is 100.
The current limit on number of search requests per hour is 450.
So, that's a maximum of 45,000 tweets per hour - is that enough for you?
tl:dr - use the most restrictive set of search parameters to limit the results to those you actually need.
With https://dev.twitter.com/docs/api/1/get/statuses/user_timeline I can get 3,200 most recent tweets. However, certain sites like http://www.mytweet16.com/ seems to bypass the limit, and my browse through the API documentation could not find anything.
How do they do it, or is there another API that doesn't have the limit?
You can use twitter search page to bypass 3,200 limit. However you have to scroll down many times in the search results page. For example, I searched tweets from #beyinsiz_adam. This is the link of search results:
https://twitter.com/search?q=from%3Abeyinsiz_adam&src=typd&f=realtime
Now in order to scroll down many times, you can use the following javascript code.
var myVar=setInterval(function(){myTimer()},1000);
function myTimer() {
window.scrollTo(0,document.body.scrollHeight);
}
Just run it in the FireBug console. And wait some time to load all tweets.
The only way to see more is to start saving them before the user's tweet count hits 3200. Services which show more than 3200 tweets have saved them in their own dbs. There's currently no way to get more than that through any Twitter API.
http://www.quora.com/Is-there-a-way-to-get-more-than-3200-tweets-from-a-twitter-user-using-Twitters-API-or-scraping
https://dev.twitter.com/discussions/276
Note from that second link: "…the 3,200 limit is for browsing the timeline only. Tweets can always be requested by their ID using the GET statuses/show/:id method."
I've been in this (Twitter) industry for a long time and witnessed lots of changes in Twitter API and documentation. I would like to clarify one thing to you. There is no way to surpass 3200 tweets limit. Twitter doesn't provide this data even in its new premium API.
The only way someone can surpass this limit is by saving the tweets of an individual Twitter user.
There are tools available which claim to have a wide database and provide more than 3200 tweets. Few of them are followersanalysis.com, keyhole.co which I know of.
You can use a tool I wrote that bypasses the limit.
It saves the Tweets in a JSON format.
https://github.com/pauldotknopf/twitter-dump
You can use a Python library snscrape to do it. Or you can use ExportData tool to get all tweets for the user, which returns already preprocessed CSV and spreadsheet files. The first option is free, but has less information and requires more manual work.
Does facebook graph api provide support for searching Post messages by specific location?
It's going to be a multi-step process.
Find the places within your criteria. https://graph.facebook.com/search?q=coffee&type=place¢er=37.76,-122.427&distance=1000&fields=id
Loop thru each of those ids, and build yourself a multi FQL query with up to 50 queries (50 max is all facebook allows) fql?q=SELECT post_id, message, attachment FROM stream WHERE source_id = {pageId}
Query results, and add to your list of posts
repeat 2-3 with the remaining ids returned from the search.
display list of posts.
You can now directly search for posts, photos near a location:
https://developers.facebook.com/docs/reference/api/#searching
It was announced yesterday:
http://developers.facebook.com/blog/post/2012/03/07/building-better-stories-with-location-and-friends/
An example from the docs:
https://graph.facebook.com/search?type=location¢er=37.76,-122.427&distance=1000&access_token=whatever
No.there is no way to search posts by location using graph api.please check this for more details https://developers.facebook.com/tools/explorer
If there is some Place object for your location, then yes you can.
I'm trying to get the number of results from google for search term in Java.
eg. For the term "computer" :About 3,070,000,000 results.
Is it possible?
You should be able to do this with Java sample library for custom google search API.
http://code.google.com/p/google-api-java-client/wiki/APIs#CustomSearch_API
Also you can reduce the data transferred during service requests by specifying members that contain data elements that your application needs.
It is not as straightforward as one might think... You can write a Java program that reads the results of a Google search on a term like "computer", but the number of results will not be statically present in the output.
You will have to use the Google Custom Search Engine, or the JSON/Atom Custom Search API.