I am creating a java application that interact with Twitter using the library Twitter4J.
I want to download 10 000 nodes from twitter, and then do the statistics on the graph created. The graph is initially saved in a dataset (.txt file).
I must also save enough ReTweet for each node (so I have to check their timeline).
Skipping the instantiation of twitter to perform queries, I have two problems and doubts:
1) How do I manage the problem that the Twitter API have a limited number of required into a slot 15 minutes?
I tried this:
public static RateLimitStatus getApplicationRateLimitStatus(Twitter twitter)
{
try {
Map<String ,RateLimitStatus> rateLimitStatus = twitter.getRateLimitStatus();
for (String endpoint : rateLimitStatus.keySet())
{
if (endpoint.equals("/application/rate_limit_status"))
return rateLimitStatus.get(endpoint);
}
} catch (TwitterException te)
{
te.printStackTrace();
System.out.println("Failed to get rate limit status: " + te.getMessage());
System.exit(-1);
}
return null;
}
public static void control(Twitter twitter)
{
RateLimitStatus app_rate_limit_st = null;
RateLimitStatus user_timeline_limit_st = null;
RateLimitStatus credentials_limit_st = null;
RateLimitStatus friends_list_limit_st = null;
RateLimitStatus followers_list_limit_st = null;
int ctr_req = 7;
try {
if ((app_rate_limit_st = MyRateLimit.getApplicationRateLimitStatus(twitter)).getRemaining() < ctr_req)
{
System.out.println("I'm waiting "+app_rate_limit_st.getSecondsUntilReset()+" seconds for Application Rate Limit, app request remaining: "+app_rate_limit_st.getRemaining());
Thread.sleep((long)app_rate_limit_st.getSecondsUntilReset()*1000);
System.out.println("I woke up!!!");
}
}catch(InterruptedException e) {e.printStackTrace();}
}
In this block of code I checked only requests ApplicationRequest type, but in my application I checked also required type FriendList, FollowersList, UserTimeline and Credentials.
By running my application, is raised that notification
I have exceeded the number of available applications, and can not understand why.
2)
Another problem is with which algorithm should proceed to download the nodes.
I thought about taking a popular node (has many friends and followers, and who interact much with each other).
I tried to take the node, on the friends, his followers, and then the friends and followers of the friends and the friends and followers of followers.
It's a clever technique? They would know better?
Thanks.
Just a thought on how you can workaround the rate limit problem -
Create multiple twitter oauth credentials, you can maintain a list/set of Twitter instance configured with each available credential, when you reach a rate limit for say id1, you can switch to use id2 to fetch records.
Instead of using getApplicationRateLimitStatus, check for functional rate limit status and make a switch, this would help you plan the switch based on available limit for that API.
--Adding code/comments below as per the review comments,
You can do something like below, for each request you can use a connector, in your case you may need to cache few information that can be used to make next call like sinceId and maxId.
You would need to create/register multiple twitter accounts and generate credentials for each of them. I had tried this approach to fasten up fetching information for about 1 M users and it was effective.
You can also cache some recurring information and save few hits to Twitter, like in a network of 10 people its possible to have some percentage of common users/followers so to lookup user information a previously fetched user can be skipped in next request.
getTweetConnector() method will ensure you get a connector that has reset its ratelimit.
Since you are fetching information through multiple API's you can batch up connectors for specific request such that API's with higher rateLimit can have more connectors.
public class TweetUserInfo {
private Set<Twitter> mTwitterConnectorsSet;
private BufferedReader mUserFileReader;
TweetUserInfo(){
mTwitterConnectorsSet = new HashSet<Twitter>();
}
private void initTweetConnectors(String inFile) {
BufferedReader br = null;
try {
String line = null;
String[] lines = new String[4];
int linesIndex = 0;
br = new BufferedReader(new FileReader(inFile));
while ((line = br.readLine()) != null) {
if (linesIndex == 4) {
createAndAddTwitterConnector(lines);
linesIndex = 0;
}
lines[linesIndex] = line;
++linesIndex;
}
if (linesIndex == 4) {
createAndAddTwitterConnector(lines);
}
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (br != null)br.close();
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
private void createAndAddTwitterConnector(String[] lines) {
ConfigurationBuilder twitterConfigBuilder = new ConfigurationBuilder();
twitterConfigBuilder.setDebugEnabled(true);
for (int i = 0; i < lines.length; ++i) {
String[] input = lines[i].split("=");
if (input[0].equalsIgnoreCase("consumerkey")) {
twitterConfigBuilder.setOAuthConsumerKey(input[1]);
}
if (input[0].equalsIgnoreCase("consumersecret")) {
twitterConfigBuilder.setOAuthConsumerSecret(input[1]);
}
if (input[0].equalsIgnoreCase("accesstoken")) {
twitterConfigBuilder.setOAuthAccessToken(input[1]);
}
if (input[0].equalsIgnoreCase("accesstokensecret")) {
twitterConfigBuilder.setOAuthAccessTokenSecret(input[1]);
}
}
Twitter twitter = new TwitterFactory(twitterConfigBuilder.build()).getInstance();
mTwitterConnectorsSet.add(twitter);
}
private Twitter getTweetConnector() {
for (Twitter tc : mTwitterConnectorsSet) {
try {
if (tc.getRateLimitStatus() != null) {
if (tc.getRateLimitStatus().containsKey("/users/lookup")) {
if (tc.getRateLimitStatus().get("/users/lookup") != null) {
System.out.println("tc - "+tc);
System.out.println("tc rate - "+tc.getRateLimitStatus().get("/users/lookup").getRemaining());
if (tc.getRateLimitStatus().get("/users/lookup").getRemaining() > 2) {
return tc;
}
}
}
}
} catch (TwitterException e) {
e.printStackTrace();
}
}
return null;
}
}
Hope this helps.
Related
I would like to know how to determine the data usage for the specific application above Android version pie programmatically. I found from Android 10 onwards subscriber Id is not accessible to third party applications. Hence I'm not able to fetch data usage using networkStatsManager. But I found there are applications in Play store which provide the solution . I would like to know the implementation for the same.
I'm not sure on this but they actually use the /proc/net/dev file generated by the phone to get the data .
An example to do that without permissions would be to use a function like this
public static long[] getNetworkUsageKb() {
BufferedReader reader;
String line;
String[] values;
long[] totalBytes = new long[2];//rx,tx
try {
reader = new BufferedReader(new FileReader("/proc/net/dev"));
while ((line = reader.readLine()) != null) {
if (line.contains("eth") || line.contains("wlan")){
values = line.trim().split("\\s+");
totalBytes[0] +=Long.parseLong(values[1]);//rx
totalBytes[1] +=Long.parseLong(values[9]);//tx
}
}
reader.close();
}
catch (FileNotFoundException e) {
e.printStackTrace();
}
catch (IOException e) {
e.printStackTrace();
}
//transfer to kb
totalBytes[0] = totalBytes[0] / 1024;
totalBytes[1] = totalBytes[1] / 1024;
return totalBytes;
}
SO Question on data in proc-net-dev
So ... I have a bit of software that's supposed to communicate with a memcached server (using no external libraries).
For testing purposes, let's settle on a simple get hello\r\n command.
I start memcached with the -vv option, this is what the command produces via telnet:
<15 new auto-negotiating client connection
15: Client using the ascii protocol
<15 get hello
>15 END
Now here is what the same command issued from my software produces:
<15 new auto-negotiating client connection
I'm connecting as following:
private void reconnect(){
InetSocketAddress remote;
int nofServers = m.servers.size();
for(int i = 0; i < R; ++i){
boolean success = false;
while(!success) {
try {
SocketChannel oldConnection = connections.get(i);
if (oldConnection != null) oldConnection.close();
remote = m.servers.get((myServerIndex + i) % nofServers).address();
SocketChannel chan = SocketChannel.open(remote);
chan.configureBlocking(false);
chan.register(this.selector, SelectionKey.OP_READ);
connections.set(i, chan);
success = true;
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
}
After that, the software falls into simple enough a NIO loop:
#Override
public void run() {
MyRequest curr = null;
this.canHandleNewRequest = true;
while (true) {
if (canHandleNewRequest) {
curr = myQueue.poll();
}
if (canHandleNewRequest && curr != null) {
canHandleNewRequest = false;
for (int i = 0; i < R; ++i) {
connections.get(i).keyFor(this.selector).interestOps(SelectionKey.OP_WRITE);
}
}
try {
selector.select();
Iterator<SelectionKey> it = selector.selectedKeys().iterator();
while (it.hasNext()) {
SelectionKey k = it.next();
it.remove();
if (!k.isValid()) continue;
if (k.isConnectable()) finishConnection(k);
else if (k.isReadable()) this.read(k, curr);
else if (k.isWritable()) this.write(k, curr);
}
} catch (IOException ex) {
ex.printStackTrace();
reconnect();
}
if(curr != null && /*have sent to all servers I need to*/){
curr = null;
this.canHandleNewRequest = true;
}
}
}
where
private void write(SelectionKey k, MyRequest currentRequest){
try {
SocketChannel chan = (SocketChannel) k.channel();
ByteBuffer out = currentRequest.getSendBuffer(); //DO NOT FLIP (to resume sending from last sent position)
assert(chan != null);
assert(out != null);
//System.out.println(new String(out.array()));
chan.write(out); //TODO: make this work!
if(!out.hasRemaining()) { //do not enable read until whole command has been sent
currentRequest.partiallySent();
k.interestOps(SelectionKey.OP_READ);
}
}catch(IOException ex){
ex.printStackTrace();
}
//TODO: create response structure
}
I even tried to substitute the write method for a dummy command provider:
else if(k.isWritable()){
SocketChannel chan = (SocketChannel)k.channel();
ByteBuffer msg = ByteBuffer.wrap("get hello\r\n".getBytes());
msg.flip();
while(msg.hasRemaining()) {
System.out.println("writing ...");
chan.write(msg);
}
k.interestOps(SelectionKey.OP_READ);
}
but this only gets stuck in the "writing" loop (never terminates).
You should think that at least the server should react to that command but it doesn't.
So ... how do I get this working?
The second line from the log providing the command via telnet produces,
15: Client using the ascii protocol
makes me think there might be something I need to send to the server prior to engaging in actual memcached commands... except I seem to miss it in the protocol.
Help would be appreciated.
EDIT
This seems to be the issue: flipping a buffer in the getSendBuffer method and then returning it is not the same as returning it unflipped and then flipping it in the write method.
I find this rather strange. Can this be or is this merely masking a different error?
With NIO you should always check whether all of the buffer has been written, which is not being done doing in the first write block. Having said that, unless there's a lot of data being written, the whole buffer is usually written in a single call to write. So, it's unlikely to be the root problem here.
In the alternative writing code block the hasRemaining condition is negated, it should be:
while(msg.hasRemaining()) {
System.out.println("writing ...");
chan.write(msg);
}
Could you include what's being sent first? Is the first command terminated with \r\n?
This my class TagVertex contain one method that read the tag value=string from textual file
and return it
public class TagVertex extends Vertex<String> {
#Override
public String computeVertexValue() {
String s = null;
try {
BufferedReader bf = new BufferedReader(new FileReader(MyDataSource.TagList1K));
for(int i = 1; i < Integer.parseInt(this.getVertexId().substring(this.getVertexId().indexOf("g")+1)); i++){
bf.readLine();
}
s= bf.readLine();
bf.close();
} catch (Exception e) {
e.printStackTrace();
}
this.setVertexValue(s);
return s;
}
the method is called 1000 times ==>the file is read 1000 times too
It is better to use database instead of textual file?
Accessing the hard drive is always a very slow operation. Databases usually also access the hard drive, so they aren't necessarily faster. They can be even slower, because when the database doesn't run on the same system, the network latency is added (even when it runs on localhost, you have latency due to interprocess communication).
I would recommend you to read the file once and cache the value. When you need to be aware immediately when the file is changed, you could use the new WatchService API to reload the file when it was changed. Here is a tutorial. When it isn't that important that changes on filesystem level are registred immediately, you could also save the time the vertex information was read from the hard drive, and only re-read the value when it's older than a few seconds.
You can create your own in memory database like this.
private static final List<String> lines = new ArrayList<>();
#Override
public String computeVertexValue() {
if (lines.isEmpty())
try {
BufferedReader br = new BufferedReader(new FileReader(MyDataSource.TagList1K));
for (String line; (line = br.readLine()) != null; )
lines.add(line);
br.close();
} catch (Exception e) {
e.printStackTrace();
}
return lines.get(Integer.parseInt(this.getVertexId().substring(this.getVertexId().indexOf("g") + 1)));
}
I don't know if this is the best approach so that's why I'm asking your help guys.
This is my problem, I'm developing an app where you have the 'read' button, when the user hit this button then the program start to read some values and save this values in my database, okay ?
So I think when the user hit 'read' I start a thread, that's because the user can do another thing if he wants without the app been freeze.
But I cannot access the values read by this thread.
There's another way to do that ?
EDIT:
private void jtb_readerItemStateChanged(java.awt.event.ItemEvent evt) {
// getting some values provided by the user
if (buttonReaderState()){
if (supervisory == null)
supervisory = new Supervisory(ip, broadcast, deviceID);
supervisory.start();
}
}
// Supervisory class
public void start(){
Scan scan = new Scan();
Thread t = new Thread(scan);
t.start();
threadState = true;
}
class Scan extends Thread{
public void run(){
// inside the tread I have to initiate another 'supervisory' object, is that right ?
Supervisory s = new Supervisory(ip, broadcast, deviceID);
while (threadState){
try {
s.active();
} catch (IOException ioe) {
ioe.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
public void active() throws IOException, Exception {
// getting this values from my hardware like temperature and anothers things.
for (int i = 0; i < ois.size(); i++) {
ObjectIdentifier oi = ois.get(i);
//analog input
if (i == 1)
aI = Float.parseFloat(getPresentValue(localDevice, oi));
//analog output
if (i == 2)
aO = Float.parseFloat(getPresentValue(localDevice, oi));
//binary input
if (i == 3)
bI = getBinaryValue(getPresentValue(localDevice, oi));
//binary output
if (i == 4)
bO = getBinaryValue(getPresentValue(localDevice, oi));
}
}
After reading this values I would like to show this values in the interface that I'm building, but it seems that I cannot have access to these values (aI,aO,bI,bO).
Pass a reference to the interface you have. E.g. you can add a JFrame owner field to Supervisory class and pass your values there.
I have a project that uses JMF, and records for a short time (a few seconds to a couple of minutes) both the web camera, and audio inputs, and then writes the results to a file.
The problem with my project is that this file is never produced properly, and cannot be played back.
While I've found numerous examples of how to do multiplexed transmission of audio and video over RTP, or conversion of an input file from one format to another , I haven't seen a working example yet that captures audio and video, and writes it to a file.
Does anyone have an example of functioning code to do this?
I've found the reason why I was not able to generate a file from two separate capture devices under JMF, and it relates to ordering of the start commands. In particular, things like Processors will take a datasource, or merging datasource, assign and synchronize the time base(s) and start/stop the sources for you, so the extra work I was trying to do starting the datasources manually is utterly redundant, and throws a wrench in the works.
This was a lot of painful trial and error, and I would suggest you read every line of code, understand the sequencing, and understand what has been included, and what has been left out and why before trying to implement this yourself. JMF is quite the bear if you're not careful.
Oh, and remember to catch exceptions. I had to omit that code due to length restrictions.
Here's my final solution:
public void doRecordingDemo() {
// Get the default media capture device for audio and video
DataSource[] sources = new DataSource[2];
sources[0] = Manager.createDataSource(audioDevice.getLocator());
sources[1] = Manager.createDataSource(videoDevice.getLocator());
// Merge the audio and video streams
DataSource source = Manager.createMergingDataSource(sources);
// Create a processor to convert from raw format to a file format
// Notice that we are NOT starting the datasources, but letting the
// processor take care of this for us.
Processor processor = Manager.createProcessor(source);
// Need a configured processor for this next step
processor.configure();
waitForState(processor, Processor.Configured);
// Modify this to suit your needs, but pay attention to what formats can go in what containers
processor.setContentDescriptor(new FileTypeDescriptor(FileTypeDescriptor.QUICKTIME));
// Use the processor to convert the audio and video into reasonable formats and sizes
// There are probably better ways to do this, but you should NOT make any assumptions
// about what formats are supported, and instead use a generic method of checking the
// available formats and sizes. You have been warned!
for (TrackControl control : processor.getTrackControls()) {
if (control.getFormat() instanceof VideoFormat || control.getFormat() instanceof AudioFormat) {
if (control.getFormat() instanceof AudioFormat) {
// In general, this is safe for audio, but do not make assumptions for video.
// Things get a little wonky for video because of how complex the options are.
control.setFormat(new AudioFormat(AudioFormat.GSM));
}
if (control.getFormat() instanceof VideoFormat) {
VideoFormat desiredVideoFormat = null;
Dimension targetDimension = new Dimension(352, 288);
// Search sequentially through this array of formats
VideoFormat[] desiredFormats = new VideoFormat[] {new H263Format(), new JPEGFormat(), new RGBFormat(), new YUVFormat()};
for (VideoFormat checkFormat : desiredFormats) {
// Search the video formats looking for a match.
List<VideoFormat> candidates = new LinkedList<VideoFormat>();
for (Format format : control.getSupportedFormats()) {
if (format.isSameEncoding(checkFormat)) {
candidates.add((VideoFormat) format);
}
}
if (!candidates.isEmpty()) {
// Get the first candidate for now since we have at least a format match
desiredVideoFormat = candidates.get(0);
for (VideoFormat format : candidates) {
if (targetDimension.equals(format.getSize())) {
// Found exactly what we're looking for
desiredVideoFormat = format;
break;
}
}
}
if (desiredVideoFormat != null) {
// If we found a match, stop searching formats
break;
}
}
if (desiredVideoFormat != null) {
// It's entirely possible (but not likely) that we got here without a format
// selected, so this null check is unfortunately necessary.
control.setFormat(desiredVideoFormat);
}
}
control.setEnabled(true);
System.out.println("Enabled track: " + control + " (" + control.getFormat() + ")");
}
}
// To get the output from a processor, we need it to be realized.
processor.realize();
waitForState(processor, Processor.Realized);
// Get the data output so we can output it to a file.
DataSource dataOutput = processor.getDataOutput();
// Create a file to receive the media
File answerFile = new File("recording.mov");
MediaLocator dest = new MediaLocator(answerFile.toURI().toURL());
// Create a data sink to write to the disk
DataSink answerSink = Manager.createDataSink(dataOutput, dest);
// Start the processor spinning
processor.start();
// Open the file
answerSink.open();
// Start writing data
answerSink.start();
// SUCCESS! We are now recording
Thread.sleep(10000); // Wait for 10 seconds so we record 10 seconds of video
try {
// Stop the processor. This will also stop and close the datasources
processor.stop();
processor.close();
try {
// Let the buffer run dry. Event Listeners never seem to get called,
// so this seems to be the most effective way.
Thread.sleep(1000);
} catch (InterruptedException ex) {
Logger.getLogger(getClass().getName()).log(Level.SEVERE, null, ex);
}
try {
// Stop recording to the file.
answerSink.stop();
} catch (IOException ex) {
Logger.getLogger(getClass().getName()).log(Level.SEVERE, null, ex);
}
} finally {
try {
// Whatever else we do, close the file if we can to avoid leaking.
answerSink.close();
} catch (Exception ex) {
Logger.getLogger(getClass().getName()).log(Level.SEVERE, null, ex);
}
try {
// Deallocate the native processor resources.
processor.deallocate();
} catch (Exception ex) {
Logger.getLogger(getClass().getName()).log(Level.SEVERE, null, ex);
}
}
}
// My little utility function to wait for a given state.
private void waitForState(Player player, int state) {
// Fast abort
if (player.getState() == state) {
return;
}
long startTime = new Date().getTime();
long timeout = 10 * 1000;
final Object waitListener = new Object();
ControllerListener cl = new ControllerListener() {
#Override
public void controllerUpdate(ControllerEvent ce) {
synchronized (waitListener) {
waitListener.notifyAll();
}
}
};
try {
player.addControllerListener(cl);
// Make sure we wake up every 500ms to check for timeouts and in case we miss a signal
synchronized (waitListener) {
while (player.getState() != state && new Date().getTime() - startTime < timeout) {
try {
waitListener.wait(500);
} catch (InterruptedException ex) {
Logger.getLogger(getClass().getName()).log(Level.SEVERE, null, ex);
}
}
}
} finally {
// No matter what else happens, we want to remove this
player.removeControllerListener(cl);
}
}