An Android app I'm writing involves quite a lot of downloading of content (think podcatcher/RSS).
I would like to be able to give the user an indication of how many bytes they've downloaded, so they can make the decision whether they want to use Wifi or not.
To this end, I have found a way of counting the number of bytes read by the app, by wrapping an InputStream in a simple CountingInputStream.
However, this does not take into consideration basic things like packet headers and HTTP headers. More importantly, it does not take into consideration any compression that content may be encoded with.
So, how many bytes did my application download over the network? I'm not so interested in the number of bytes uploaded, but if know how, don't be shy.
I have gone down a fairly low level approach as I am feeding the input stream into an XML PullParser. I will also be needing to do a similar exercise with dumping bytes (images in this case) straight onto the SD Card.
Is this the best approach? What am I missing?
ufff... I think this is pretty transparent to underlying protocol, so you can't count all these bytes used in session or link layer, and operators like to charge even for control bytes which are not in any way visible to end user. Also they count traffic in both directions (your reqest to server takes also some), so - good question is: how to measure needed traffic/money for downloading that picture... ?
This isn't a direct answer, but you could try asking someone who has solved a similar problem before, e.g. a data counter application. I've used NetCounter by Cyril Jaquier (http://www.jaqpot.net/netcounter/), and he claims his software is open source. I couldn't get his download link to work, but there's a contact email address. If you got his source code, you should be able to use the same method as him.
As I know, there are two ways to count data traffic. One is /sys/class/net/{interface}/statistics as mentioned in android app named netCounter, the other is /proc/net/dev which is used in android app named wifi-tether. But I don't know the difference between these two methods nor which is better.
The number of bytes received by a particular app is stored in /proc/uid_stat//tcp_rcv where app_uid is the uid of your app on the particular device.
Related
The NeuralDataSet objects that I've seen in action haven't been anything but XOR which is just two small data arrays... I haven't been able to figure out anything from the documentation on MLDataSet.
It seems like everything must be loaded at once. However, I would like to loop through training data until I reach EOF and then count that as 1 epoch.. However, everything I've seen all the data must be loaded into 1 2D array from the beginning. How can I get around this?
I've read this question, and the answers didn't really help me. And besides that, I haven't found a similar question asked on here.
This is possible, you can either use an existing implementation of a data set that supports streaming operation or you can implement your own on top of whatever source you have. Check out the BasicMLDataSet interface and the SQLNeuralDataSet code as an example. You will have to implement a codec if you have a specific format. For CSV there is an implementation already, I haven't checked if it is memory based though.
Remember when doing this that your data will be streamed fully for each epoch and from my experience that is a much higher bottleneck than the actual computation of the network.
I want to capture packets from a crowded network and classify it by user request. I use java and jpcap for capturing packets. now I'm using array threads for each req packet. it's realy slow and there is many conflicts among threads. what's the best data structure for this purpose?
after that, I'm saving packets (req/response) into a file, that is very slow for such a heavy work. what should I do for saving my datas?
thank you
I'll need some example code to help you with the first part, because your question is a little vague. what's the best data structure for this purpose? could be anything depending on what exactly you need that structure for.
The JpcapWriter is pretty fast. I need to capture roughly 330mb of raw packet data from a 2 minutes of capture, and I haven't had a problem with being able to write it to disk while continuing to capture.
I'm attempting to transfer a large two dimensional array (17955 X 3) from my server to the client using Asynchronous RPC calls. This is taking a very long period of time which is especially bad because the data is needed in order to initialize the application. I've read that using a JSON object might be faster, but I'm not sure how to do the conversion in Java as I'm pretty new to the language and GWT, and I don't know if the speed difference is significant. I also read somewhere that I can zip the data, but I only read that in a forum and I'm not sure if it's actually possible as I couldn't find information for it elsewhere. Is there any way to transfer large amounts of data from server to client? Thanks for your time.
Read this article on adding JSON capabilities to GWT. In regards to compression this article explains gzipping with GWT.
Also the size of your array is still very large even with the compression you may achieve with gzipping, which will vary depending on how much data is repeated in your array. You may want to consider logically breaking up the array in multiple RPC calls if at all possible.
I would recommend revisiting your design if your application needs such a large amount of data to initialize.
As other's pointed out, you should re-consider your design because even if you are able to solve the data transfer speed issue somehow you will likely find other issues waiting for you:
Processing large amount of data in the browser can be slow.
Lot of data means a lot of used-up memory
What you can think about is:
Partitioning the data:
How is your user going to cope with a lot of data. Your user will probably need some kind of user interface aid to be able to work with such a huge data. If you are going to use paging, tabs or other means to partition the data for user's consumption, why not load the data on demand. For example, you can load a single page of records if you are using a paging grid or you can load a single tab worth of records if you are going to use tabs. Similary, if you are going to allow filtering on the records, you can set a default filter after the load to keep the data to a minumum.
Summarizing the data:
You can also summarize the data on the server, if you are not going to show each row to the user. For example you can initially show summary for each group of records and let the user drill-down in a specific group
For my BlackBerry application, I am using a single thread to retrieve images from the server one at a time. My application has a number of images and it takes too long to load all the images. How can I speed this up?
If these are static images, you can also do something like CSS sprites - stitch them all together into one big image, then in code you display the portion of the large image that corresponds to the original image you want.
The last two arguments to Graphics.drawImage(...) indicate where to start drawing from the original image, and that's how you would select the part you want.
Use multiple threads instead of one. Also, if this is a server that you control, consider pre-sizing the images for the target devices or having the device send its size to the server to generate and cache device specific images.
its too late but sorry for that.
i have used observer pattern for it.
Link:-http://en.wikipedia.org/wiki/Observer_pattern
thankx
#Peter
Threads on a mobile phone is a bad idea. Firstly threading on phones suck! secondly phones can't really handle more then one http connection at a time stuff bombs out.
#Userbb
You can do sneeky things like stream them via a socket connection OR include multiple images in a single http request (creating a connection and http headers have overhead).
and also deff do what #peter suggested about resizing serverside.
I was wondering if anyone had experience retrieving data with the 3270 protocol. My understanding so far is:
Connection
I need to connect to an SNA server using telnet, issue a command and then some data will be returned. I'm not sure how this connection is made since I've read that a standard telnet connection won't work. I've also read that IBM have a library to help but not got as far as finding out any more about it.
Parsing
I had assumed that the data being returned would be a string of 1920 characters since the 3278 screen was 80x24 chars. I would simply need to parse these chars into the appropriate fields. The more I read about the 3270 protcol the less this seems to be the case - I read in the documentation provided with a trial of the Jagacy 3270 Java library that attributes were marked in the protocol with the char 'A' before the attribute and my understanding is that there are more chars denoting other factors such as whether fields are editable.
I'm reasonably sure my thinking has been too simplistic. Take an example like a screen containing a list of items - pressing a special key on one of the 24 visible rows drills down into more detailed information regarding that row.
Also it's been suggested to me that print commands can be issued. This has some positive implications - if the format of the string returned is not 1920 since it contains these characters such as 'A' denoting how users interact with the terminal, printing would eradicate these. Also it would stop having to page through lots of data. The flip side is I wouldn't know how to retrieve the data from the print command back to Java.
So..
I currently don't have access to the SNA server but have some screen shots of what the terminal will look like once I get a connection and was therefore going to start work on parsing. With so many assumptions and not a lot of idea on what the data will look like I feel really stumped. Does anyone have any knowledge of these systems that might help me back on track?
You've picked a ripper of a problem there. 3270 is a very complex protocol indeed. I wouldn't bother about trying to implement it, it's a fool's errand, and I'm speaking from painful personal experience. Try to find a TN3270 (Telnet 3270) client API.
This might not specifically answer your question, but...
If you are using Rational Developer for z/OS, your java code should be able to use the integrated HATS product to deal with the 3270 stream. It might not fit your project, but I thought I would mention it if all you are trying to do is some simple screen scraping, it makes things very easy.