I want to show my users some statistics such as hits/second on Google App Engine. I started to roll my own:
On each page view, add 1 to a count in memcache.
Each minute:
Read and reset the count and also set a "since" variable to now.
Divide the number of hits by the amount of time since I last calculated.
Save the data to an entity in the datastore.
Throwing out data that's really old.
I then realised that this is non-trivial and there must be a library to do it, however I can't find one that works for me. I looked briefly at rrd4j and JRobin but I'm not sure they they're usable on Google App Engine without quite a lot of rewriting. Does anyone have any more ideas?
Try new technique mentioned in this post http://googleappengine.blogspot.com/2012/07/analyzing-your-google-app-engine-logs.html.
It requires some additional work but it's worth trying. I'm using Mache (java framework) to ingest appengine logs into BigQuery and BigQuery API to query for results. Now pick a fancy javascript charts library and impress your users. Very powerful, flexible and scalable solution.
Perhaps ProdEagle works for you. I think they do pretty much exactly what you want, and I belive they also have logic for handling data that is deleted from memcache without making a big hole in your graph.
I seem to remember Twitter commons has what you need, but I don't know if they could be easily portable to GAE: https://github.com/twitter/commons
Consider using Mixpanel. You can submit arbitrary events and then extract aggregate information from an API... or just use the provided charts & graphs.
Since no-one seems to have an answer for me, I'm going to assume that there's no common library for doing this and I'll have to write one. I'll open source it and link from here if it feels like good code.
You could probably use Google Analytics. You'd just need to copy and paste some javascript into your templates.
Related
I am creating an image/graphics intensive application on android. Thus I have decided to keep images at server side and fecth them in batches when needed for each user. Apart from this I would like to manage some minor user data at backend for any future extension to the app or dynamic loading of some content.
For this I am looking out for the easiest but not a very rigid back-end solution. After some research I have boiled down to below mentioned options(In the order of priority):-
Amazon SDK for android :- It looks like this provides a lot of pre-built components but I am not sure how flexible it is when doing some custom back-end coding/feature implementation.
Parse :- Easy to understand and use but not flexible when it comes to custom feature development.
Amazon EC2 Java Backend:- I will have to do all the server side coding from scratch here but this will provide complete independence in feature implementations. Though I would love if I can find some code samples relates to user management, backend db management and java restful web services.
Any suggestions or pointers that you guys have in the above choice would be great
Thanks in advance
I have been using Parse but I haven't explored the other 2. So, this may not be a comprehensive answer but I would try to give you some pointers based on my experience with Parse.
I have been into Android development for quite some time now but I do not have any significant expertise (I would say very minimal) on the backend. Also, you mentioned you wish to work on graphics/image intensive application. As far as the application I use Parse for is more of user data and minimal images, (requiring extensive relational database).
Parse makes it really simple to create the backend structure. And the client SDK is also very powerful. Their API's are very straight-forward and doesn't require you to worry about writing complex queries, caching them and saving the data. Given my background as I mentioned above, I would say there is no learning curve involved into getting started with the dev. You can simply start building your app right away!
Also, Parse uses AWS S3 on the backend with Mongo-DB. So, I believe computation on the server side should not be a problem. Server side logic can be implemented using ParseCloud (requires some javascript). But, if you plan to write some complex algorithms, I am not very sure how much can that be done.
Documentation of Parse on Android is quite good to get through most of the dev. Extensive doc for iPhone dev.
As far as cost structure goes, it allows 1 million free API requests per month and this is very much sufficient to get through quite a number of users. In your case, the storage should be of more concern. Parse allows 1GB free and some 20 cents above per GB.
Hope this helps!
I am looking out for the easiest but not a very rigid back-end solution
Have you considered AppEngine? Here's a tutorial about how to get app engine working for you fast
You can store up to 5 GB of blob storage for free, should be more than enough for experimenting. If you go over you can pay the $0.13/GB/mo extra for blob storage, which is more than reasonable.
I don't know what kind of app you are doing, but I'll propose one approach.
Use https://imageshack.com/ for images.
Create your user saving data application with a lightweight webservice (REST+JSON)
and expose it at heroku (https://www.heroku.com/) with your prefered language/plataform.
It could be java or ruby.
Using imageshack for images will save cloud space for you and the service is quite fast.
I am looking for an open source solution to store and monitor some application performances.
To be more precise, I use several Java components in the software I develop and I would like to gather performance statistics for each of these components in order to figure out on what I need to focus to keep fast processing.
The idea would be to send a message to a repository to store some timestamps (everytime a Java component starts or ends) and having a web interface to browse the timestamps, and do some analytics on top of them.
These needs seem really basic but unfortunately I haven't found anything on the web, probably because I don't know the right terminology for this kind of tools.
Could someone recommend me such a tool?
Thanks in advance !
Adrien
What you described is RRDtool that stores time-series data. To access it from Java, there is java-rrd.
I also get the impression that you are looking for whole solution instead of just data back-end. If so, check out following open source cluster monitoring system: cacti, ganglia and graphite. They all have web interface. Cacti and ganglia have RRD-like back-end, while graphite has its own whisper database, etc.
I'm looking for a chart library capable to handle large amount of data points - 300 millions per a chart and even more. Surely drawing, caching and approximation should be implemented with intelligence there.
Actually I need to represent waveforms but not only them.
Target platform is Java, data comes from files.
UPD: PC, Swing.
Not java, but CERN does massive data crunching and distros/plots may well have these kinds of data volumes. They use the root package which is c++. You can download it, although couldn't see a licence. It's prob open source.
Or alternatively, take a look at R which might do what you need.
I have been happy with my use of JChart2D. Switching to it from JFreeChart saved us considerable processor use, and it has traces that compute multiple inputs into a mean point for speed and memory saving. I've never used them seeing as how I haven't needed to yet. I have put extremely large sets of data into a normal trace by accident, and it didn't seem to be a problem.
There may be a better charting system out there, but this one gets the job done quick and effectively, it's free, open-source, based off of JPanels, and the author is around to answer questions and correct problems.
I don't see a way to handle that amount of data on an android phone, whatever librairy you use. You should think about doing all this processing on a server or a cloud and then put either an approximated set of data that would approximate the chart or even the result of the chart as an image file so that android phones can download it from the server without processing the data.
Regards,
stéphane
I assume that you are talking about a Swing Application.
I make use of JGoodies for all my Swing applications including Graphs and Charts.
Takes a bit getting use to it, but once you are use to it building UI's is fairly quick and easy.
The only problem is that there is a developer license cost involved.
You can download the Java Webstart examples to have a look at what it is capable of.
I'm looking for the most suitable tool for generating dynamic geographical maps in Java (with styles based on custom business data: colors, labels etc. will be dynamically set). After some searching, only GeoServer (and the underlying GeoTools library) seems to fit.
Although I'm really happy with this solution, I'm afraid I might miss something and make some decision based on incomplete input. Any suggestion ? Any viable alternative to compare against ?
Make sure to check out MapServer.
Both MapServer and GeoServer support OGC-compliant WMS requests, so they're basically inter-exchangeable if you use something like OpenLayers.
MapServer's got a handy installer that lets you install every aspect that you might need to get your server and web application going. It runs on Apache, which I find a big advantage over GeoServer, which runs on Jetty (it's all-java).
However, I'm now using GeoServer myself now. It's really easy to administer stuff via its web interface, and overall it just feels pretty polished. Maybe it's because I've been using it for a while now, but I just get stuff done quickly with it.
Take a look at OpenMap. It is a descent alternative to manipulate/display geospatial data.
The thread may be old but in case anyone else is looking for a similar solution it is worth checking out Nasa World Wind.
http://worldwind.arc.nasa.gov/java/
I have a website which is pretty good but with very less information.
So i felt like adding informtion like news regarding particular sector(for eg politics, hollywood etc). I believe crawlers are best approach to do so? Is my understanding correct, please suggest if you feel any other way to get information without using crawlers from various sources.
Secondly I am doing research from last 2 days and I cannot find a particular source which is capable of doing so. Now I want crawlers to find information, normalize and store in mysql database. Sounds pretty simple ha. But It isnt for me.
As this is very resource and time consuming . what all things should i take into consideration before choosing a crawler. Also I wish to customize it so any tool which is open source and good to be customized will be great.
Any source giving information and research about factors need to take into consideration while creating crawlers or educating about crawlers will be great.
I prefer coding in java but i can code in any other language in case you feel that you have some language.
I hope i have given enough information. Please dont hesitate if you need any more information to give suggestion.
You can use httrack to copy a target website. There is one firefox plugin as well named spiderzilla. But, they will just save the pages.
If you want to parse the data in the pages, then you can use simple_html_dom and store the information in mySQL.
Try the GNU Wget tool. You can add a lot of intelligence to the way it crawls and creates data dumps of web pages. It is open-source and customisable as well, and very fast too.