I'm planning on hosting a JRuby on Rails app on Google AppEngine/Java. I found a great blog post by Ola Bini on how to to this, but in the summary he says:
Overall, JRuby on Rails works very
well on the App Engine, except for
some smaller details. The major ones
are the startup cost and testing. As
it happens, you can’t actually get
GAE/J to precreate things. Instead
you’ll have to let the first release
take the hit of this. Now, GAE/J does
a let of preverifying of bytecodes and
so on, so startup is a bit more heavy
than on other JDKs. One runtime takes
about 20 seconds wall time to startup,
so the first hit takes some time.
I don't fully understand this. How often, under what circumstances, will a runtime need to be started up? A regular 20 second lag is likely to be an issue.
App Engine will start new runtimes for you whenever demand is outstripping the currently running instances. It will then shut down instances when demand is lower. Ultimately, this means that all of your instances could be shut down if your app is not used for a certain amount of time. Then, the next time a user tries to access your app, a new instance will need to be started, or "spun up" as some people call it.
As of March, the app engine team wouldn't give any official estimate on how long an instance will stay up:
7:40pm] nwinter: Is it possible to get a rough estimate of how long an app
instance will stick around once spawned?
[7:40pm] marzia_google: #nwinter, not really
[7:40pm] marzia_google: there are no garuntees
[7:41pm] nwinter: No average time or anything?
[7:42pm] marzia_google: #nwinter i'm not sure an average time would be
meaningful, even if i knew off hand what it was ( i don't)
[7:42pm] marzia_google: since it really can be quite variable
[7:42pm] Kardax: Re instance lifetime: So an app instance could last a few
seconds or a few hours? Just curious
[7:43pm] dan_google: nwinter: Apps are evicted by least-recently-used on an
app server. As someone noted recently (forums or chat I forget), low
traffic could mean lots of "restarts", but so could spikes in traffic which
may start new instances on multiple app servers.
[7:43pm] nwinter: #dan_google: good to know!
[7:43pm] dan_google: Kardax: Yes, depending on the weather. By which I
mean, request patterns, other apps on each app server, and so forth. Not
really predictable.
This is the transcript of a chat with the app engine team. I have deleted the non-relevant lines in the transcript like "so and so logged in." The full transcript can be found here
Related
I have built a java program within an android app that parses some data from an online website. This data is to be used from the app to show the content(organized) to the user. I tried to optimize my code as much as I could but there's not much I can do since it's a lot of data. So what I did instead was deploying the java program on HEROKU and let the server side do the work and give the result in a simple html which I can easily parse with no major delay. The thing is that this worked out pretty fine. I got a high increase in performance yet with one little problem. When I open the app for the first time in say 2 days it results to be a lot slower, but on the second run just after that it seems to be a lot faster. Now I am guessing that the HEROKU server works on some cache-like way in that the least recent run dynamic web sites get no priority until a request comes from the client side, considering that in 2,3 consequent runs I get a very high increase in performance. Now, my question is, is there a way I can sort of "give priority" to my HEROKU java program or is there another free dynamic web site that allows you to deploy a war and presents no such performance issues. To some it might seem as a no big deal. In particular I get performance increase of say from 6 seconds to 2 seconds which is actually quite a big deal since app users usually do not tolerate such kinds of delays.
Heroku puts free apps to sleep. The first request after it sleeps will restart the app, which means you have to wait longer.
For more info see Heroku's sleeping policy for Free dynos.
I'm hosting a java app app on app engine, or some reason i see that from today the response times are extremely slow - 10kms + !!! the gae status page shows everything is ok, Does anyone have an answer or similar experience ?
Second issue, i see that many request starts only a few seconds after it has been received, there is a delay in executing the request, does anyone know how i can fix it ?
p.s
I changed my instances from f1 to f2 to see if maybe it will help but the result is the same.
Thank you
The GAE Google group is likely still the best place to ask questions like this.
Could it be that you are just seeing an increased number of warmup requests? In this case going from F1 to F2 will not make a huge difference. Depending on your application instance startup can be reduced by changing the instance class. But this change alone will not reduce the time to a more reasonable response time of ~1 second.
The following best practices allow you to reduce the duration of loading requests:
Load only the code needed for startup.
Access the disk as little as possible.
In some cases, loading code from a zip or jar file is faster than loading from many separate files.
You can also try to add a few resident instances. The GAE scheduler will then put extra traffic on resident instances and launch new dynamic instances in the background. Since residents are started ahead of time this will hide some latency from users.
Can someone recommend an open source scheduler that would lend itself to being embedded
as the heart of a specialized web based scheduler? A C- or Java-based scheduler would be my
first choice to work with.
The finished project would allow someone, via the internet, to add, delete and change tasks scheduled on the local machine by monthly, weekly, daily, and by time of day. The tasks
would be fairly simple: to display messages and play WAV files on the local machine on scheduled dates and at times specified by the remote programmer.
Ok, why? Well, my wife and myself moved her mother to our town a couple of years ago, because she could not or would not tend to her own affairs including eating and taking her insulin on a rigorous schedule. She is a type one diabetic. She is a widow and had been living by herself for about ten years. My wife had been tending to her bills and affairs remotely from our home a thousand miles away. My mother-in-law had a dozen different doctors that did not of one another and she was being way over-medicated with one medication counteracting another. We found out that she had not been careful with her diabetes and that this had resulted in a trip by EMS to the emergency room on the average of every other month. Strangely, she is not totally senile, though her short term memory is pretty much shot, but, she is, and always has been, a "dyed-in-the-wool" slacker. My wife and I, both, work full time jobs, from before daylight until after dark, still, my wife manages to call her mother three times a day, to tell her to eat and take her insulin and then spends about two hours each evening with her mother before coming home. This machine would be, essentially a headless system that served no function other than to flash pre-programmed messages to small monitor and play audio "nags" at the appropriate times. "Get up and your eat breakfast", "It's time to do your insulin", "Give the dog its pill", "Get ready to go to your doctors appointment" and so on. With no keyboard or mouse and the front panel switches disabled, she is enough of a Luddite that I don't think that she will think of pulling the cord out of the wall socket.
Anyway, that's where I'm trying to go. I'm a reluctant programmer, but, I have written some large and complex programs over the years and in a number of different languages and to make the wife's life a bit easier, I can do this. A scheduler that could be modified to become a large block of code within the overall program, would save me a great deal of time and head scratching.
Get a Linux box, SSH in to it, and add entries to the crontab. As for the alerting program, that'll be specific to your task.
You got me interested. We're looking into Spring Batch at the office, but that's less about scheduling jobs and more about heavy processing. I checked out the FAQ, which led me to Quartz... it looks pretty nifty. Here are its features: http://www.quartz-scheduler.org/overview/features.html
Quartz, as mentioned by Dondo is a kind of industry standard for scheduling. It's very popular and used a lot.
Alternatively you could use the Timer API that comes with Java EE. This is a fairly basic thing, but still quite powerful. See this for a small example SIMPLEST POSSIBLE EJB 3.1 TIMER.
Java EE also gives you the tools to easily create a GUI (via Java server faces) and to have some CRUD logic to enter new tasks into your system and persist them with the Java persistence API to a DB.
Of course, if you don't have any experience yet with Java EE (or Spring, or Quartz) simply learning those technologies may be more time consuming than actually building what you have in mind.
We have an Java ERP type of application. Communication between server an client is via RMI. In peak hours there can be up to 250 users logged in and about 20 of them are working at the same time. This means that about 20 threads are live at any given time in peak hours.
The server can run for hours without any problems, but all of a sudden response times get higher and higher. Response times can be in minutes.
We are running on Windows 2008 R2 with Sun's JDK 1.6.0_16. We have been using perfmon and Process Explorer to see what is going on. The only thing that we find odd is that when server starts to work slow, the number of handles java.exe process has opened is around 3500. I'm not saying that this is the acual problem.
I'm just curious if there are some guidelines I should follow to be able to pinpoint the problem. What tools should I use? ....
Can you access to the log configuration of this application.
If you can, you should change the log level to "DEBUG". Tracing the DEBUG logs of a request could give you a usefull information about the contention point.
If you can't, profiler tools are can help you :
VisualVM (Free, and good product)
Eclipse TPTP (Free, but more complicated than VisualVM)
JProbe (not Free but very powerful. It is my favorite Java profiler, but it is expensive)
If the application has been developped with JMX control points, you can plug a JMX viewer to get informations...
If you want to stress the application to trigger the problem (if you want to verify whether it is a charge problem), you can use stress tools like JMeter
Sounds like the garbage collection cannot keep up and starts "halt-the-world" collecting for some reason.
Attach with jvisualvm in the JDK when starting and have a look at the collected data when the performance drops.
The problem you'r describing is quite typical but general as well. Causes can range from memory leaks, resource contention etcetera to bad GC policies and heap/PermGen-space allocation. To point out exact problems with your application, you need to profile it (I am aware of tools like Yourkit and JProfiler). If you profile your application wisely, only some application cycles would reveal the problems otherwise profiling isn't very easy itself.
In a similar situation, I have coded a simple profiling code myself. Basically I used a ThreadLocal that has a "StopWatch" (based on a LinkedHashMap) in it, and I then insert code like this into various points of the application: watch.time("OperationX");
then after the thread finishes a task, I'd call watch.logTime(); and the class would write a log that looks like this: [DEBUG] StopWatch time:Stuff=0, AnotherEvent=102, OperationX=150
After this I wrote a simple parser that generates CSV out from this log (per code path). The best thing you can do is to create a histogram (can be easily done using excel). Averages, medium and even mode can fool you.. I highly recommend to create a histogram.
Together with this histogram, you can create line graphs using average/medium/mode (which ever represents data best, you can determine this from the histogram).
This way, you can be 100% sure exactly what operation is taking time. If you can't determine the culprit, binary search is your friend (fine grain the events).
Might sound really primitive, but works. Also, if you make a library out of it, you can use it in any project. It's also cool because you can easily turn it on in production as well..
Aside from the GC that others have mentioned, Try taking thread dumps every 5-10 seconds for about 30 seconds during your slow down. There could be a case where DB calls, Web Service, or some other dependency becomes slow. If you take a look at the tread dumps you will be able to see threads which don't appear to move, and you could narrow your culprit that way.
From the GC stand point, do you monitor your CPU usage during these times? If the GC is running frequently you will see a jump in your overall CPU usage.
If only this was a Solaris box, prstat would be your friend.
For acute issues like this a quick jstack <pid> should quickly point out the problem area. Probably no need to get all fancy on it.
If I had to guess, I'd say Hotspot jumped in and tightly optimised some badly written code. Netbeans grinds to a halt where it uses a WeakHashMap with newly created objects to cache file data. When optimised, the entries can be removed from the map straight after being added. Obviously, if the cache is being relied upon, much file activity follows. You probably wont see the drive light up, because it'll all be cached by the OS.
I'm not new to improving my cold start time, I've spent many hours trying different things. I'd like if possible to know exactly what the Google App Engine does during a cold start.
I have a log statement as described here http://code.google.com/intl/nl/appengine/kb/java.html#performance to show when my code first gets control.
I have two apps that I have been testing, one is simple, and my code first gets control after about 1 second.
The other one has lots of files and stuff and my code first gets control after about 2 seconds. This one doesn't use any more libraries than the other one, however it does have a lot more jsps and java classes.
Could simply having more java and jsp classes cause slower cold start, even if the class isn't used?