I am having multiple servers which run the same back-end process in JAVA through cron-tab scheduling.
Earlier I had just one server on which I used to schedule all jobs. This was easier to manage.
Now with increased servers and same process running on all servers with same code base, I want to schedule same cron-jobs on all machines.
I am facing problem, because I have to face lot of trouble and do manual work to change / update all servers if there is a single change.
Can anybody tell me some way to manage scheduling on multiple servers from a common point ?
Is there any tool / library / API etc to handle this kind of situation ?
As of now, I am updating / copying (using SCP) JAR on all servers and then restart all processes by individually logging into to each of the servers.
As of now, I will prefer If I do not have to change my JAVA code to manage this(In log run, I can think of that).
However, If I can create a small JAVA application (using Quartz) OR even a shell script to handle scheduling, I will go for it.
I want to be clear that updating JAR on all servers is not my problem, I am working on setup build script(ANT) for that.
My concern is common scheduling code for all servers.
Is there a way to schedule jobs on one server which starts jobs on remote server ? If this happens, I only have to manage scheduling on one server. I wont need to worry about clock sync issues as well ..
You can create a text file, containing the crontab entries in it, and when you deploy the jar file in each server, you can update the crontab at the same time as well, using ant, by running the command crontab mycrontab.txt.
Related
I want run a background process at a specific time. I want that process to be run on the server even without running the application from End-User side. The application is made using Spring. Please suggest how to approach for it.
Thanks and regards
Souvik
I depends highly on what platform you are working on, and what you want to achieve.
If it is a simple application, that you simply want to invoke that on specific time, then you can use scheduling tools available on your platform, for example, crontab for Unix, or scheduled task (at) for Windows.
If you want the application to be run as a daemon process, and the application itself will handle the scheduling, then you need to solve two problem: 1. create a daemon process (aka system service), and 2. doing scheduling in Java.
For problem 1, there are already answer for it. Just have a search on Google on "Java System Service" will give you some other useful tools too, like Java Service Wrapper
For problem 2, there are a lot of way to perform scheduling in Java. You can do it by built-in Timer etc, or using scheduling library like Quartz
We use Eclipse to develop and Websphere 7.0 to run our code. We use a plugin for Eclipse which allows you run run a copy of WAS locally and integrated with Eclipse. This works but it's not great. Each time I have to make a change I have to run my Ant script (5 mins.) then do my install into WAS and restart the app in WAS (another 5 mins). This is not efficient (actually it's downright annoying!)
What's the best setup Websphere allows, ignoring for a min what is easy to do etc. I want to know what to aim for. Obviously I would love to be able to save in eclipse then go to my browser and refresh the screen, am I mad?
I have looked at this about using Websphere's native Ant, and it is one avenue but I'm sure there's better.
Can I use jpda like I do when starting Tomcat and develop directly in the code? I will still need Ant when doing a full re-build as I have some Java file generation to do but for the in-process builds???
Thanks-in-advance for any and all help.
There is not just one straightforward single best answer as it depends a lot on the application you are doing.
If you rely on ant to make the build and that takes five minutes for any update you do then it may be hard to get away from that. You may not have to do it like that, you may be able to do incremental compilations (only compiling the updated classes) and that should be a lot quicker (but since I don't know anything about your application it is hard for me to tell you that this is a way forward for you). The updates you make in WAS and if that requires a total restart of the application depends on many different factors.
If I update only a jsp file it usually does not require a full application restart, but if you start to update the classes and definitely when you update your web.xml or something like that, a full restart of the application may be required.
The time it takes for you to restart your application depends a lot on what actions your application will perform when you start it up. If you have an application that takes five minutes to start up, then every restart will cost a lot. Is there something you can do to reduce the restart time in certain configurations? Can you divide your application into separate deployables to reduce the startup time for each individual application?
It all depends on your needs and requirements.
We used to have long startup times for our local servers, but since we had upgraded disks (SSD-disks) and enough memory the server startup time has been reduced by 80-90%.
I am developing a server application in java (SE) + some open source libraries. This is a game, so I think it will have to be updated sometimes. As long as the app is tracking the state of the clients, using it, and also supports client sessions (in a form of dedicated threads), and singletons, which store references to these player instances, I would like to reload the JVM process (installing a new version of jar file for example), so that some of the classes and instances are not erased by GC, but hooked into the new process started.
Probable Situation:
I have a game version 1.0 running on the server. I need to add some new features in 1.2. But 100 players are fighting each other. I need to install version-1.2, players might feel some lag, but they should be able to continue with their fight even though the version of the jar has changed.
How can that be done?
Take a look at JRebel authors' nice summary on the topic:
http://zeroturnaround.com/blog/reloading_java_classes_401_hotswap_jrebel/#!/
You could look at things like OSGI and class-loading tricks and see if you can achieve what you need, however maybe there is a simpler approach which is a bit safer too.
You could have 2 servers running behind a simple load balancer and when you need to do such maintenance you start routing all new games to Server A only. When all the games on server B finish (and no new games have started because all are going to server A) you stop server B and do whatever maintenance you need to do together with any tests to verify everything is OK before you introduce it back into the 'cluster'.
You then do the same thing and route new games to server B, so that when all games on server A finish you cans switch that off too and do the maintenance.
You will need to develop this simple load balancer which just acts as a connection router but is aware of your game sessions. Maybe you also need to have some information synchronisation between each server instance (if there is any need of communicating lists of players online, chat messages etc., depends what features you have).
First thing to note is that your current game state might be inconsistent with new version. So not all modifications could be installed like this. You can use remote caching system with master/slave configuration. So that all session is stored on remote machine and while one server is stopped, second server can continue working with the same state. First you put down slave, update it and start. Then switch roles and put down second server for update.
We are running multiple load tests every day against one of our GAE apps. We use the following pattern:
Start a load test and let it run for a few hours.
Look at graphs.
Optionally deploy a new version of our app with performance improvements.
Go back to 1.
Each load test creates a couple hundred front end instances. We would like to terminate those between individual load tests even when we are not deploying a new version of our app.
Is there a way to terminate all dynamic instances? Right now we either have to deploy a new version or terminate all instances by hand.
We had a similar problem - I found that disabling the app in Application Settings and then re-enabling it terminated all 88 instances we had running, without any other adverse effects.
Maybe have them all periodically probe the datastore (or memcache) for a kill value?
My team built a Java application using the Hadoop libraries to transform a bunch of input files into useful output.
Given the current load a single multicore server will do fine for the coming year or so. We do not (yet) have the need to go for a multiserver Hadoop cluster, yet we chose to start this project "being prepared".
When I run this app on the command-line (or in eclipse or netbeans) I have not yet been able to convince it to use more that one map and/or reduce thread at a time.
Given the fact that the tool is very CPU intensive this "single threadedness" is my current bottleneck.
When running it in the netbeans profiler I do see that the app starts several threads for various purposes, but only a single map/reduce is running at the same moment.
The input data consists of several input files so Hadoop should at least be able to run 1 thread per input file at the same time for the map phase.
What do I do to at least have 2 or even 4 active threads running (which should be possible for most of the processing time of this application)?
I'm expecting this to be something very silly that I've overlooked.
I just found this: https://issues.apache.org/jira/browse/MAPREDUCE-1367
This implements the feature I was looking for in Hadoop 0.21
It introduces the flag mapreduce.local.map.tasks.maximum to control it.
For now I've also found the solution described here in this question.
I'm not sure if I'm correct, but when you are running tasks in local mode, you can't have multiple mappers/reducers.
Anyway, to set maximum number of running mappers and reducers use configuration options mapred.tasktracker.map.tasks.maximum and mapred.tasktracker.reduce.tasks.maximum by default those options are set to 2, so I might be right.
Finally, if you want to be prepared for multinode cluster go straight with running this in fully-distributed way, but have all servers (namenode, datanode, tasktracker, jobtracker, ...) run on a single machine
Just for clarification...
If hadoop runs in local mode you don't have parallel execution on a task level (except you're running >= hadoop 0.21 (MAPREDUCE-1367)). Though you can submit multiple jobs at once and these getting executed in parallel then.
All those
mapred.tasktracker.{map|reduce}.tasks.maximum
properties do only apply to the hadoop running in distributed mode!
HTH
Joahnnes
According to this thread on the hadoop.core-user email list, you'll want to change the mapred.tasktracker.tasks.maximum setting to the max number of tasks you would like your machine to handle (which would be the number of cores).
This (and other properties you may want to configure) is also documented in the main documentation on how to setup your cluster/daemons.
What you want to do is run Hadoop in "pseudo-distributed" mode. One machine, but, running task trackers and name nodes as if it were a real cluster. Then it will (potentially) run several workers.
Note that if your input is small Hadoop will decide it's not worth parallelizing. You may have to coax it by changing its default split size.
In my experience, "typical" Hadoop jobs are I/O bound, sometimes memory-bound, way before they are CPU-bound. You may find it impossible to fully utilize all the cores on one machine for this reason.