I am creating a daemon that will run certain scheduled tasks for logging but I am worried about bottle necking certain points.
Effectively I have some logging tasks that I want to execute every 15 minutes and some I only want to execute every 30 minutes and so on up to tasks that only need running once a month. Basically I have a list of checks to make at each time interval. These are put into a queue and processed by a thread pool.
At the moment I see the tasks as running something like this
15
15, 30
15, 30, 60
15, 30, 60, 120
15, 30, 60, 120, 240...
This means that if the daemon starts at 00:00 hours then by 04:00 hours there will be five processes running simultaneously and this is not the end of it. At present this has led the next task scheduled for 15 minutes to run slowly and have access to a restricted amount of bandwidth.
It is not neccessary for the tasks to run on the hour however. So if the 15 minute task runs on the hour the 30 minute may start at 5 minutes past the hour so as to minimise overlap. It would even be possible to split the two 30 minutes tasks (e.g. 00:00 and 00:30) across the four 15 minute process to reduce being hit by an 'all at once' type problem but this really gets my head swimming.
Are there any well known methodologies for managing this type of issue?
You should definitely have a look at Quartz, especially the cron style triggers.
Cheers,
Well, for long run (as I see from your question), you will have to look for something like quartz.
Apart from this, I have couple of more concerns and suggestions here:
Use [ScheduledExecutorService][2] for managing these threads. Even with ScheduledExecutorService, looks like you want to run them at separate interval irrespective of their execution time. SchedulewithFixedRate rather than ScheduleWithFixedDelay.
Even if you implement anything like this, your logic will fail if your threads start to run on multiple hosts. 2 hosts running hourly threads will effectively run every 30 minutes.
I would prefer to have a centralized management in terms of Database to keep track of last run and all. This along with ExecutorService would be scalable and accurate.
Suppose you have 1000 schedules, create 1000 rows in the DB.
Columns will somewhat be like this
id, P_Key
ScheduleName, AnyIdentifier for the daemon to run or task to do.
lastRunTime, lastTime it was run.
granularity, 15 mins, 30 mins etc.
You can keep CreationTime and ModificationTime as best practices.
Related
I am testing a certain "functionality" that happens after log in.
The test case is 500 users exercising that functionality within 5 minutes.
I can add a synchronising timer after the log in, to ensure all 500 threads have logged in but then it will do all 500 "functionality" tasks at once, rather than 5 minutes, which will crash the app (it thinks there's a DDoS attack and shuts down).
Right now, I am handling this by giving some think time after login, to slow down login to a stable figure that I can predict and then start "functionality" at each thread's turn, as scheduled by: the main scheduler + the the log in response time + the think time...
But that's a bit fuzzy.
Is there a way to "ramp up" tasks once already running?
I can think in two options.
The first one is two use random times. You would use the range from 0 seconds to 300 - 1 that is [0-300) or using millis [0-300000). Then sleep the thread basesd on this ramdon time.
This approach can be a little more realist, because for instance, in a specific second of the given interval you don't have any threads starting and in other particular second you have 2-3. This still should be well balanced in general, since you won't make all petitions at start.
The second one is to start the threads uniformly. During your configuration time (login and before firing the threads) you can use something like an AtomicInteger, initializing it with new AtomicInteger(0) and calling getAndIncrement() to assign the possition of the thread, in the range [0-500) and then when you fire the threads sleep 300.0 * id / 500.0 milliseconds to execute the task/petition.
By default JMeter executes requests as fast as it can, you can "throttle" the execution to desired throughput (request per minute) rate using Constant Throughput Timer.
Example Test Plan would look like:
Thread Group
Login
Synchronizing Timer
Functionality
Constant Throughput Timer
Constant Throughput Timer follows JMeter Scoping Rules so you can apply it either to single sampler or to a group of samplers.
I have created two Handlers, which each have Runnable Tasks which run in a loop through:
// do it again 1 hundeth of a sec later
QuestionHandler.postDelayed(QuestionHandlerTask, 10);
And through:
// and do it again 1 sec later
GameHandler.postDelayed(GameHandlerTask, 1000);
Both the tasks start at 10 and count downwards and their results are displayed in labels. The reason i have used 2 handler's is because i want 1 timer to count down by milleseconds and 1 to count down by seconds.
However, the GameHandlerTask that runs once per second runs significantly faster than the other. By the time it reaches 1 the QuestionHandlerTask is still at 4 seconds:
Is this a commonly known problem or have i done something seriously wrong? Is there some way i can make it so my timers run at the same speed? Is there other methods that i can use that would be better in this circumstance (where i want precise timing)?
This is because the screen cannot refresh 100 times per second, which is what is happening with the 10 millisecond timer. So the screen tries to show every single value that you told it to show, but can only do so at its maximum refresh rate.
It is not the timer that is slow, it is the screen that can't keep up with the timer.
To solve this, try to calculate how much does the number need to go down by in 40 milliseconds, and you start a timer of 40ms. This will update the screen 25 times a second, which the screen should be able to handle.
I previously asked Why are the durations of hadoop mapper tasks always a multiple of 3 seconds? and by setting the value of Task.PROGRESS_INTERVAL to a small number (>1000) I was able to get fast running tasks to finish in 3 seconds rather than the 6 they were taking. These tasks are literally processing no data and I'd like to get them to finish more quickly if possible.
What are the parameters that are causing the tasks to take 3 seconds
We are using quartz scheduler in a clustered environment (Two nodes in a cluster, pointing to a single Oracle database). Currently we have two jobs which runs pretty much every one hour.
We have a separate database schema for the quartz jobs. What we've noticed is that quartz checks the database every 15 seconds (default value for clusterCheckinInterval).
We don't like this and like to make it less frequest. What we have in mind is to give is a 1 minute frequency, but most of the example configurations has given clusterCheckinInterval as 20000.
Can some body please recommend a suitable value for the clusterCheckinInterval?
From the Quartz documentation:
org.quartz.jobStore.clusterCheckinInterval
Set the frequency (in milliseconds) at which this instance "checks-in"* with the other instances of the cluster. Affects the quickness of detecting failed instances.
In a Quartz clusters the clusterCheckinInterval tells how responsive your cluster is to the failover (considering Quartz jobs). The smaller the interval is, the more quickly your application can respond. Actually this value is used by cluster nodes to check if there are recoverable jobs running on a broken node. If yes, Quartz tries to re-run them.
In general the default value is good enough, but you have to take into consideration the frequency of jobs and the effect that a missed job run can cause.
If you have a number of jobs that must run on every second, then you have to set the interval to 1000 (in milliseconds).
If you have jobs that run on every second, but it is not critical to run them all the time, then 5-15 sec is good enough (depending the fault tolerance of the system).
If you have hour-long running jobs which run a few times a day, you can raise the interval even to 60 sec.
My opinion is that I would not consider 20-30 database request per minute as a "load", so I would set it to 2 or 3 seconds (2000 or 3000 in millisec) .
Been experimenting with Jmeter, and I'd like to know the best way to accomplish:
20 users logging onto an application, over 20 minutes, and performing some actions for another 20 minutes, before logging off over a period of 20 minutes. I.e. have 200 users logging on, and then once ALL of them are logged on, begin 20 minute timer. Once 20 minutes are up, start to log the ones who logged on earliest off.
I realise this MAY or MAY NOT BE a realistic testing scenario, but i'd like to see if it's possible.
At the moment I have a test plan whereby a user logs on, performs some actions, and then logs off. I can't see how I can ramp up and ramp down.
There's an option in Test Plan "Run thread groups consecutively". Set it to checked.
Then add 3 thread groups to your test plan. I'd suggest using Thread Group for first (20 threads, loop count 1, ramp up time 1), Ultimate Thread Group (20 threads starting immediately and holding load for 20min) for second and Thread Group again for third (20 threads, loop count 1, ramp up time 1).
Place appropriate samplers inside each TG - first just logs in, second does actions, third logs off.
That's it. If you have any troubles - let me know.
You'll need several thread groups in JMeter starting off and running at different intervals, in that way you could ensure that the users who start first will end first.
Also see a related question on this.
You can have no of users=20, ramp up time=1200 sec (1 per min), difference of time between test start and test end time=20 min to achieve that.
I think I had a similar problem in the past
Here Want I've did:
First set your thread group to have 20 thread with a rampup period of 60 seconds
After the login put a "test action" (in the sampler menu)
target = current thread, with the action pause and 20 minutes (1 200 000 ms) or more if you want to be safe.
After this test action, put all your navigating request.
Once your navigation is done, put another "test action" with the same setting has the previous one
(target = current thread, with the action pause and 20 minutes (1 200 000 ms))
put the logouf request after the sampler.
This should cover you case.
Take note that the sampler just pause your thread so the first thread that start should be the first thread that end.
If you want to scale it to 200 you just need to change your thread group rampup period to 6 or 5 seconds.
hope it's help.