We Keep Coding

GAE task are failing in push

I am using GAE task queue to update bulk data in Datastore. Number of records are around 1-2M. To do this I scheduled a cron Job and a queue in this way
<queue>
<name>queueName</name>
<rate>20/s</rate>
<bucket-size>300</bucket-size>
<retry-parameters>
<task-retry-limit>1</task-retry-limit>
</retry-parameters>
<max-concurrent-requests>800</max-concurrent-requests>
</queue>
Each task is doing following task
Fetching 1500 record from datastore using a cursor.
If the next cursor exists create a new task and push in the queue.
Process 1500 fetched record, means updating all 1500 in datastore back.
the expected task to add should be around 667, but I can only see 40 tasks in logs.
In logs, I can see the 40 tasks are added in the queue in 40 sec. I m not getting any error in the logs.
Can anybody help me to understand what is happening? Why I m not able to add all the task.
Thanks

In your approach the task enqueueing appears to be very tightly coupled with the task request processing, in the sense that the request for one such task in the queue needs to be processed to enqueue the next task. So you need to take a look at your task processing rate limiting factors you may hit. The ones from your queue configuration are pretty generous, but there are others.
If you configured your app with threadsafe and if your app design takes advantage of it an instance of your app will be able to handle multiple requests concurrently, up to a maximum depending on its max-concurrent-requests config and its processing latency. Without the threadsafe config that maximum is 1.
Once an instance hits the max number of task requests it can process concurrently it won't start processing new tasks from the queue (so it won't execute step #1 - enqueueing a new task) until it completes processing at least one of the tasks already in progress. The task enqueueing rate per app instance is thus effectively limited - each running instance can contribute to the overall number of tasks in the queue only with a number equal to the max number of tasks it can process in parallel.
But your app is configured for automatic scaling, so once you manage to quickly "fill up" all your running instances, the scheduler will start new instances for it. As new instances are started they will be able to process more of the tasks in the queue and thus also enqueue new tasks, contributing with the above-mentioned amount to the total number of tasks in the queue.
But this growth in the number of enqueued tasks can be much slower than while instances didn't hit their max processing rate - it takes some time to measure how new instances helps with traffic to determine if more instances are needed or not. The overall growth in the number of tasks in the queue will have a "staircase" profile, with the height of a step being the max number of concurrent requests an instance can handle and the number of steps being the number of new instances started +1.
Since you aren't seeing any actual task enqueuing errors I can only suspect that you're somehow hitting a rate limit in processing your enqueued tasks or somehow that processing completely stops. There can be many reasons for it, including, for example:
hitting your app's daily budget (most likely due to the number of instance-hours)
hitting automatic scaling limits
You'd have to investigate your app from this perspective to pinpoint the culprit.
Side note: I assume this is on GAE, not on the development server (which doesn't respect the task queue configs and most likely can't get even close to GAE's parallel processing capability).

Using multiple threadpools and connection pool

I am currently using 5 threadpools and I want to find optimal sizing for these pools. This is some kind of prior analysis. Pools are divided by usage: for handling commands (cmdPool), handling inventory transactions (invPool), pool for database transactions (dbPool), also pool for common things that simply need to run async like I/O (fastPool) and for scheduled tasks (timerPool). I do not have any statistical data that could be used for solving problem yet.
For database query I am using HikariCP with default values. I will try to change count of maximum connections and minimum idle connections later to find optimal performance. But for now, when using Hikari pool it will be always called from one of the pools to not affect main thread. Usual database query is called under dbPool but only when code block is not part of already runnable submited into one of the thread pools.
Actual setup looks it just works right in application. So my questions are:
1.) How will impact performance and resources when I decide to stop using cachedThreadPool and use pool with some minimum idle threads like timerPool or I should stick with cached ?
2.) Is right solution to set a maximum pool size to prevent spikes when like 100 clients will join in small period of time and let them keep wait for some short time while other task will complete.
3.) Is there any better solution how to manage many kinds of tasks ?
cmdPool = Executors.newFixedThreadPool(3);
invPool = Executors.newFixedThreadPool(2);
dbPool = Executors.newCachedThreadPool();
fastPool = Executors.newCachedThreadPool();
timerPool = new ScheduledThreadPoolExecutor(5);
timerPool.allowCoreThreadTimeOut(true);
timerPool.setKeepAliveTime(3, TimeUnit.MINUTES);
So first of all, every action depends on how many clients are connected, lets assume values like 5-25 clients. Pools should be designed to maintain even extremes like 100 clients and not make too many threads in small time period.
Expected uses may vary and are not same every second even may happen no task will come to run at all. Expected usage of cmdPool is like 3-8 uses per second (lightweight tasks). For invPool is usage nearly same like for cmdPool 2-6 uses per second (also lightweight tasks). As for dbPool this is more unpredictable than all others, but still expected usage is from 5-20 uses per second (lightweight and mediumweight tasks) also depends on how busy is network. Timer and fast pools are designed to take any kind of task and just do it, there is expected use of 20-50 uses per second.
I appreciate any suggestions, thank you.

The best solution is to adapt your application to the expected traffic.
You can do that in many manners:
Design it with a microservice architecture leaving the orchestrator to handle peak of traffic
Design the application that reads the parameters of the size of thread pools on the fly (from a database from a file, from a configuration server...), so you can change the values when needed
If you need only to tune your application but you don't need to change the values on the fly put your configurations in a file (or database). Check different configurations to find the most adapted to your needs
What is important is to move away a code similar to this one:
cmdPool = Executors.newFixedThreadPool(3);
and replace it with a code similar to this one
#Value("${cmdPoolSize}")
private int cmdPoolSize;
...
cmdPool = Executors.newFixedThreadPool(cmdPoolSize);
where the size of the pool is not taken from the code, but from an external configuration.
A better way is also to define the kind of pool with parameters:
#Value("${cmdPoolType}")
private String cmtPoolType;
#Value("${cmdPoolSize}")
private int cmdPoolSize;
...
if (cmdPoolType.equals("cached")) {
cmdPool = Executors.newCachedThreadPool();
} else if (cmdPoolType.equals("fixed")) {
cmdPool = Executors.newFixedThreadPool(cmdPoolSize);
}
Where you choose the reasonable kind of available pools.
In this last case you can also use a spring configuration file and change it before starting the application.

Prevent from slow job taking over a thread pool

I have a system where currently every job has it's own Runnable class and I pre defined a fixed number of threads for every job.
My understanding is that it is a wrong practice, because:
You have to tailor the number of threads with respect to the machine running the process.
Each threads can only take one type of job.
Would you agree on that? (current solution is wrong)
So, I'd like to use something like Java's ThreadPool instead. I was conflicted with an argument claiming that by doing so, slow jobs will take over most of the thread pool, leaving no place to the other jobs. Whereas, with the current solution, a fixed number of threads were assigned to the slow worker and it won't hurt the others.
(Notice that you can't know a-priori if a job will be "slow")
How can a system be both adaptive in the number of threads it uses, but at the same time not be bounded to the most slow job?

You could try getting the time it takes for the job to complete (With a hand-made Timer class of sorts. Then you normalize this value by dividing this time by the maximum time any given thread has taken. Finally, you multiply this number by a fixed number which varies depending on how many threads you want running per job per second. This will be the requested amount of threads this process should be using. You can adjust that according.
Edit: You can set minimum and maximum values that regulate how many threads a job is entitled to. You could alternatively request threads from a very spacious job when another thread enters the system.
Hope that helps!

It's more of a business problem. Let's say I am a telecom operator. I bar my subscribers from making outgoing calls when they don't clear their dues. When they make payment I clear a flag and in a second the subscriber can make calls. But a lot of other activities go on in my system like usage processing, billing, bill formatting etc.
Now let's assume I have a system wide common pool of threads and I started the billing of 50K subscribers. All my threads are now processing the relatively long running billing jobs and a huge queue is building up.
A poor customer now makes a payment and wants to make an urgent call. But I have no thread left in my pool to clear the flag. The customer had to wait for an hour before he can make the call. That's SLA breach.
What I should have done is create separate thread pools. If the call unblocking jobs are not very frequent and short, I can create a separate pool for it with core size 5 maybe. For billing jobs I'd rather create a pool with core size 25 and max-size 30.
So, my system limits won't anyway exceed because I know in even the worst situation I won't have more than 30 threads.
This will also make it easy to debug. If I have a different thread name pattern for each pool amd my system has some issues. I can easily take a thread dump and understand if the billing or the payment stuff is the culprit.
So, I think the existing design is based on some business use case which you need to thoroughly understand before proposing a solution.

What is the best bucket size for a task queue filled with many deferred tasks in Google App Engine?

My Google App Engine application is adding a large number of deferred tasks to a task queue. The tasks are scheduled to run every x seconds. If I understand the bucket-size property b correctly, a high value would prevent the deferred tasks to run until b tasks have been added. However, there is a close-to-realtime requirement that the tasks run as scheduled. I do not want that the tasks are blocked until the bucket-size is reached. Instead they should run as close to their scheduled time as possible.
To support this use case, should I use a bucket-size of 1 and a rate of 500 (which is the current maximum rate)? Which other approaches exist to support this? Thanks!

The bucket size does not prevent tasks from running individually. It plays a different role.
Suppose you have an empty queue with rate of 500 tasks per second, and several hours where no tasks are added or started. Then suddenly a large number of tasks are added at once. How many of these tasks would you like started immediately? Set this number as your bucket size. For example, with a bucket size of 1000, 1000 tasks will be started immediately (then 500 per second going forward).
How does this work? The bucket is topped up by 500 tokens every second (the queue's rate), up to the maximum being the bucket size. When there are tasks are available to start, they will only be started while the bucket is not empty, and one token will be removed from the bucket as each task is started.

You should NOT use taskqueues (TQ) for deferred tasks that are important to run close-to-realtime using the assumption that bucket/rate setting will assure high throughput. There have been several discussion threads in Google groups about infrequent delays with task start times that are minutes or more in length. Bucket size and rates will not have an affect on this -- your TQ tasks will simply sit there while your high-throughput TQ is idle. To date I have not ever seen any explanation from Google as to why this occurs. Again, if you utilize TQs for close-to-real-time tasks you MUST handle as an exception the infrequent times when your tasks will delay for minutes prior to starting. (I in fact do this, and have not yet been negatively affected, but you have to have code in place to handle a result = delayed task). My great hope is that with the new server/application testing underway, Google will find an easy way to kill this incredibly big issue with TQs (fingers crossed).

Threads/backend in appengine java

I want to run some kind of Thread continuously in app engine. What the thread does is
checks a hashmap and updates entries as per some business continuously.
My hashmap is a public memeber variable of class X. And X is a singleton class.
Now I know that appengine do not support Thread and it has somethinking called backend.
Now my question is: If I run backend continiously for 24*7 will I be charged?
There is no heavy processing in backend. It just updates a hashmap based on some condition.
Can I apply some trick so that am not charged? My webapp is not for commercial use and is for fun.

Yes, backends are billed per hour. It does not matter how much they are used: https://developers.google.com/appengine/docs/billing#Billable_Resource_Unit_Costs
Do you need this calculation to happen immediatelly? You could run a cron job, say ever 5 min and perform the task.

Or you can too enqueue a 10 minutes task and re-enqueue when is near to arrive to its 10 minutes limit time. For that you can use the task parameters to pass the state of the process to the next task or also you can use datastore.