We Keep Coding

Lot of Netty Threads in Native.epollWait

I am initializing Netty Thread Pool like this:-
bossGroup = new EpollEventLoopGroup(1);
workerGroup = new EpollEventLoopGroup(1);
This creates more than 80 threads for a single application running on a VM allocated with 40 vCPUs.Is there a way that these threads can be reduced?
As it is not expected that the number of server and clients to increase?
Using Netty final 4.1.32

The code you showed will only create 2 EpollEventLoop instances and so only two Threads will end up in epollWait(...). You must execute this code multiple times or have a different place where you configure it differently to end up with more Threads. Another possibility is that you execute it multiple times but miss to call shutdownGracefully() on the EpollEventLoopGroup once you are done using it.

Problems maintaining trivial RPS numbers with jMeter for API testing

I am doing API load testing with JMeter. I have a Macbook Air (client) connected with ethernet to a machine being tested with the load (server).
I wanted to do a simple test. Hit the server with 5 requests per second (RPS). I create a concurrency thread group with 60 threads, a throughput shaping timer with 5 RPS for one minute, my HTTP request and hit the play button and run the test.
I expect to see my Hits per Second listener indicating a flat line of 5 hits per second, instead I see a variable rate, starting with 5 and then dropping to 2 and then later to 4... Sometimes there is more than the specified 5 RPS (e.g. 6 RPS) the point is that it's not a constant 5. It's too much of a variable rate - it's all over the place. And I don't get any errors.
My server, takes between 500ms to 3s to return an answer based on how much load is present - this is what I am testing. What I want to achieve with this test is to return as much as possible a response in 500ms time under load and I am not getting that. I have to start wondering if it's JMeter's fault in some way, but that's a topic for another day.
When I replace my HTTP sample request with a dummy sampler, I get the RPS I desire.
I thought I had a problem with JMeter resources, so I change heap size/memory to 1GB, use the -XX:+ DisableExplicitGC and -d64 flag and run in CLI mode. I never got any errors, not before setting the flags and not after. Also, I believe that 5 RPS is a small number so I don't expect resources to be a problem.
Something worth noting is that sometimes, the threads start executing towards the end of the test rather than at the start, I find this very odd behaviour.
What's next? Time to move to a new tool?

Is it possible (and wise) to execute other "spark-submit" inside a JavaRDD?

I'm trying to execute a Spark program with spark-submit (in particular GATK Spark tools, so the command is not spark-submit, but something similar): this program accept an only input, so I'm trying to write some Java code in order to accept more inputs.
In particular I'm trying to execute a spark-submit for each input, through the pipe function of JavaRDD:
JavaRDD<String> bashExec = ubams.map(ubam -> par1 + "|" + par2)
.pipe("/path/script.sh");
where par1 and par2 are parameters that will be passed to the script.sh, which will handle (splitting by "|" ) and use them to execute something similar to spark-submit.
Now, I don't expect to obtain speedup compared to the execution of a single input because I'm calling other Spark functions, but just to distribute the workload of more inputs on different nodes and have linear execution time to the number of inputs.
For example, the GATK Spark tool lasted about 108 minutes with an only input, with my code I would expect that with two similar inputs it would last something similar to about 216 minutes.
I noticed that that the code "works", or rather I obtain the usual output on my terminal. But in at least 15 hours, the task wasn't completed and it was still executing.
So I'm asking if this approach (executing spark-submit with the pipe function) is stupid or probably there are other errors?
I hope to be clear in explaining my issue.
P.S. I'm using a VM on Azure with 28GB of Memory and 4 execution threads.

Is it possible
Yes, it is technically possible. With a bit caution it is even possible to create a new SparkContext in the worker thread, but
Is it (...) wise
No. You should never do something like this. There is a good reason for Spark disallowing nested parallelization in the first place. Anything that happens inside a task is a black-box, therefore it cannot be accounted during DAG computation and resources allocation. In the worst case scenario job will just deadlock with the main job waiting for the tasks to finish, and tasks waiting for the main job to release required resource.
How to solve this. The problem is rather roughly outlined so it hard to give you a precise advice but you can:
Use driver local loop to submit multiple jobs sequentially from a single application.
Use threading and in-application scheduling to submit multiple jobs concurrently from a single application.
Use independent orchestration tool to submit multiple independent applications, each handling one set of parameters.

Selenium Fixed Thread Pool performance issues

I am currently trying to fire off a large number of selenium processes via
ExecutorService executor = Executors.newFixedThreadPool(10);
and am noticing performance issues.
The code that I'm deploying I first test and keep a copy of in a TestModule that runs the process once and uses the apache Stopwatch to monitor the time it takes to run the single thread.
When I run this code I see the following results:
Stopwatch time: 00:00:11.043
This is the time it takes from the initial driver.get(MY_WEBSITE_URL) all the way through inspecting elements and other tasking I wish to accomplish.
However, if I do the following in code, I get really slow results.
QueryAgent queryAgent = new QueryAgent();
queryAgent.startUp();
new Thread(queryAgent).start();
Inside of QueryAgent
private ExecutorService executor = Executors.newFixedThreadPool(10);
MyPojo pojo = MyPojoImpl.doStuff();
All of the code inside of "doStuff()" is the same code in my Test Module. If I am running 10 threads and each one should take no more than ~20 seconds to process. I would expect to see ~30 runs accomplished in a minute ~1800 in an hour.
Yet, looking at my logs I'm getting no more than 5 requests in a minute.
Is there a better way to run these requests in parallel?
EDIT1:
After looking at the comment below and doing a "top" it appears that once I hit about 4 instances of phantomjs - my CPUs are hitting 100%
But my memory usage is shy of 1GB. At this point it appears the bottle neck is the CPU(s). Any ideas?

How do I execute a Thread Group within another Thread Group?

I have a scenario in which I have to register a number of users and than run parallel with threads as the number of registered users and execute same set of actions by all the users in parallel. for this I have a jmx with few actions that should happened only once (in a setup thread with one thread count) and another thread group that runs with say 5 threads which is the number of the previously registered users, and I execute some operations using these users.
Now I want execute this whole scenario in parallel using 5 threads.
How do I come about doing this?
I used the include controller but thread groups are not executed as expected, I don't get 25 iterations for the actions that happen in a 5 threads group in the included jmx.

I'm not precisely sure what you're doing, and I know little of jmx, but here's a couple of ideas. One (or both) might be relevant.
The first one is that your threads might be sharing an instance field. If they have a common counter, for instance, you will do something 5 times rather than 25 times. Make sure your common variables (instance and class fields) are properly synchronized. Use local variables whenever possible. You must use them when their value applies to each thread rather than all threads.
The second is that you might be displaying results--or event stopping the program--before all threads have done their work. It's worst on single-core machines, but threads can and do run in any order imaginable, and in a few orders that are not. They can run one at a time, with the one started last running first. One can stop in the middle, and let all the other run to completion, then start up again. A bunch can run simultaneously (on different cores or swapping rapidly) while others do nothing.
I'd suggest putting in a bunch of logging/output statements (System.out.println is good enough) and seeing for yourself what's happening. It'll take you a while to make sense of your output, but once you do, you'll be able to start bringing things under control.