We Keep Coding

A limited thread & creating more on demand

I have a program where the user fetches some data from the stored files. That can be 1, 2, 3, 10, 50, 100, 1000 etc. files. I want that files to be fetched via separated thread. But when the thread reaches e.g. 50th file then a new thread is created. And if it's not enough then 3rd, 4th, 5th, etc. threads are created until all the data from the files would be retrieved.
So, the question is: how to make a limit of 50 files to check for 1 thread and how to create new ones for the next files processing? Maybe some ExecutorService, synchronizer, or something which I have some difficulties to understand yet...

Jmeter threadgroup sharing counter between threads

I want to create a Thread Group in jmeter to create users and their devices with many (for example 5000) devices.
I have no problem to create this test and I want to have the name of the users and device in a correlative order.
1. user1--device1
2. user2--device2
3. user3--device3
I already created the test but to create 5000 users and devices in the same Thread Group I need to run 5000 iterations with 3 requests each, using 1 thread, because otherwise I have the same user name repeated by multiple threads (for example with 3 threads)
1. user1--device1
2. user2--device2
3. user3--device3
4. user1--device1
5. user2--device2
6. user3--device3
7. user1--device1
8. user2--device2
9. user3--device3
My question is: is there any way to share the loop counter between threads in order to create 5000 users/devices with more than one thread (for example 20 threads).
This will help me a lot because instead of waiting for 20 minutes, it will be a minute to create the users.
Many thanks!
http://pastebin.com/S1izFC9r
Added explanation I want for example maximun 9 devices (counter_max) but I want to run it with 3 threads. I want the result be like that
thread1--user1-device1
thread2--user2--device2
thread3--user3--device3
thread1--user4--device4
thread2--user5--device5
thread3--user6--device6
thread1--user7--device7
thread2--user8--device8
thread3--user9--device9

I think you can implement it using __counter() function in "global" mode, like: ${__counter(FALSE,)}
__counter() function returns incremented value each time it's being called
in the "global" mode counter value is shared across threads
So you should be able to use as many threads as required and each thread will use the next counter value to create your users and devices with multiple threads.
See How to Use a Counter in a JMeter Test article for comprehensive information on using "counter" config element and function.

process 1 table in hibernate with multithreads

I am working in a application using spring+hibernate.
I have a situation where I have to fetch a set of records from 1 table(with status flag 0), process each of them (will generate data for other tables), then set the status flag to 1.
problem is all of this is being done by 1 thread, and is very slow. I want to achieve lets say i make 10 threads. each of them will take one records, process, save, done. Then the process will speed up 10 times.
Pls look at picture. Any advise on how to do this?
---- current situation ----
http://imgur.com/DSgIy
------ desired situation ------
http://imgur.com/Myz5J

Wrap the process method in a Runnable and execute it using a TaskExecutor.
You can play with TaskExecutor parameters like thread pool size using the spring task namespace or annotations, at you choice.
see http://static.springsource.org/spring/docs/current/spring-framework-reference/html/scheduling.html

using different threads to do processing in web application

I have a Java EE web application. Now when a particular request comes (say /xyz url patter) I want to do complex procesing as follows
Each of the following 3 steps are very complex and takes time.
Get data from one table from DB.Table has huge data and querying takes time.
Make a web service call to some other webserive A and get its data.
Make another web service call to some otheer webserice B and get its data .
Do some processing by using output of 1, 2, 3
1, 2, and 3 are independent of each other so can be called in parallel.
Now the questions are:
Can I do operations 1, 2, and 3 in three separate threads?
Is it advisable to create 3 threads for each request?
Should I use thread pooling?

To address your first question I go through the 4 steps:
Yes, if the database driver you are using allows concurrent access, respectively is safe to use from different threads.
A web service is normally designed to deal with different requests at the same time so this should work as well, the question here is how many threads you want to use (and how long it takes to process one request) and whether the web service will guard itself against too many requests at once.
The same applies here.
Yes, but you have to do synchronization here, as in: wait until all threads have received their results. You can realize this with a java.util.concurrent.CyclicBarrier
Second question
That depends on your data and especially how fast the web services will answer, you should try it out.
Third question Definitively, that's what they are for. This will also help you to structure your application.

1) Can i do operations 1 ,2 and 3 in three separate threads?
Yes, you can.
2) Is it advisable to create 3 threads for each request?
As long as these things don't depend on each other, and as long as you're not depending on getting these in the same transaction, then it seems like it should be ok. You will have to handle the case where one or more threads don't succeed, of course. You'll need a separate watchdog thread to cancel the threads if they take too long or if one comes back with a failure.
3) Should I use thread pooling?
Regardless of what else you do, whenever you use threads you should use a pool. That way if there's a problem where threads don't complete or go into some bad state or otherwise become unavailable, you protect your application from running out of threads.

How do I execute a Thread Group within another Thread Group?

I have a scenario in which I have to register a number of users and than run parallel with threads as the number of registered users and execute same set of actions by all the users in parallel. for this I have a jmx with few actions that should happened only once (in a setup thread with one thread count) and another thread group that runs with say 5 threads which is the number of the previously registered users, and I execute some operations using these users.
Now I want execute this whole scenario in parallel using 5 threads.
How do I come about doing this?
I used the include controller but thread groups are not executed as expected, I don't get 25 iterations for the actions that happen in a 5 threads group in the included jmx.

I'm not precisely sure what you're doing, and I know little of jmx, but here's a couple of ideas. One (or both) might be relevant.
The first one is that your threads might be sharing an instance field. If they have a common counter, for instance, you will do something 5 times rather than 25 times. Make sure your common variables (instance and class fields) are properly synchronized. Use local variables whenever possible. You must use them when their value applies to each thread rather than all threads.
The second is that you might be displaying results--or event stopping the program--before all threads have done their work. It's worst on single-core machines, but threads can and do run in any order imaginable, and in a few orders that are not. They can run one at a time, with the one started last running first. One can stop in the middle, and let all the other run to completion, then start up again. A bunch can run simultaneously (on different cores or swapping rapidly) while others do nothing.
I'd suggest putting in a bunch of logging/output statements (System.out.println is good enough) and seeing for yourself what's happening. It'll take you a while to make sense of your output, but once you do, you'll be able to start bringing things under control.