I have the following piece of code:
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public class CheckResults {
private int sheepCount = 0;
private void incrementAndReport() {
synchronized(this) {
System.out.print((++sheepCount)+" ");
}
}
public static void main(String[] args) {
ExecutorService service = null;
try {
service = Executors.newFixedThreadPool(20);
var manager = new CheckResults();
for(int i = 0; i < 10; i++)
service.submit(() -> manager.incrementAndReport());
} finally {
if(service != null) service.shutdown();
}
}
}
This simply prints
1 2 3 4 5 6 7 8 9 10
Now before I go to visualVM or other tool, the program gets terminated.
So what is the best way to get the complete dump of the threads which would be Waiting/Running etc..It would help me understanding/validating my multithreading models.
This may be a trivial question but I could not find a good write up or help.
Thanks
Your question is not completely clear, but I gather you want multiple thread dumps, taken as your app proceeds.
Write a thread dump routine. Perhaps like this:
// `threadDump` method taken from: https://www.baeldung.com/java-thread-dump
private static String threadDump ( boolean lockedMonitors , boolean lockedSynchronizers )
{
StringBuffer threadDump = new StringBuffer( System.lineSeparator() );
ThreadMXBean threadMXBean = ManagementFactory.getThreadMXBean();
for ( ThreadInfo threadInfo : threadMXBean.dumpAllThreads( lockedMonitors , lockedSynchronizers ) )
{
String message = "Thread: " + threadInfo.getThreadId() + " | " + threadInfo.getThreadName();
threadDump.append( message ).append( System.lineSeparator() );
// threadDump.append( threadInfo.toString() );
}
return threadDump.toString();
}
Call that routine within your code at appropriate points. I’m guessing you want to capture the thread dump after each task is submitted.
ExecutorService service = null;
try {
service = Executors.newFixedThreadPool(20);
var manager = new CheckResults();
for(int i = 0; i < 10; i++)
{
service.submit(() -> manager.incrementAndReport());
String threadReport = Whatever.threadDump() ; // Do what you want with the string, log it, dump it to console, etc.
}
} finally {
if(service != null) service.shutdown();
}
Or perhaps you want to call the thread dump routine within the code of your task submitted to the executor service.
ExecutorService service = null;
try {
service = Executors.newFixedThreadPool(20);
var manager = new CheckResults();
for(int i = 0; i < 10; i++)
{
service.submit
(
() ->
{
manager.incrementAndReport();
String threadReport = Whatever.threadDump() ; // Do what you want with the string, log it, dump it to console, etc.
}
);
}
} finally {
if(service != null) service.shutdown();
}
Related
While testing concurrency, I found something unexpected.
Concurrency was controlled using concurrentHashMap and AtomicLong.
public class HumanRepository {
private final static Map<Long, Human> STORE = new ConcurrentHashMap<>();
private AtomicLong sequence = new AtomicLong();
public void save(Human human) {
STORE.put(sequence.incrementAndGet(), human);
}
public int size() {
return STORE.size();
}
public Long getSeq() {
return sequence.get();
}
}
I tested saving in multiple threads.
#Test
void name() throws NoSuchMethodException, InterruptedException {
final int threads = 3_500;
final ExecutorService es = Executors.newFixedThreadPool(threads);
final CountDownLatch count = new CountDownLatch(threads);
final HumanRepository repository = new HumanRepository();
for (int i = 0; i < threads; i++) {
try {
es.execute(() -> repository.save(new Human("aa")));
} finally {
count.countDown();
}
}
count.await();
System.out.println("seq = " + repository.getSeq());
System.out.println("size = " + repository.size());
}
I tested it with 3500 threads simultaneously. The result I expected is 3500 for both seq and size.
But sometimes I get seq=3499, size=3500.
That's weird. It is strange that seq does not come out as 3500, and even though the size is 3500, it does not make sense that seq is 3499.
I don't know why the data number and seq in the map are not the same and 3500 is not coming out.
** If you do Thread.sleep(400L); after count.await();, surprisingly, the value of seq is 3500
You are not actually waiting for all tasks to complete. Which means that if you get the 3500/3500 output, it's by chance.
Specifically, you decrease the countdown latch on the main thread after scheduling the job, instead of inside of the job, once it's done. That means your countdownlatch is basically just another glorified loop variable that doesn't do any inter-thread communication. Try something like this instead:
for (int i = 0; i < threads; i++) {
es.execute(() -> {
repository.save(new Human("aa"));
count.countDown();
});
}
You are calling count.countDown() outside the thread executing the HumanRepository.save(). So its possible that the main thread is not synchronized for the completion of the threads.
So you may see the results of repository.getSeq() while one thread is running. Can you try with the following code?
final int threads = 3_500;
final ExecutorService es = Executors.newFixedThreadPool(threads);
final CountDownLatch count = new CountDownLatch(threads);
final HumanRepository repository = new HumanRepository();
for (int i = 0; i < threads; i++) {
try {
es.execute(() -> {
repository.save(new Human("aa"));
count.countDown();
});
} finally {
}
}
count.await();
System.out.println("seq = " + repository.getSeq());
System.out.println("size = " + repository.size());
Here's what I'm trying to do. I am recording data from different sensors in a while loop until the user stops the recording. I want to record as much data as possible per second. The sensors require different time to return a value, between 200ms and 3 seconds. Therefore, sequentially calling the sensors successively is not an option.
Sequentially calling the sensors looks like this:
List<DataRow> dataRows= new ArrayList<DataRow>();
while (recording) {
DataRow dataRow = new DataRow();
dataRow.setDataA(sensorA.readData());
dataRow.setDataB(sensorB.readData());
dataRow.setDataC(sensorC.readData());
dataRows.add(dataRow);
}
Depending on the sensor, reading the data looks (much simplified) like that
public class SensorA {
public SensorAData readData(){
sensorA.startSensing();
try {
TimeUnit.MILLISECONDS.sleep(750);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
return sensorA.readAndConvertByteStream();
}
}
To utilize Multithreading can SensorA implement Callable and receive Future objects in the loop? Or should the while loop be placed within a run() method implementing the interface Runnable?
Basically, can Java (or a thread) write to the correct dataRow object even if the loop is already at least one iteration further? If not, how can one solve this problem?
If I understand your needs correctly, this may be solution you want:
In each iteration n sensors are read by n concurrent threads,
If all threads has sensors data collected, new result row is added to list
Working code:
public class TestX {
private final ExecutorService pool = Executors.newFixedThreadPool(3);
private final int N = 10;
// all sensors are read sequentially and put in one row
public void testSequential() {
int total = 0;
long t = System.currentTimeMillis();
for (int i = 0; i < N; i++) {
System.out.println("starting iteration " + i);
int v1 = getSensorA(); // run in main thread
int v2 = getSensorB(); // run in main thread
int v3 = getSensorC(); // run in main thread
// collection.add( record(v1, v2, v3)
total += v1 + v2 + v3;
}
System.out.println("total = " + total + " time = " + (System.currentTimeMillis() - t) + " ms");
}
// all sensors are read concurrently and then put in one row
public void testParallel() throws ExecutionException, InterruptedException {
int total = 0;
long t = System.currentTimeMillis();
final SensorCallable s1 = new SensorCallable(1);
final SensorCallable s2 = new SensorCallable(3);
final SensorCallable s3 = new SensorCallable(3);
for (int i = 0; i < N; i++) {
System.out.println("starting iteration " + i);
Future<Integer> future1 = pool.submit(s1); // run in thread 1
Future<Integer> future2 = pool.submit(s2); // run in thread 2
Future<Integer> future3 = pool.submit(s3); // run in thread 3
int v1 = future1.get();
int v2 = future2.get();
int v3 = future3.get();
// collection.add( record(v1, v2, v3)
total += v1 + v2 + v3;
}
System.out.println("total = " + total + " time = " + (System.currentTimeMillis() - t) + " ms");
}
private class SensorCallable implements Callable<Integer> {
private final int sensorId;
private SensorCallable(int sensorId) {
this.sensorId = sensorId;
}
#Override
public Integer call() throws Exception {
switch (sensorId) {
case 1: return getSensorA();
case 2: return getSensorB();
case 3: return getSensorC();
default:
throw new IllegalArgumentException("Unknown sensor id: " + sensorId);
}
}
}
private int getSensorA() {
sleep(700);
return 1;
}
private int getSensorB() {
sleep(500);
return 2;
}
private int getSensorC() {
sleep(900);
return 2;
}
private void sleep(long ms) {
try {
Thread.sleep(ms);
} catch (InterruptedException e) {
// ignore
}
}
public static void main(String[] args) throws ExecutionException, InterruptedException {
new TestX().testSequential();
new TestX().testParallel();
}
}
and output:
starting iteration 0
starting iteration 1
starting iteration 2
starting iteration 3
starting iteration 4
starting iteration 5
starting iteration 6
starting iteration 7
starting iteration 8
starting iteration 9
total = 50 time = 21014 ms
starting iteration 0
starting iteration 1
starting iteration 2
starting iteration 3
starting iteration 4
starting iteration 5
starting iteration 6
starting iteration 7
starting iteration 8
starting iteration 9
total = 50 time = 9009 ms
-- EDIT --
in java 8 you can use method reference to get rid of Callable classes and just write:
Future<Integer> future1 = pool.submit( this::getSensorA() );
Future<Integer> future2 = pool.submit( this::getSensorB() );
Future<Integer> future3 = pool.submit( this::getSensorC() );
I implement a recommendation algorithm in Java program.
However, I have serious problems. The dataset is too large and it's computation is too slow. So, I need to do parallel programming in Java.
For example,
for (int i=0; i < 10000000 ; i++) { ~~~ }
I want to split this sentences such as
process 1: for (int i=0; i < 10000 ; i++)
process 2: for (int i=10001; i < 20000 ; i++)
process 3: for (int i=20001; i < 30000 ; i++)
...
I know similar methods in Python. How to do parallel programming in Java?
Hope this will help you.
public class MyRunnable implements Runnable {
private final long countUntil;
MyRunnable(long countUntil) {
this.countUntil = countUntil;
}
#Override
public void run() {
long sum = 0;
for (long i = 1; i < countUntil; i++) {
sum += i;
}
System.out.println(sum);
}
}
public class Main {
public static void main(String[] args) {
// We will store the threads so that we can check if they are done
List<Thread> threads = new ArrayList<Thread>();
// We will create 500 threads
for (int i = 0; i < 500; i++) {
Runnable task = new MyRunnable(10000000L + i);
Thread worker = new Thread(task);
// We can set the name of the thread
worker.setName(String.valueOf(i));
// Start the thread, never call method run() direct
worker.start();
// Remember the thread for later usage
threads.add(worker);
}
int running = 0;
do {
running = 0;
for (Thread thread : threads) {
if (thread.isAlive()) {
running++;
}
}
System.out.println("We have " + running + " running threads. ");
} while (running > 0);
}
}
i got it from here
I need to execute tens of thousands of IO bound operations per second in one application that each have a 3 second delay. In node.js this can be trivially accomplished as follows:
var events = require('events');
var eventEmitter = new events.EventEmitter();
var count = 0;
var connectHandler = function connected() {
setTimeout(function() {
count++;
console.log('task complete ' + count);
}, 3000);
}
for (var i = 0; i < 30000; i++) {
eventEmitter.on('mockedIOTask', connectHandler);
}
eventEmitter.emit('mockedIOTask');
How can this be accomplished in a Java application? I have tried the following but it only executes 3 tasks at a time (for obvious reasons):
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.ForkJoinPool;
public class Main {
public static void main(String[] args) {
//ExecutorService threadPool = Executors.newCachedThreadPool();
for (int i = 0; i < 30000; i++) {
final int count = i;
ForkJoinPool.commonPool().submit(() -> {
try {
Thread.sleep(3000);
System.out.println("task complete" + count);
} catch (InterruptedException e) {
}
});
}
// Keep the program alive
try {
Thread.sleep(50000);
} catch (InterruptedException e) {
}
}
}
You can execute 30K tasks after 3 seconds using a ScheduledExecutorService
ScheduledExecutorService ses = Executors.newSingleThreadScheduledExecutor();
List<Future<?>> tasks = new ArrayList<>();
for (int i = 0; i < 30000; i++) {
final int count = i;
tasks.add(ses.submit(() -> System.out.println("task complete" + count)));
}
for (Future f : tasks) f.get();
ses.shutdown();
If you want to add IO bound operations, I would start with blocking ones, before migrating to using a library like netty to handle the non-blocking IO.
I want to use the ExecutorService to run a series of the same Runnable/Callable tasks. I've searched around for a tutorial or an example, but nothing that involves actually setting the value of an existing Runnable/Callable object and then using submit() to send that object back into the ExecutorService.
Basically, here's what I want to do:
Get a list of servers.
Iterate thru the list of servers, calling InetAddress.getByName(host) to grab data on each host.
Collect that data into Server beans for storage in a database.
So, right now, with 10,000(+) servers, it takes forever. So, my thought was to use the ExecutorService to manage a pool of threads. What I can't seem to figure out is how to detect when one thread is finished so I can grab the data. Then I need to get the next server in the list, place it into the Task and then submit() back to the ExecutorService.
That said, what I've read so far seems to point to the following, ExecutorService, submit(), Callable, Future.
So, as a psuedo-process:
Get list of servers.
Set up ExecutorService with numThreads number of threads
Iterate numThreads and create numThreads WorkerTask() objects.
Submit() WorkerTask() to ExecutorService for processing.
Detect when a WorkerTask() has finished, grab the Callable (Future) result.
Get the next server.
Set the server value into the WorkerTask() <-- How? This is elusive ...
Submit() the WorkerTask() (with the new value) back to the ExecutorService.
Iterate again.
ExecutorService.shutdown()...
So, a tutorial or example of this would be great...especially an example of placing a new value into a WorkerTask(). Also, I'd appreciate any thoughts on this proposed solution? Is this bad, good, or if there is another way, I'm open.
02/09/2014 - Edit and Add
Hi, So the following is a first cut at this. But to answer some question being posed:
- I've solved the issue of placing new data in a Worker and resubmitting to the ExecutorService...see the code snippet.
- I've also solved the issue of "get the stuff"...I simply cast the Future() results to the Worker class...see the code snippet.
- Finally, while I could just allocate every server to a Worker() and a Future(), I'm concerned that the current 10,000 will grow and memory will become an issue.
That said, here's a first attempt, and this works pretty well. Runs much faster and only uses the getNumbnerThreads() Worker and Future objects:
public List<ServerBean> lookupHostIps ( List<ServerBean> theServerList ) {
//ServerBean serverDto = null;
ServerBean ipDto = null;
List<ServerBean> theResults = new ArrayList<ServerBean>();
List<HostLookupWorker> theWorkers = new ArrayList<HostLookupWorker>( getNumberThreads() );
List<Future<HostLookupWorker>> theFutures = new ArrayList<Future<HostLookupWorker>>( getNumberThreads() );
ExecutorService executor = Executors.newFixedThreadPool ( getNumberThreads() );
// WORKERS : Create the workers...prime them with a server
// bean...
//
for (int j = 0; j < getNumberThreads(); j++) {
//for (int j = 0; j < theServerList.size(); j++) {
theWorkers.add ( new HostLookupWorker( theServerList.get(j) ) );
Future<HostLookupWorker> theFuture = executor.submit ( theWorkers.get ( j ) );
theFutures.add ( j, theFuture );
}
int lloopItems = getNumberThreads(); /* loops thru all servers */
//int lloopThreads = 0; /* loops thru threads */
int lidxThread = 0; /* what thread is ready */
//int lidxFuture = 0; /* what future is ready */
boolean lblnNext = false; /* is a thread done/ready */
int lidxWorkers = 0; /* tracks the futures */
while ( lloopItems < theServerList.size() ) {
// READY : Is one of the threads ready for more work?
if ( lblnNext ) {
// VALUE : Grab the thread by index and set the next
// server value.
theWorkers.get ( lidxThread ).setBean ( theServerList.get(lloopItems) );
getLog().debug ( "Thread [" + lidxThread + "] Assigned Host ["+theServerList.get(lloopItems).getServerName ()+"] " );
// FUTURE : Package a new Future<HostLookupWorker>
// and submit it to the thread pool.
Future<HostLookupWorker> theFuture = executor.submit ( theWorkers.get ( lidxThread ) );
theFutures.add ( lidxThread, theFuture );
lblnNext = false; /* reset to allow for another thread */
lloopItems++; /* increment the main loop counter */
}
while ( !(lblnNext) ) {
try {
if ( theFutures.get(lidxWorkers).get() != null ) {
// GET THE STUFF : Grab the results from the Future...
HostLookupWorker ltheItem = theFutures.get(lidxWorkers).get();
if ( ltheItem.getValue () != null ) {
if (!ltheItem.getValue ().contains("Cannot find host")){
ipDto = new ServerBean ();
ipDto.setServerId ( ltheItem.getBean ().getServerId() );
ipDto.setServerName ( ltheItem.getBean ().getServerName() );
ipDto.setIpAddress ( ltheItem.getValue () );
theResults.add(ipDto);
}
lidxThread = lidxWorkers; /* this thread is ready for more work */
lblnNext = true; /* flag the upper condition to assign new work */
getLog().debug ( "Thread [" + lidxThread + "] Host ["+ltheItem.getHost ()+"] has IP ["+ltheItem.getValue()+"]" );
}
}
else {
getLog().debug ( "Thread [" + lidxThread + "] NULL" );
}
lidxWorkers++; /* next worker/future */
if ( lidxWorkers >= getNumberThreads() ) {
lidxWorkers = 0;
}
}
catch(ExecutionException e){
getLog().error ( e );
}
catch(InterruptedException e){
getLog().error ( e );
}
}
}
executor.shutdown ();
return theResults;
}
Here's the Worker/Thread class :
import java.net.*;
import java.util.concurrent.Callable;
import com.lmig.cdbatch.dto.ServerBean;
public class HostLookupWorker implements Callable {
private InetAddress node = null;
private String value = null;
private boolean busy = false;
private ServerBean bean = null;
public HostLookupWorker () {
this.busy = false;
}
// public HostLookupWorker ( String theHost ) {
// this.busy = false;
// this.host = theHost;
// }
public HostLookupWorker ( ServerBean theItem ) {
this.busy = false;
this.bean = theItem;
//this.host = theItem.getServerName ().trim ();
}
public String lookup ( String host ) {
if ( host != null ) {
// get the bytes of the IP address
try {
this.node = InetAddress.getByName ( host );
}
catch ( UnknownHostException ex ) {
this.value = "Not Found [" + getHost() + "]";
return "Not Found [" + host + "]";
}
if ( isHostname(host) ) {
getBean().setIpAddress ( node.getHostAddress() );
return node.getHostAddress();
}
else { // this is an IP address
//return node.getHostName();
return host;
}
}
return host;
} // end lookup
public boolean isHostname(String host) {
// Is this an IPv6 address?
if (host.indexOf(':') != -1)
return false;
char[] ca = host.toCharArray();
// if we see a character that is neither a digit nor a period
// then host is probably a hostname
for (int i = 0; i < ca.length; i++) {
if (!Character.isDigit(ca[i])) {
if (ca[i] != '.')
return true;
}
}
// Everything was either a digit or a period
// so host looks like an IPv4 address in dotted quad format
return false;
} // end isHostName
// public void run() {
// value = lookup ( getHost() );
//
// }
public Object call() throws Exception {
Thread.sleep ( 10000 );
this.busy = true;
this.value = lookup ( getHost() );
this.busy = false;
return this;
}
public String getHost() {
return getBean().getServerName ().trim ();
}
public void setHost(String host) {
getBean().setServerName ( host.trim () );
}
public InetAddress getNode() {
return node;
}
public void setNode(InetAddress node) {
this.node = node;
}
public String getValue() {
return value;
}
public void setValue(String value) {
this.value = value;
}
public boolean isBusy() {
return busy;
}
public void setBusy(boolean busy) {
this.busy = busy;
}
public ServerBean getBean() {
return bean;
}
public void setBean(ServerBean bean) {
this.bean = bean;
}
}
So, to summarize:
- The process does work, and works fast.
- I need to fix the code a little, as there are getNumberThreads() - 1 Futures left unprocessed when the larger while () loop finally finishes...
So, what I'm struggling with now is how to detect when a thread is finished...I've seen multiple examples, one testing for Future() != null, others testing for Future() == null. So which one is right?
I think the best aproach in this case would be create a task for every server since they will be executed by the threads in the pull and then use tge future objects to retrieve server information returned by the task.