We Keep Coding

Creating n threads, max 5 active at a time, wait for all to finish before continuing

I'm building a web crawler with Selenium and MVC design pattern in Java. I'm trying to achieve three things:
To create n number of threads depending on the length of my database (sidb) storing search settings.
Because I'm having a TablePanel in my GUI with the data, i need all threads to finish before proceeding. Otherwise i will lose some data because table.refresh() method is called to early.
As more than 5 Firefox windows running at once will slow down my computer too much i would like a maximum of 5 threads running at any given time.
The following code solves the first problem, and to some degree the second. I had to make a nasty Thread.sleep() to prevent it from running refresh on the table too early.
public void runSearchItems() {
for (int i = 0; i < sidb.getSize(); i++) {
final int num = i;
Thread tn = new Thread(new Runnable() {
#Override
public void run() {
if (sidb.getSearhItem().get(num).getFormevent().getDomainBox().equalsIgnoreCase("www.someURL.com") == true) {
String searchField = sidb.getSearhItem().get(num).getFormevent().getSearchField();
int searchCat = sidb.getSearhItem().get(num).getFormevent().getSearchCategory();
boolean defect = sidb.getSearhItem().get(num).getFormevent().isDefectCheck();
boolean region = sidb.getSearhItem().get(num).getFormevent().isRegionCheck();
String arrange = sidb.getSearhItem().get(num).getFormevent().getArrangeBy();
ArrayList<SiteData> ls = wb.searchWebSite(searchField, searchCat, defect, region, arrange);
for (int j = 0; j < ls.size(); j++) {
db.addSiteData(ls.get(j));
}
}
}
});
tn.start();
try {
tn.join();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
try {
Thread.sleep(3000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
What is the correct way to do this?
Thanks!

Small changes to your code
public void runSearchItems() {
List<Thread> allThreads = new ArrayList<>();
for (int i = 0; i < sidb.getSize(); i++) {
final int num = i;
Thread tn = new Thread(new Runnable() {
#Override
public void run() {
if (sidb.getSearhItem().get(num).getFormevent().getDomainBox().equalsIgnoreCase("www.someURL.com") == true) {
String searchField = sidb.getSearhItem().get(num).getFormevent().getSearchField();
int searchCat = sidb.getSearhItem().get(num).getFormevent().getSearchCategory();
boolean defect = sidb.getSearhItem().get(num).getFormevent().isDefectCheck();
boolean region = sidb.getSearhItem().get(num).getFormevent().isRegionCheck();
String arrange = sidb.getSearhItem().get(num).getFormevent().getArrangeBy();
ArrayList<SiteData> ls = wb.searchWebSite(searchField, searchCat, defect, region, arrange);
for (int j = 0; j < ls.size(); j++) {
db.addSiteData(ls.get(j));
}
}
}
});
allThreads.add(tn);
}
for(Thread t:allThreads)
t.start();
for(Thread t:allThreads)
try {
t.join();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
But yes, you are better of with Thread Pool
as an example https://howtodoinjava.com/java/multi-threading/java-thread-pool-executor-example/

thread execution should stop when random number being generated in for loop (1-100) equals natural numbers generated (1,2,3...100)

main theme of the program is that thread should be stopped when i run two threads ie one thread is generating random numbers and other generating natural numbers the thread execution must be stopped when they both generate same number
class methmatch {
public void match() {
}
}
class Randgen extends Thread {
static int value;
public void run() {
Random random = new Random();
for (int i = 1; i <= 100; i++) {
value = 1 + random.nextInt(100);
System.out.println(value);
try {
Thread.sleep(100);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}
class Ordergen extends Thread {
static int j;
public void run() {
for (int i = 1; i <= 1; i++) {
for ( j = 1; j <= 100; j++)
{
System.out.println(j);
try {
Thread.sleep(100);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}
}
public class Randmatch {
public static void main(String[] args) {
Ordergen obj1 = new Ordergen();
Randgen obj2 = new Randgen();
methmatch obj3= new methmatch();
obj1.start();
obj2.start();
}
}

How to consume in Producer-Consumer using Semphores?

I am trying out the Producer-Consumer problem using Semaphore. The program looks fine to me except for one place.
public class ProducerConsumerWithSemaphores
{
private final ArrayList<Integer> list = new ArrayList<>(5);
private final Semaphore semaphoreProducer = new Semaphore(1);
private final Semaphore semaphoreConsumer = new Semaphore(0);
private void produce() throws InterruptedException
{
for(int i = 0;i< 5;i++)
{
semaphoreProducer.acquire();
list.add(i);
System.out.println("Produced: " + i);
semaphoreConsumer.release();
}
}
private void consumer() throws InterruptedException
{
while (!list.isEmpty()) /// This line is where I have the doubt
{
semaphoreConsumer.acquire();
System.out.println("Consumer: " + list.remove(list.size()-1));
semaphoreProducer.release();
Thread.sleep(100);
}
}
public static void main(String[] args)
{
final ProducerConsumerWithSemaphores obj = new ProducerConsumerWithSemaphores();
new Thread(new Runnable()
{
#Override
public void run()
{
try
{
obj.produce();
} catch (InterruptedException e)
{
e.printStackTrace();
}
}
}).start();
new Thread(new Runnable()
{
#Override
public void run()
{
try
{
obj.consumer();
} catch (InterruptedException e)
{
e.printStackTrace();
}
}
}).start();
}
}
Is it okay to check the list if it is not empty before acquiring the semaphore? Will this cause any problem in multithreaded environment?

private void consumer() throws InterruptedException
{
while (!list.isEmpty()) /// This line is where I have the doubt
The problem is, if consumer runs faster than producer, your consumer quit immediately, then you have no consumer!!
The correct example looks like,
Producer–consumer problem#Using semaphores. I believe your intention is not to use true as endless loop because you want Producer/Consumer to quit when job is done. If that's your intention, you can 1. set a totalCount to end the loop. 2. Or a boolean flag which will be set by producer after putItemIntoBuffer when producer put the last one. The flag must be protected as well as the buffer.(update: this method doesn't work if there's multiple producers/consumers) 3. Simulate EOF ( idea taken from producer - consume; how does the consumer stop?)
Will this cause any problem in multithreaded environment?
Your critical section (your list) is not protected . Usually we use 3 semaphores. The 3rd one is used as a mutex to protect the buffer.
To stop producers/consumers,
Example code with method 1:
public class Test3 {
private Semaphore mutex = new Semaphore(1);
private Semaphore fillCount = new Semaphore(0);
private Semaphore emptyCount = new Semaphore(3);
private final List<Integer> list = new ArrayList<>();
class Producer implements Runnable {
private final int totalTasks;
Producer(int totalTasks) {
this.totalTasks = totalTasks;
}
#Override
public void run() {
try {
for (int i = 0; i < totalTasks; i++) {
emptyCount.acquire();
mutex.acquire();
list.add(i);
System.out.println("Produced: " + i);
mutex.release();
fillCount.release();
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
class Consumer implements Runnable {
private final int totalTasks;
Consumer(int totalTasks) {
this.totalTasks = totalTasks;
}
#Override
public void run() {
try {
for (int i = 0; i < totalTasks; i++) {
fillCount.acquire();
mutex.acquire();
int item = list.remove(list.size() - 1);
System.out.println("Consumed: " + item);
mutex.release();
emptyCount.release();
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
public void runTest() {
int numProducer = 3;
int tasksPerProducer = 10;
int numConsumer = 6;
int tasksPerConsumer = 5;
for (int i = 0; i < numProducer; i++) {
new Thread(new Producer(tasksPerProducer)).start();
}
for (int i = 0; i < numConsumer; i++) {
new Thread(new Consumer(tasksPerConsumer)).start();
}
}
public static void main(String[] args) throws IOException {
Test3 t = new Test3();
t.runTest();
}
}
Example code with method 3:
public class Test4 {
private Semaphore mutex = new Semaphore(1);
private Semaphore fillCount = new Semaphore(0);
private Semaphore emptyCount = new Semaphore(3);
private Integer EOF = Integer.MAX_VALUE;
private final Queue<Integer> list = new LinkedList<>(); // need to put/get data in FIFO
class Producer implements Runnable {
private final int totalTasks;
Producer(int totalTasks) {
this.totalTasks = totalTasks;
}
#Override
public void run() {
try {
for (int i = 0; i < totalTasks + 1; i++) {
emptyCount.acquire();
mutex.acquire();
if (i == totalTasks) {
list.offer(EOF);
} else {
// add a valid value
list.offer(i);
System.out.println("Produced: " + i);
}
mutex.release();
fillCount.release();
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
class Consumer implements Runnable {
#Override
public void run() {
try {
boolean finished = false;
while (!finished) {
fillCount.acquire();
mutex.acquire();
int item = list.poll();
if (EOF.equals(item)) {
// do not consume this item because it means EOF
finished = true;
} else {
// it's a valid value, consume it.
System.out.println("Consumed: " + item);
}
mutex.release();
emptyCount.release();
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
public void runTest() {
int numProducer = 3;
int tasksPerProducer = 10;
for (int i = 0; i < numProducer; i++) {
new Thread(new Producer(tasksPerProducer)).start();
}
int numConsumer = numProducer; // producers will put N EOFs to kill N consumers.
for (int i = 0; i < numConsumer; i++) {
new Thread(new Consumer()).start();
}
}
public static void main(String[] args) throws IOException {
Test4 t = new Test4();
t.runTest();
}
}

Instead of using two semaphores why dont you use a single semaphore to such that the synchronization is made between threads link
Additional you can use ArrayBlockingQueue which are thread safe to properly demonstrate the Producer Consumer Problem.

Get Size of BlockingQueue while Multithreading

I have a multi-threaded process, 5 threads, and another thread working as a status object reporting the size of the BlockingQueue. Problem is the Status thread reports 100% first, which is correct, but then goes right to 0%.
I want it to count down the percentage.
Here is my code:
Thread[] workers = new Thread[NUMBER_OF_THREADS];
for (int x = 0; x < NUMBER_OF_THREADS; x++) {
workers[x] = new Thread(new S3ObjectDownloader(filesToDownload, currentYear));
workers[x].start();
}
for (int x = 0; x < NUMBER_OF_THREADS; x++) {
try {
workers[x].join();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
Here is the status object instantiation:
int downloadSize = filesToDownload.size();
Thread statusThread = new Thread(new Status(filesToDownload, currentYear,downloadSize,"DOWNLOADING..."));
statusThread.start();
Here is that actual Status object run method:
public void run() {
while(!queue.isEmpty()){
try {
float completion = (queue.size()*1)/this.queueSize;
System.out.println(this.jobeName+" : "+this.conferenceYear+ " completion..."+MessageFormat.format("{0,number,#.##%}",completion));
TimeUnit.SECONDS.sleep(30);;
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
Im adding in the actual S3ObjectDownloader:
public void run() {
//aws credentials
this.s3 = new AmazonS3Client(new ClasspathPropertiesFileCredentialsProvider());
//log4j configuration
PropertyConfigurator.configure("/home/ubuntu/log4j.properties");
//attempt to poll the queue
while (!queue.isEmpty()) {
String fileName = queue.poll() + ".mp4";
String FULL_PATH = "best_of_ats/" + this.conferenceYear + "/videos/" + fileName;
File f = new File("/home/ubuntu/" + fileName);
if (fileName != null && !f.exists() && s3.doesObjectExist(BUCKET_NAME, FULL_PATH)) {
OutputStream out = null;
InputStream in = null;
S3Object s3obj = null;
try {
s3obj = s3.getObject(this.BUCKET_NAME,
FULL_PATH);
in = s3obj.getObjectContent();
//System.out.println("Downloading File " + FULL_PATH + "....");
} catch (AmazonS3Exception s3e) {
// s3e.printStackTrace();
//System.out.println("Problem downloading file..." + FULL_PATH);
s3e.printStackTrace();
logger.info("Problem with file..." + FULL_PATH);
continue;
}
try {
out = new FileOutputStream(new File(fileName));
int read = 0;
byte[] bytes = new byte[1024];
while ((read = in.read(bytes)) != -1) {
out.write(bytes, 0, read);
}
out.flush();
out.close();
in.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
//System.out.println("problem writing output..." + FULL_PATH);
logger.info("problem writing output..." +FULL_PATH);
continue;
}
}
} // end while...
}
And here is the Status Class:
public class Status implements Runnable {
private String conferenceYear;
private Queue<String>queue;
private int queueSize;
private String jobeName;
public Status(Queue<String> queue, String conferenceYear, int queueSize, String jobName){
this.conferenceYear = conferenceYear;
this.queue = queue;
this.queueSize = queueSize;
this.jobeName = jobName;
}
#Override
public void run() {
while(!queue.isEmpty()){
try {
float completion = (queue.size()*1)/this.queueSize;
System.out.println(this.jobeName+" : "+this.conferenceYear+ " completion..."+MessageFormat.format("{0,number,#.##%}",completion));
TimeUnit.SECONDS.sleep(30);;
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
}
Here is the calling class:
public static void main(String[] args) {
BlockingQueue<String> filesToDownload = new LinkedBlockingDeque<String>(1024);
BlockingQueue<String> filesToPreview = new LinkedBlockingDeque<String>(1024);
BlockingQueue<String> filesToUpload = new LinkedBlockingDeque<String>(1024);
String currentYear = String.valueOf(Calendar.getInstance().get(Calendar.YEAR));
// DB connection.
ATSStoreDB db = new ATSStoreDB();
PreparedStatement st = null;
Connection conn = null;
conn = db.getConnection();
// get ids from ats_store.products.
try {
st = conn.prepareStatement(sql);
st.setString(1, currentYear);
ResultSet rs = st.executeQuery();
// add each id to IDS.
while (rs.next()) {
filesToDownload.add(rs.getString("product_id"));
filesToPreview.add(rs.getString("product_id"));
filesToUpload.add(rs.getString("product_id"));
}
conn.close();
} catch (SQLException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
/*
* Distribute IDS to several threads.
*/
//start up the Status Object class.
int downloadSize = filesToDownload.size();
Thread statusThread = new Thread(new Status(filesToDownload, currentYear,downloadSize,"DOWNLOADING..."));
statusThread.start();
/**
* download the files
*/
Thread[] workers = new Thread[NUMBER_OF_THREADS];
for (int x = 0; x < NUMBER_OF_THREADS; x++) {
workers[x] = new Thread(new S3ObjectDownloader(filesToDownload, currentYear));
workers[x].start();
}
for (int x = 0; x < NUMBER_OF_THREADS; x++) {
try {
workers[x].join();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
/**
* create previews
*/
int previewSize = filesToPreview.size();
statusThread = new Thread(new Status(filesToPreview, currentYear,previewSize,"PREVIEWING..."));
statusThread.start();
workers = new Thread[NUMBER_OF_THREADS];
for (int x = 0; x < NUMBER_OF_THREADS; x++) {
workers[x] = new Thread(new Worker(filesToPreview, currentYear));
workers[x].start();
}
for (int x = 0; x < NUMBER_OF_THREADS; x++) {
try {
workers[x].join();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}

I can spot one problem with your code right away:
float completion = (queue.size()*1)/this.queueSize;
What's the point of *1? Both queue.size() and this.queueSize are integers. You're turning integer division into... integer division. A good compiler will probably optimize it right away. You probably meant to write something like
float completion = (queue.size() * 1.0f) / this.queueSize;

Java - Multithreading with ImageIO

I have a program that loads slowly, which I guess is due to the amount of image resources I have to load at the beginning. I thought multi-threading would help, but now I'm not so sure. Here is my automatic multi-threading method.
private static Thread[] t;
private static int currentThreads;
public static void loadWithThreads(Object[] array, IntegerRunnable r) {
final int threads = Runtime.getRuntime().availableProcessors();
t = new Thread[threads];
for (int i = 0; i < threads; i ++) {
t[i] = new Thread("HMediaConverter") {
final int id = currentThreads;
int items = (array.length / threads) * currentThreads;
#Override
public void run() {
super.run();
for (int i = items; i < (items + (array.length / threads)); i ++) {
r.run(i);
}
//Recycle this thread so it can be used for another time.
try {
t[id].join();
lock.notifyAll();
currentThreads --;
} catch (InterruptedException e) {
e.printStackTrace();
}
}
};
t[i].setPriority(Thread.MAX_PRIORITY);
t[i].start();
currentThreads ++;
}
}
And here is my image loading code:
public static ImageIcon loadImageIcon(String path) {
return new ImageIcon(ImageIO.read(Tools.class.getClassLoader().getResource(path));
}
Surely there is a way to speed things up? I'm running this on a perfectly good Intel i5, it shouldn't be this slow, so it must be my code.

Loading 113 images of a total of 159.14mb with...
public static void loadWithoutThreads(File[] array) {
for (File file : array) {
try {
ImageIO.read(file);
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
Took ~15s
With...
public static void loadWithThreads(File[] array) {
final int threads = Runtime.getRuntime().availableProcessors();
t = new Thread[threads];
CountDownLatch latch = new CountDownLatch(threads);
for (int i = 0; i < threads; i++) {
t[i] = new Thread("HMediaConverter") {
final int id = currentThreads;
int items = (array.length / threads) * currentThreads;
#Override
public void run() {
try {
System.out.println("Starting " + id);
for (int i = items; i < (items + (array.length / threads)); i++) {
try {
System.out.println(i + ": " + array[i]);
ImageIO.read(array[i]);
} catch (IOException ex) {
ex.printStackTrace();
}
}
} finally {
latch.countDown();
}
}
};
t[i].setPriority(Thread.MAX_PRIORITY);
System.out.println("Start " + i);
t[i].start();
currentThreads++;
}
try {
latch.await();
} catch (InterruptedException ex) {
ex.printStackTrace();
}
}
took ~11s
With...
public static void loadWithExecutor(File[] images) {
ExecutorService service = Executors.newFixedThreadPool(2);
List<ImageLoadingTask> tasks = new ArrayList<>(images.length);
for (File file : images) {
tasks.add(new ImageLoadingTask(file));
}
try {
List<Future<BufferedImage>> results = service.invokeAll(tasks);
} catch (InterruptedException ex) {
ex.printStackTrace();
}
service.shutdown();
}
public static class ImageLoadingTask implements Callable<BufferedImage> {
private File file;
public ImageLoadingTask(File file) {
this.file = file;
}
#Override
public BufferedImage call() throws Exception {
return ImageIO.read(file);
}
}
Took ~7s
The ExecutorService is more efficient because when one thread is processing a larger file, the other can be processing a number of small files. This is achieved by pooling the threads that aren't doing any work until they are needed, allowing a thread to perform a lot of short work, while the other thread(s) are also busy. You don't need to wait as long
Have a look at Executors for more details

The following is a re-write that should work that is close to what the op wrote. A re-write into A fixed-size thread pool would probably be better.
//import java.util.concurrent.atomic.AtomicInteger;
private static Thread[] t;
private static AtomicInteger completedLoads = new AtomicInteger(0);
public static void loadWithThreads(Object[] array, IntegerRunnable r) {
final int threads = Runtime.getRuntime().availableProcessors();
t = new Thread[threads];
completedLoads = new AtomicInteger(0);
int targetLoads = array.length;
int itemsPerThread = (array.length / threads);
for (int i = 0; i < threads; i ++) {
t[i] = new Thread("HMediaConverter" + i) {
int startItem = itemsPerThread * i;
#Override
public void run() {
super.run();
for (int i = startItem; i < startItem + itemsPerThread; i ++) {
try {
r.run(i);
}
finally {
completedLoads.incrementAndGet();
}
}
}
};
t[i].setPriority(Thread.MAX_PRIORITY);
t[i].start();
}
// Wait for the images to load
while (completedLoads.get() < targetLoads)
{
try {
Thread.sleep(100);
}
catch (InterruptedException ie) {
// ignore
}
}
}

Isolate which part does the slowing down - e.g by running System.currentTimeMillis() btween major segmnst then show us where is the biggest time - or show us all the program.
Threads handling as noted is questionable and you shouldn't use methods such as join etc out of the box unless you have seen it sometwhere provably working.
So post times and we'll take it from there - it could be the images it could be the threads