We Keep Coding

Control amount of threads reading a file in java (only using synchronized, wait, notify and sleep)

In the problem I am trying to solve, each thread has to read the whole file, (maybe each thread will deliver its content to another task or any other purpose). After reading it, the thread should sleep a bit then try to read the file again, and only a given number(n) of threads should read the file. My attempt to solve this dealing with controling the amount of threads working is in the code below :
import java.util.*;
class Reader implements Runnable{
Thread t;
Controler c;
public Reader(Controler c){
t = new Thread(this);
this.c = c;
t.start();
}
public void run(){
Random ran = new Random();
int napTime;
while(true){
try{
w.intentarLeerArchivo(t);
//Specification says that each reader
//should wait a bit before trying to
//read the file again
napTime = ran.nextInt(1000);
t.sleep(napTime);
}catch(InterruptedException e){
System.out.println("InterruptedException");
}
}
}
}
class Controler{
Random ran;
LinkedList <Reader> readers;
int n;
int count;
public Controler(int n){
readers = new LinkedList <Reader>();
this.n = n;
count = 0;
ran = new Random();
}
public synchronized void getPermission(){
try{
while(count >= n){
wait();
}
notify();
}catch(InterruptedException e){
System.out.println("InterruptedException");
}
}
public synchronized void increaseCount(){
count++;
}
public synchronized void decreaseCount(){
count--;
System.out.println("There are " + count + " threads reading");
}
public void intentarLeerArchivo(Thread t){
int readTime = 1000;
try{
getPermission();
System.out.println("Thread " + t.getId() +" empezó a leer");
increaseCount();
t.sleep(readTime);
System.out.println("Thread " + t.getId() +" is reading");
System.out.println("Thread " + t.getId() + " finished reading");
decreaseCount();
} catch(InterruptedException e){
System.out.println("InterruptedException");
}
}
}
class Initializer{
int numReaders;
int maxReaders;
public Initializer(int numReaders, int maxReaders){
this.numReaders = numReaders;
this.maxReaders = maxReaders;
}
public void init(){
Controler c = new Controler(maxReaders);
for(int i = 0; i < numReaders; i++){
new Reader(c);
}
}
}
public class FileShare{
public static void main(String [] args){
Initializer c = new Initializer(100, 50);
c.init();
}
}
There are a few lines I wrote in order to debug. They print the state of each thread and the number of threads that are reading whenever one of them ends reading. But when I run the program, it turns out that suddenly there are more Threads reading the file than the ones there were supposed to be doing so. I guess it has something to do with my synchronization manipulation. What am I doing wrong?

When a thread is in the
while(...){wait()}
section of your implementation, it wait until someone notify it to go on. Right now, when you finish waiting, you notify right away.
Think about it, if once I get in I notify someone to come, he will not wait until I'm done before coming in. You want to use notify when you leave the file.

Reentrant Lock , busy waiting resolved by the new tryLock method

I was going through java doc description of lockInterruptibly method in ReentrantLock class. My intention was to see if the threads waiting to acquire lock are getting interrupted, may be I am doing it horribly wrong. I know there is an explicit way to call interrupt on Thread and it might be that executorService that I am using has wrapped that concept under it's API.
This behavior is seen with lock method as well
My purpose is to learn this concept in detail
import java.util.Date;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.locks.Lock;
public class LockInterruptibly extends Thread {
static ExecutorService es = Executors.newFixedThreadPool(5);
static Lock lock1 = new java.util.concurrent.locks.ReentrantLock();
public void methodA() {
if (lock1.tryLock()) {
try {
lock1.lockInterruptibly();
System.out.println("lock acquired by " + this.getName() + " of method A");
Thread.sleep(5000);
} catch (InterruptedException e) {
System.out.println("this thread " + this.getName() + " was interrupted");
e.printStackTrace();
}
} else {
System.out.println(this.getName() + "failed to acquire lock");
}
}
public void methodB() {
for (int i = 0; i < 5; i++) {
System.out.println("Printed by " + this.getName() + " - " + i);
}
lock1.unlock();
System.out.println(this.getName() + " is exiting at time " + new Date(System.currentTimeMillis()));
}
#Override
public void run() {
methodA();
methodB();
}
public static void main(String args[]) {
System.out.println(new Date(System.currentTimeMillis()));
for (int i = 0; i < 10; i++) {
Runnable r = new Thread(new LockInterruptibly());
es.submit(r);
}
System.out.println(new Date(System.currentTimeMillis()));
}
}
Now look at the console output below :
console logs showing the relative order, when each thread acquires lock and releases it
My questions is:
1)Why is this interleaving behavior? Why more than 1 thread are able to acquire lock (at least according to console output) , it's almost like a recursive behavior of acquiring locks. or is it just because console output is not in sync with what is happening actually?
2) Has it something to do with executor's way of treating time consuming threads and is normal behavior?

Thanks for your comments ! I was reading about the new Lock api and how you can try to lock before actually 'acquiring' it. So I wanted to code whether the threads are really non blocking or not. The updated code above assigns 5 threads and 10 tasks to executor. All the threads which fail to acquire lock, go on to print the 'for' loop. That means they are "not busy waiting" while the lock-acquiring thread is working in 'critical section'
In contrast I also implemented the synchronized way of doing it
import java.util.ArrayList;
import java.util.Date;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public class Synchronized extends Thread {
static ExecutorService es = Executors.newFixedThreadPool(5);
static ArrayList<Object> toBeLocked = new ArrayList<Object>();
public void methodA() {
synchronized (toBeLocked) {
try {
System.out.println("lock acquired by " + this.getName() + " of method A");
Thread.sleep(5000);
} catch (InterruptedException e) {
System.out.println("this thread " + this.getName() + "was interrupted");
}
}
for (int i = 0; i < 5; i++) {
System.out.println("Printed by " + this.getName() + " - " + i);
}
System.out.println(this.getName() + " is exiting at time " + new Date(System.currentTimeMillis()));
}
#Override
public void run() {
methodA();
}
public static void main(String args[]) {
System.out.println(new Date(System.currentTimeMillis()));
for (int i = 0; i < 10; i++) {
Runnable r = new Thread(new Synchronized());
es.submit(r);
}
System.out.println(new Date(System.currentTimeMillis()));
}
}
and found out that indeed all those threads were busy-waiting . Now with the new way of doing it I observed that all the threads which failed to acquire lock went ahead and never cared about returning .Are there any design patterns that answer both, optimum use of thread pools and being able to notify the next most worthy candidate.

Notify seems to be waking up more than one thread

I am doing a sample program with wait() and notify(), but when notify() is called, more than one thread is wakes up instead of one.
The code is:
public class MyQueue<T> {
Object[] entryArr;
private volatile int addIndex;
private volatile int pending = -1;
private final Object lock = new Object();
private volatile long notifiedThreadId;
private int capacity;
public MyQueue(int capacity) {
entryArr = new Object[capacity];
this.capacity = capacity;
}
public void add(T t) {
synchronized (lock) {
if (pending >= 0) {
try {
pending++;
lock.wait();
System.out.println(notifiedThreadId + ":" + Thread.currentThread().getId());
} catch (InterruptedException e) {
e.printStackTrace();
}
} else if (pending == -1) {
pending++;
}
}
if (addIndex == capacity) { // its ok to replace existing value
addIndex = 0;
}
try {
entryArr[addIndex] = t;
} catch (ArrayIndexOutOfBoundsException e) {
System.out.println("ARRAYException:" + Thread.currentThread().getId() + ":" + pending + ":" + addIndex);
e.printStackTrace();
}
addIndex++;
synchronized (lock) {
if (pending > 0) {
pending--;
notifiedThreadId = Thread.currentThread().getId();
lock.notify();
} else if (pending == 0) {
pending--;
}
}
}
}
public class TestMyQueue {
public static void main(String args[]) {
final MyQueue<String> queue = new MyQueue<>(2);
for (int i = 0; i < 200; i++) {
Runnable r = new Runnable() {
#Override
public void run() {
for (int i = 0; i < Integer.MAX_VALUE; i++) {
queue.add(Thread.currentThread().getName() + ":" + i);
}
}
};
Thread t = new Thread(r);
t.start();
}
}
}
After some time, I see two threads being wake up by single thread. The output looks like:
91:114
114:124
124:198
198:106
106:202
202:121
121:40
40:42
42:83
83:81
81:17
17:189
189:73
73:66
66:95
95:199
199:68
68:201
201:70
70:110
110:204
204:171
171:87
87:64
64:205
205:115
Here I see 115 thread notified two threads, and 84 thread notified two threads; because of this we are seeing the ArrayIndexOutOfBoundsException.
115:84
115:111
84:203
84:200
ARRAYException:200:199:3
ARRAYException:203:199:3
What is the issue in the program?

What is the issue in the program?
You have a couple of problems with your code that may be causing this behavior. First, as #Holder commented on, there are a lot of code segments that can be run by multiple threads simultaneously that should be protected using synchronized blocks.
For example:
if (addIndex == capacity) {
addIndex = 0;
}
If multiple threads run this then multiple threads might see addIndex == capacity and multiple would be overwriting the 0th index. Another example is:
addIndex++;
This is a classic race condition if 2 threads try to execute this statement at the same time. If addIndex was 0 beforehand, after the 2 threads execute this statement, the value of addIndex might be 1 or 2 depending on the race conditions.
Any statements that could be executed at the same time by multiple threads have to be properly locked within a synchronized block or otherwise protected. Even though you have volatile fields, there can still be race conditions because there are multiple operations being executed.
Also, a classic mistake is to use if statements when checking for over or under flows on your array. They should be while statements to make sure you don't have the class consumer producer race conditions. See my docs here or take a look at the associated SO question: Why does java.util.concurrent.ArrayBlockingQueue use 'while' loops instead of 'if' around calls to await()?

reasons for deadlock in producer/consumer and debugging

Though the below is well known topic I want Your ideas please.
I had written a small program as below: All the producers are queued up and also the consumers. I couldn't understand why it is so. What are the scenarios where it can block completely.
Let us consider Producers/consumers are waiting for lock on array and what making consumers /producers to exit out the synchronized block. I mean it has to move atleast slowly but deadlock must not happen. I believe.
Here I have 2 questions:
1. What are the scenarios that deadlock is happening.
2. How to understand what is happening under the hood. I mean how to debug.
public class ProducerConsumer implements Runnable {
boolean producer = false;
private volatile int i = 0;
int[] array = new int[10];
public static String getThreadName() {
return Thread.currentThread().getName();
}
public void producer() {
try {
synchronized (array) {
while (i > 9) {
System.out.println("Producer of " + getThreadName()
+ " is waiting i " + i);
array.wait();
System.out.println("Producer of " + getThreadName()
+ " came out of sleep i " + i);
}
System.out.println("Producer of " + getThreadName()
+ " in synchronized block i" + i);
array[i] = generateRandom();
System.out.println("Producer of " + getThreadName()
+ " inserted in array " + array[i] + " index " + i);
i++;
array.notifyAll();
}
Thread.sleep(100);
} catch (InterruptedException e) {
System.out.println("Producer of " + getThreadName()
+ " interrupted " + e);
}
}
public void consumer() {
try {
synchronized (array) {
while (i < 0) {
System.out.println("Consumer of " + getThreadName()
+ " is waiting i " + i);
array.wait();
System.out.println("Consumer of " + getThreadName()
+ " came out of sleep i " + i);
}
System.out.println("Consumer of " + getThreadName()
+ " in synchronized block extracted value " + array[i]
+ " of index " + i);
i--;
array.notifyAll();
}
Thread.sleep(100);
} catch (InterruptedException e) {
System.out.println("Consumer of " + getThreadName()
+ " interrupted " + e);
}
}
public static int generateRandom() {
Random random = new Random();
return random.nextInt(10);
}
public static void main(String[] args) {
ProducerConsumer pc = new ProducerConsumer();
for (int i = 0; i < 4; i++) {
if (i % 2 == 0)
new Thread(pc, "producer thread").start();
else {
new Thread(pc, "consumer thread").start();
}
}
}
public void run() {
while (true) {
if (getThreadName().equalsIgnoreCase("producer thread")) {
producer();
} else {
consumer();
}
}
}
}
It struck as below output:
Consumer of consumer thread in synchronized block extracted value 0 of index 0
Producer of producer thread in synchronized block i-1
Producer of producer thread in synchronized block i-1
Consumer of consumer thread is waiting i -1
Consumer of consumer thread is waiting i -1

Your code is incorrect in many places.
I expect that all threads just finish with exceptions, either because of
IllegalMonitorException (calling notify() on ProducerConsumer object
but there is no synchronized block on this ProducerConsumer object)
ArrayIndexOfBoundsException (i can become 10 in produce() method)
Have you checked error output?

Java provides a neat implementation of concurrent programs via its java.util.concurrent package. So rather than trying to reinvent the wheel, and getting it all wrong, you should use the Concurrent API to handle locking in a safer way. Here's a simulation of a Producer-Consumer:
import java.util.Random;
import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
/**
* We want a Producer thread to create random values, and the Consumer thread to
* consume it. One caveat is that if the Producer has already created a random
* value, which the Consumer thread hasn't consumed yet, the Producer thread
* blocks or waits. On the flip side, the Consumer thread waits for the Producer
* thread to produce some value if the Producer thread hasn't already.
* <p/>
* Write a Program to simulate such a situation.
*/
public class ProducerConsumerCommunication
{
private volatile boolean running = true;
private ArrayBlockingQueue<Integer> buffer = new ArrayBlockingQueue<>(1);
private Random random = new Random(System.currentTimeMillis());
public ProducerConsumerCommunication()
{
ExecutorService service = Executors.newCachedThreadPool();
service.execute(new ProducerTask());
service.execute(new ConsumerTask());
service.shutdown();
}
public static void main(String[] args)
{
new ProducerConsumerCommunication();
}
private class ProducerTask implements Runnable
{
public void run()
{
while (running)
{
try
{
Thread.sleep(random.nextInt(2000));
Integer value = random.nextInt();
buffer.put(value); // Blocks if buffer is full.
System.out.println("Value Put: " + value);
}
catch (InterruptedException e)
{
e.printStackTrace();
}
}
}
}
private class ConsumerTask implements Runnable
{
public void run()
{
while (running)
{
try
{
Thread.sleep(random.nextInt(2000));
Integer value = buffer.take(); // Blocks if buffer is empty.
System.out.println("Value Taken: " + value);
}
catch (InterruptedException e)
{
e.printStackTrace();
}
}
}
}
}
Try running it and see for yourself as to how easy and intuitive it is to implement such scenarios using the Concurrent API. It also keeps your code clean and lets you focus on the problem at hand.
Reasons for deadlock in Producer Consumer problem aren't that many. If one thread has lock on an object A and is waiting for lock on object B to be released, while if other thread has lock on object B at the same time and is waiting for lock on object A to be released, deadlock situation arises.

Useful example of a shutdown hook in Java?

I'm trying to make sure my Java application takes reasonable steps to be robust, and part of that involves shutting down gracefully. I am reading about shutdown hooks and I don't actually get how to make use of them in practice.
Is there a practical example out there?
Let's say I had a really simple application like this one below, which writes numbers to a file, 10 to a line, in batches of 100, and I want to make sure a given batch finishes if the program is interrupted. I get how to register a shutdown hook but I have no idea how to integrate that into my application. Any suggestions?
package com.example.test.concurrency;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.PrintWriter;
public class GracefulShutdownTest1 {
final private int N;
final private File f;
public GracefulShutdownTest1(File f, int N) { this.f=f; this.N = N; }
public void run()
{
PrintWriter pw = null;
try {
FileOutputStream fos = new FileOutputStream(this.f);
pw = new PrintWriter(fos);
for (int i = 0; i < N; ++i)
writeBatch(pw, i);
}
catch (FileNotFoundException e) {
e.printStackTrace();
}
finally
{
pw.close();
}
}
private void writeBatch(PrintWriter pw, int i) {
for (int j = 0; j < 100; ++j)
{
int k = i*100+j;
pw.write(Integer.toString(k));
if ((j+1)%10 == 0)
pw.write('\n');
else
pw.write(' ');
}
}
static public void main(String[] args)
{
if (args.length < 2)
{
System.out.println("args = [file] [N] "
+"where file = output filename, N=batch count");
}
else
{
new GracefulShutdownTest1(
new File(args[0]),
Integer.parseInt(args[1])
).run();
}
}
}

You could do the following:
Let the shutdown hook set some AtomicBoolean (or volatile boolean) "keepRunning" to false
(Optionally, .interrupt the working threads if they wait for data in some blocking call)
Wait for the working threads (executing writeBatch in your case) to finish, by calling the Thread.join() method on the working threads.
Terminate the program
Some sketchy code:
Add a static volatile boolean keepRunning = true;
In run() you change to
for (int i = 0; i < N && keepRunning; ++i)
writeBatch(pw, i);
In main() you add:
final Thread mainThread = Thread.currentThread();
Runtime.getRuntime().addShutdownHook(new Thread() {
public void run() {
keepRunning = false;
mainThread.join();
}
});
That's roughly how I do a graceful "reject all clients upon hitting Control-C" in terminal.
From the docs:
When the virtual machine begins its shutdown sequence it will start all registered shutdown hooks in some unspecified order and let them run concurrently. When all the hooks have finished it will then run all uninvoked finalizers if finalization-on-exit has been enabled. Finally, the virtual machine will halt.
That is, a shutdown hook keeps the JVM running until the hook has terminated (returned from the run()-method.