I'm trying to make sure my Java application takes reasonable steps to be robust, and part of that involves shutting down gracefully. I am reading about shutdown hooks and I don't actually get how to make use of them in practice.
Is there a practical example out there?
Let's say I had a really simple application like this one below, which writes numbers to a file, 10 to a line, in batches of 100, and I want to make sure a given batch finishes if the program is interrupted. I get how to register a shutdown hook but I have no idea how to integrate that into my application. Any suggestions?
package com.example.test.concurrency;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.PrintWriter;
public class GracefulShutdownTest1 {
final private int N;
final private File f;
public GracefulShutdownTest1(File f, int N) { this.f=f; this.N = N; }
public void run()
{
PrintWriter pw = null;
try {
FileOutputStream fos = new FileOutputStream(this.f);
pw = new PrintWriter(fos);
for (int i = 0; i < N; ++i)
writeBatch(pw, i);
}
catch (FileNotFoundException e) {
e.printStackTrace();
}
finally
{
pw.close();
}
}
private void writeBatch(PrintWriter pw, int i) {
for (int j = 0; j < 100; ++j)
{
int k = i*100+j;
pw.write(Integer.toString(k));
if ((j+1)%10 == 0)
pw.write('\n');
else
pw.write(' ');
}
}
static public void main(String[] args)
{
if (args.length < 2)
{
System.out.println("args = [file] [N] "
+"where file = output filename, N=batch count");
}
else
{
new GracefulShutdownTest1(
new File(args[0]),
Integer.parseInt(args[1])
).run();
}
}
}
You could do the following:
Let the shutdown hook set some AtomicBoolean (or volatile boolean) "keepRunning" to false
(Optionally, .interrupt the working threads if they wait for data in some blocking call)
Wait for the working threads (executing writeBatch in your case) to finish, by calling the Thread.join() method on the working threads.
Terminate the program
Some sketchy code:
Add a static volatile boolean keepRunning = true;
In run() you change to
for (int i = 0; i < N && keepRunning; ++i)
writeBatch(pw, i);
In main() you add:
final Thread mainThread = Thread.currentThread();
Runtime.getRuntime().addShutdownHook(new Thread() {
public void run() {
keepRunning = false;
mainThread.join();
}
});
That's roughly how I do a graceful "reject all clients upon hitting Control-C" in terminal.
From the docs:
When the virtual machine begins its shutdown sequence it will start all registered shutdown hooks in some unspecified order and let them run concurrently. When all the hooks have finished it will then run all uninvoked finalizers if finalization-on-exit has been enabled. Finally, the virtual machine will halt.
That is, a shutdown hook keeps the JVM running until the hook has terminated (returned from the run()-method.
Related
I'm studying about parallel programming and testing this brute force code, to crack a password with 7 positions without and with parallelism, but when applying parallel programming, the code is running slower than before. Does anyone know why? How can I use parallel programming in this code to make it faster? thanks.
public class Worker extends Thread{
private static final int[] valuesCaracter = {48, 48, 48, 48, 48, 48, 48};
private static final int arraySize = valuesCaracter.length;
private static char[] symbols = new char[arraySize];
private static String password = null;
#Override
public void run() {
generatePasswordCaracters();
}
public static int ajustaValor(int valueASCII) {
return switch (valueASCII) {
case 58 -> 65;
case 91 -> 97;
default -> valueASCII;
};
}
public static String generatePasswordCaracters() {
for (int i = 0; i < arraySize; i++) {
while (valuesCaracter[i] <= 122) {
for (int j = 0; j < symbols.length; j++) {
symbols[j] = (char) valuesCaracter[j];
}
password = new String(symbols);
System.out.println(password);
valuesCaracter[6]++;
for(int k = arraySize - 1; k >= 1; k--) {
if (valuesCaracter[k] > 122) {
valuesCaracter[k - 1]++;
valuesCaracter[k] = 48;
} else {
valuesCaracter[k] = ajustaValor(valuesCaracter[k]);
}
}
}
}
return password;
}
}
import net.lingala.zip4j.ZipFile;
import net.lingala.zip4j.model.FileHeader;
import java.io.File;
import java.util.List;
public class App {
public static void main(String[] args) {
Worker[] Workers =
{new Worker(),
new Worker(),
new Worker(),
new Worker(),
new Worker(),
new Worker()};
for (Worker workerStart : Workers) {
workerStart.start();
}
try {
for (Worker workerJoin : Workers) {
workerJoin.join();
}
}catch (InterruptedException e) {
System.out.println(e.getMessage());
}
try {
ZipFile zipFile = new ZipFile(new File("C:\\vsCodeProjects-Java\\BruteForce\\src\\file.zip"));
if (zipFile.isEncrypted()) {
zipFile.setPassword(Worker.generatePasswordCaracters().toCharArray());
}
List<FileHeader> fileHeaderList = zipFile.getFileHeaders();
for (FileHeader o : fileHeaderList) {
zipFile.extractFile(o, "C:\\vsCodeProjects-Java\\BruteForce\\src");
System.out.println("This is the password");
}
} catch (Exception e) {
System.out.println("Wrong password");
}
}
}
This "parallel" code is being executed sequentially.
Main thread blocked at each step of iteration, wait for every worker to do its job. Javadoc says join():
Waits for this thread to die.
So actually although this you've created multiple threads your code will be even slower than sequential execution because threads are not cheap.
Choose either: don't control the execution of your threads, or perform these tasks in a single thread.
Sidenote: implementing Runnable is more flexible than extending Thread. Because you can start a thread only once, but the same runnable task could be reused as many times as needed.
And if your goal to parallelize a single task, you might take a look at ForkJoin framework, which will allow splitting the task in to multiple. For that Worker class should extend either RecursiveTask or RecursiveAction and override method compute().
I'd like to keep a counter of executed threads, to use in the same threads that I am executing.
The problem here is that although the counter increases, it increases unevenly and from the console output I got this (I have a for loop that executes 5 threads with ExecutorService):
This is a test. N:3
This is a test. N:4
This is a test. N:4
This is a test. N:4
This is a test. N:4
As you can see instead of getting 1,2,3,4,5 I got 3,4,4,4,4.
I assume this is because the for loop is running fast enough to execute the threads, and the threads are fast enough to execute the code requesting for the counter faster than the counter can update itself (does that even make sense?).
Here is the code (it is smaller and there is no meaningful use for the counter):
for (int i = 0; i < 5; i++)
{
Thread thread;
thread = new Thread()
{
public void run()
{
System.out.println("This is test. N: "+aldo );
//In here there is much more stuff, saying it because it might slow down the execution (if that is the culprit?)
return;
}
};
threadList.add(thread);
}
//later
for (int i = 0; i < threadList.size(); i++)
{
executor.execute(threadList.get(i));
aldo = aldo + 1;
}
executor.shutdown();
try
{
executor.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
}
catch (InterruptedException e)
{
}
Yes, aldo the counter ( with a few other lists, I think) are missing from the code (they are very simple).
The best way I know of doing this is by creating a custom thread class with a constructor that passes in a number. The variable holding the number can then be used later for any needed logging. Here is the code I came up with.
public static void main(String[] args) {
class NumberedThread implements Runnable {
private final int number;
public NumberedThread(int number) {
this.number = number;
}
#Override
public void run() {
System.out.println("This is test. N: " + number);
}
}
List<Thread> threadList = new ArrayList<>();
for (int i = 1; i < 6; i++) threadList.add(new Thread(new NumberedThread(i)));
ExecutorService executor = Executors.newFixedThreadPool(10);;
for (Thread thread : threadList) executor.execute(thread);
executor.shutdown();
try {
executor.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
}
catch (InterruptedException ignored) { }
}
You could also use a string object instead if you wanted to name the threads.
aldo is not modified by the tasks in the thread, but instead is modified in the main thread, here:
for (int i = 0; i < threadList.size(); i++) {
executor.execute(threadList.get(i));
//here...
aldo = aldo + 1;
}
Also, since you want a counter that can increase its value in several threads, then you may use an AtomicInteger rather than int.
Your code should look like this:
AtomicInteger aldo = new AtomicInteger(1);
for (int i = 0; i < 5; i++) {
executor.execute( () -> {
System.out.println("This is test. N: " + aldo.getAndIncrement());
});
}
I will run the program over very slow ssh connection. Will it slow down or block the
System.out.println();
on big loads of printing. So if it prints few gigabytes right into console, but my connection is slow - where undiplayed data will appear? what is the size of tty memory? If I will lose connection for a while - will it run still?
No. PrintWriter does not wait for confirmation of completion.
Will it block the program it tty will have latency
Java's console output is blocking, so potentially your code may block, especially when you writing a lot of data.
what is the size of tty memory?
I am pretty sure that it depends on your kernel, this old thread suggests that it was 4096 bytes at some moment:
I've looked in the kernel code (linux\drivers\char\serial.c) and there is a #define called SERIAL_XMIT_SIZE. At first I thought maybe I could change that but it seems that the transmit buffer is actually fixed to be a memory page (4k).
If I will lose connection for a while - will it run still?
Yes, and if there is no one connected to the tty, then it will run much faster, as it will be able to discard the data.
Also small test application that simulates your use-case.
Echo.java
import java.io.IOException;
public class Echo {
public static void main(String[] args) throws InterruptedException, IOException {
final byte[] data = new byte[Test.BODY_LENGTH + Test.END_MARKER.length];
int index = 0;
outer: while (true) {
data[index++] = (byte) System.in.read();
final int dataOffset = index - Test.END_MARKER.length;
if (dataOffset < 0) {
continue;
}
for (int i = 0; i < Test.END_MARKER.length; i++) {
if (data[dataOffset + i] != Test.END_MARKER[i]) {
continue outer;
}
}
System.out.print(new String(data, 0, index));
return;
}
}
}
Test.java
import java.io.File;
import java.io.IOException;
import java.util.concurrent.ThreadLocalRandom;
import java.util.concurrent.TimeUnit;
public class Test {
public static final byte[] END_MARKER = "$TERMINATE$".getBytes();
public static final int BODY_LENGTH = 1024768;
public static void main(String[] args) throws IOException, InterruptedException {
StringBuilder data = new StringBuilder();
for (int i = 0; i < BODY_LENGTH; i++) {
data.append((char) ('a' + ThreadLocalRandom.current().nextInt(('z' - 'a' + 1))));
}
final Process process = new ProcessBuilder("java", Test.class.getPackage().getName() + ".Echo")
.directory(new File("out/production/week 3")) // Change to your output directory
.start();
process.getOutputStream().write(data.toString().getBytes());
process.getOutputStream().write(END_MARKER);
process.getOutputStream().flush();
System.out.println("Written!");
final boolean exitedAfterWroteData = process.waitFor(5, TimeUnit.SECONDS);
System.out.println(exitedAfterWroteData ? "Complete" : "Running"); // Will print running after 5 seconds
int read = 0;
while (process.getInputStream().read() > -1) {
read++;
}
if (read != data.toString().getBytes().length + END_MARKER.length) {
throw new IllegalStateException("Expected echo to print exactly " + BODY_LENGTH + END_MARKER.length + " symbols!");
}
final boolean exitedAfterWeReadData = process.waitFor(50, TimeUnit.MILLISECONDS);
System.out.println(exitedAfterWeReadData ? "Complete" : "Running"); // Will print complete after a few milliseconds
}
}
I have applied two process critical section solution to two threads instead of processes. My code is:
class Main
{
static boolean flag[];
static int turn;
static int count;
synchronized static void print(char ch,int n)
{
int i;
System.out.println(ch);
for(i=0;i<n;i++){
System.out.println(i);
}
}
public static void main(String[] args) throws IOException
{
flag = new boolean[2];
flag[0] = flag[1] = false;
turn = 0;
count = 0;
ThreadLevelOne t1 = new ThreadLevelOne('a');
ThreadLevelTwo t2 = new ThreadLevelTwo('b');
t1.start();
t2.start();
}
static class ThreadLevelOne extends Thread{
private char ch;
public ThreadLevelOne(char ch){
this.ch = ch;
}
public void run(){
while(true)
{
flag[0] = true;
turn = 1;
while(flag[1] && turn == 1);
print(ch,3);
count++;
System.out.println("Counter is : " + count);
flag[0] = false;
}
}
}
static class ThreadLevelTwo extends Thread{
private char ch;
public ThreadLevelTwo(char ch){
this.ch = ch;
}
public void run()
{
while(true)
{
flag[1] = true;
turn = 0;
while(flag[0] && turn == 0);
print( ch, 4);
count++;
System.out.println("Counter is : " + count);
flag[1] = false;
}
}
}
}
On executing the above code, it does not run infinitely but halts at arbitrary counter value on each execution. Is this a valid application of the two process solution to threads? If yes, then why is program halting at arbitrary counter value? If no, then how can this be achieved in threads?
Edit after the answer of codeBlind:
output: Program execution halts at this stage
Even if i dont increment the counter value, then also the program halts after a certain time
You're a victim of concurrently executing non-atomic operations, specifically count++, as well as the way you are using flags in each thread. But for simplicity's sake, let's talk about count++. The ++ operator actually executes three commands, each in their own clock-cycle:
read value of count
add 1 to value retrieved from count
store new value into count
The problem you're seeing is a result of these commands being interleaved across two threads. Thread A may not have stored the new count value by the time that Thread B attempts to read it.
A quick fix would be to use AtomicInteger for count instead of primitive int - AtomicInteger guarantees thread safety for integer operations.
EDIT
There are other race conditions in this code as well. Each thread's while loop argument (e.g. flag[0] && turn == 0) is non-atomic, but both threads are capable of modifying turn. You've left open the possibility that one thread could set turn before the other thread's while argument is fully evaluated, causing your threads to deadlock down the road.
If you only wish to guarantee that each thread must not be inside the while loop while the other thread is, then you should instead write each of your while loops to look something like this:
while(true){
synchronized(Main.class){
print( ch, 4);
count++;
System.out.println("Counter is : " + count);
}
}
If you want to guarantee that each thread must "take turns", you should look into using wait() and notify().
Ok so I figured it out, the issue is that each thread needs to pause in order for the other thread to run.
Instead of just spinning the cpu using:
while(flag[0] && turn == 0);
You need to pause the thread by calling the sleep method.
while(flag[0] && turn == 0){
try {
this.sleep(100);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
I have some thread-related questions, assuming the following code. Please ignore the possible inefficiency of the code, I'm only interested in the thread part.
//code without thread use
public static int getNextPrime(int from) {
int nextPrime = from+1;
boolean superPrime = false;
while(!superPrime) {
boolean prime = true;
for(int i = 2;i < nextPrime;i++) {
if(nextPrime % i == 0) {
prime = false;
}
}
if(prime) {
superPrime = true;
} else {
nextPrime++;
}
}
return nextPrime;
}
public static void main(String[] args) {
int primeStart = 5;
ArrayList list = new ArrayList();
for(int i = 0;i < 10000;i++) {
list.add(primeStart);
primeStart = getNextPrime(primeStart);
}
}
If I'm running the code like this and it takes about 56 seconds. If, however, I have the following code (as an alternative):
public class PrimeRunnable implements Runnable {
private int from;
private int lastPrime;
public PrimeRunnable(int from) {
this.from = from;
}
public boolean isPrime(int number) {
for(int i = 2;i < from;i++) {
if((number % i) == 0) {
return false;
}
}
lastPrime = number;
return true;
}
public int getLastPrime() {
return lastPrime;
}
public void run() {
while(!isPrime(++from))
;
}
}
public static void main(String[] args) {
int primeStart = 5;
ArrayList list = new ArrayList();
for(int i = 0;i < 10000;i++) {
PrimeRunnable pr = new PrimeRunnable(primeStart);
Thread t = new Thread(pr);
t.start();
t.join();
primeStart = pr.getLastPrime();
list.add(primeStart);
}
}
The whole operation takes about 7 seconds. I am almost certain that even though I only create one thread at a time, a thread doesn't always finish when another is created. Is that right? I am also curious: why is the operation ending so fast?
When I'm joining a thread, do other threads keep running in the background, or is the joined thread the only one that's running?
By putting the join() in the loop, you're starting a thread, then waiting for that thread to stop before running the next one. I think you probably want something more like this:
public static void main(String[] args) {
int primeStart = 5;
// Make thread-safe list for adding results to
List list = Collections.synchronizedList(new ArrayList());
// Pull thread pool count out into a value so you can easily change it
int threadCount = 10000;
Thread[] threads = new Thread[threadCount];
// Start all threads
for(int i = 0;i < threadCount;i++) {
// Pass list to each Runnable here
// Also, I added +i here as I think the intention is
// to test 10000 possible numbers>5 for primeness -
// was testing 5 in all loops
PrimeRunnable pr = new PrimeRunnable(primeStart+i, list);
Thread[i] threads = new Thread(pr);
threads[i].start(); // thread is now running in parallel
}
// All threads now running in parallel
// Then wait for all threads to complete
for(int i=0; i<threadCount; i++) {
threads[i].join();
}
}
By the way pr.getLastPrime() will return 0 in the case of no prime, so you might want to filter that out before adding it to your list. The PrimeRunnable has to absorb the work of adding to the final results list. Also, I think PrimeRunnable was actually broken by still having incrementing code in it. I think this is fixed, but I'm not actually compiling this.
public class PrimeRunnable implements Runnable {
private int from;
private List results; // shared but thread-safe
public PrimeRunnable(int from, List results) {
this.from = from;
this.results = results;
}
public void isPrime(int number) {
for(int i = 2;i < from;i++) {
if((number % i) == 0) {
return;
}
}
// found prime, add to shared results
this.results.add(number);
}
public void run() {
isPrime(from); // don't increment, just check one number
}
}
Running 10000 threads in parallel is not a good idea. It's a much better idea to create a reasonably sized fixed thread pool and have them pull work from a shared queue. Basically every worker pulls tasks from the same queue, works on them and saves the results somewhere. The closest port of this with Java 5+ is to use an ExecutorService backed by a thread pool. You could also use a CompletionService which combines an ExecutorService with a result queue.
An ExecutorService version would look like:
public static void main(String[] args) {
int primeStart = 5;
// Make thread-safe list for adding results to
List list = Collections.synchronizedList(new ArrayList());
int threadCount = 16; // Experiment with this to find best on your machine
ExecutorService exec = Executors.newFixedThreadPool(threadCount);
int workCount = 10000; // See how # of work is now separate from # of threads?
for(int i = 0;i < workCount;i++) {
// submit work to the svc for execution across the thread pool
exec.execute(new PrimeRunnable(primeStart+i, list));
}
// Wait for all tasks to be done or timeout to go off
exec.awaitTermination(1, TimeUnit.DAYS);
}
Hope that gave you some ideas. And I hope the last example seemed a lot better than the first.
You can test this better by making the exact code in your first example run with threads. Sub your main method with this:
private static int currentPrime;
public static void main(String[] args) throws InterruptedException {
for (currentPrime = 0; currentPrime < 10000; currentPrime++) {
Thread t = new Thread(new Runnable() {
public void run() {
getNextPrime(currentPrime);
}});
t.run();
t.join();
}
}
This will run in the same time as the original.
To answer your "join" question: yes, other threads can be running in the background when you use "join", but in this particular case you will only have one active thread at a time, because you are blocking the creation of new threads until the last thread is done executing.
JesperE is right, but I don't believe in only giving hints (at least outside a classroom):
Note this loop in the non-threaded version:
for(int i = 2;i < nextPrime;i++) {
if(nextPrime % i == 0) {
prime = false;
}
}
As opposed to this in the threaded version:
for(int i = 2;i < from;i++) {
if((number % i) == 0) {
return false;
}
}
The first loop will always run completely through, while the second will exit early if it finds a divisor.
You could make the first loop also exit early by adding a break statement like this:
for(int i = 2;i < nextPrime;i++) {
if(nextPrime % i == 0) {
prime = false;
break;
}
}
Read your code carefully. The two cases aren't doing the same thing, and it has nothing to do with threads.
When you join a thread, other threads will run in the background, yes.
Running a test, the second one doesn't seem to take 9 seconds--in fact, it takes at least as long as the first (which is to be expected, threding can't help the way it's implemented in your example.
Thread.join will only return when the thread.joined terminates, then the current thread will continue, the one you called join on will be dead.
For a quick reference--think threading when starting one iteration does not depend on the result of the previous one.