I am using multiple threads to upload files to the server. The Java Applet is responsible for displaying the UI. Initially I start 5 threads using ThreadPoolExecutor & assign 5 files to them . After each upload, I get a notification from the server. When a thread completes execution , another new thread is assigned with a file until all the files are uploaded to the server.
Basic code structure as follows:
i> a method startUpload() is being called from the Java Applet which is responsible for handling the upload functionality.
class Upload extends Runnable{
...............................
..............................
public void startUpload() {
............................... //other initialisations done
int waitTime = 500;
Random random = new Random();
ExecutorService executor = new ThreadPoolExecutor(5, 5, 50000L, TimeUnit.MILLISECONDS, new ArrayBlockingQueue<Runnable>(300));
while (it.hasNext()) {
int time = random.nextInt(1000);
waitTime += time;
newFile = new File((String) it.next());
executor.execute(new Runnable() {
#Override
public void run() {
try{
Thread.sleep(wait);
}
catch(Exception e){
}
processFile1(newFile);
}
});
}
try {
Thread.sleep(waitTime);
executor.shutdown();
executor.awaitTermination(waitTime, TimeUnit.MILLISECONDS);
} catch (Exception e) {
}
}
}
The problem I am facing currently.
i> The UI is only updating at the end when all the files are upload. In the intermediate stage the UI is in a hanged state. It seems like the EDT is going to a blocked state.
The same code for UI rendering was working fine when I was using Thread class , notify/ sleep to implement the same functionality . I changed the code to ThreadPoolExecutor since I saw in a no of blogs/articles that its a better way of implementing multithreading from Java ver 5.0.
ii> Another thing which I noticed with the ThreadPoolExecutor , when I am uploading multiple files with size 1KB (for testing purpose) , if I remove all the wait() from the above code , the following line assigns a new file but the the same file is always being uploaded everytime by the multiple threads.
newFile = new File((String) it.next());
But on adding sleep() withing the run() , the multiple threads upload different files to the server.
Is there any implementation issue with the above code ?
Problem 1: newFile is a (static?) field instead of a local variable.
What you want is to make sure that the local capture of newFile is different each loop. As such, it should look more like:
while(it.hasNext()) {
final File newFile = new File((String) it.next());
executor.execute(new Runnable() {
#Override
public void run() {
processFile1(newFile); // Local only to this iteration of the loop.
}
}
}
Your code is all wrapped in a Runnable instance. Can you let us know what Thread this is called from? If it's on the EDT then that would explain why the UI locks up.
A small issue is the lack of generics on your iterator. In theory, you should be iterating over a collection of Strings:
Collection<String> listOfFiles = ...
Iterator<String> it = listOfFiles.iterator();
while(it.hasNext()) {
String filename = it.next(); // No cast necessary
}
The UI is hanging because you are blocking the EDT thread. This code is the culprit:
try {
Thread.sleep(waitTime);
executor.shutdown();
executor.awaitTermination(waitTime, TimeUnit.MILLISECONDS);
} catch (Exception e) {
}
The idea of an ExecutorService is that you create it one time during initialization and never shut it down until the program is ready to exit. An idiom for this might be:
ExecutorService executor = Executors.newFixedThreadPool(5);
Runtime.getRuntime().addShutdownHook(new Thread() {
public void run() {
executor.shutdown();
}
});
As #Bringer128 mentioned, the second problem is caused by the fact that you are changing the value of a static or member variable and not assigning the File reference to a new location. If the code were correct, we would expect to see the newFile declared as final File newFile because non-final local variables may not be referenced in an inner-class.
Related
I am trying to speed up a program that uses ExecutorService to run tasks in parallel. It basically works like this:
Initialize a fixed size thread pool of size n
Read a bunch (about 2500 files) of XML files containing input for the tasks
Process the XML files using the worker threads from the pool
Everything works as expected, but the problem I have is that each worker thread has an instance of a class that does calculations on the input data. This instance is stored in a ThreadLocal. Now, all these thread local instances are created when the correspondig worker thread is started, that means after all XML input files are read.
Since initialization of the calculation objects takes quite some time, I'd rather have the thread pool initialize all worker threads right from the start, so that the initializations of the calculation objects can run parallel to the reading of the input files.
Here is some code to help you get an idea about how it currently works (I stripped code not related to the problem).
Initializing the thread pool and the thread local:
private final ExecutorService executor = Executors.newFixedThreadPool(Math.max(1, Runtime
.getRuntime().availableProcessors() - 1));
private ThreadLocal<Calculator> calculator = new ThreadLocal<Calculator>() {
#Override
protected Calculator initialValue() {
try {
Calculator instance = createCalculator();
return instance;
} catch (Throwable e) {
throw new RuntimeException(e);
}
}
};
Submitting a new calculation:
#Override
public FutureTask<Output> calc(Input input) {
FutureTask<Output> task = new FutureTask<>(
new Callable<Rueckgabe>() {
#Override
public Output call() throws Exception {
try {
return calculator.get().calc(input);
} catch (Throwable e) {
System.err.println("Exception: " + e.getMessage());
e.printStackTrace(System.err);
return null;
}
}
});
executor.execute(task);
return task;
}
What is the right way to have the ExecutorService start all worker threads for the pool right from the start? Or do I have to something like n dummy tasks to force initialization?
PS: I have to use Java 7 for the foreseeable future due to IT restrictions.
Add a Threadfactory to the Executor which would do the initialization.
executor = Executors.newFixedThreadPool(numprocrssors, new ThreadFactory ( ){
public Thread newThread(Runnable r) {
return new Thread(r){
{calculator.get();} // this is an initialization statement, added to all constructors.
};
}});
currently, i'm trying to write a programm, which should execute a seperate Java-program multiple times, but with different parameters. This executed Java-program calls a Thread-Class. Within this class, a connection to a (Game)Server is established. Once connected, the Threads sends a command to turn the connected player around every 10 milliseconds. I have 2 "solutions" for this:
The easy (working) one:
public class GeneralAgentTest {
public static void main(String [] args){
Thread thread = new Thread(new HexagonRunner("127.0.0.1",6000,"UnitedTestors",-30,-15));
thread.start();
}
}
This is working correctly, but not actually my goal. I need to start several of this Threads (new Thread(new HexagonRunner("127.0.0.1",6000,"UnitedTestors",-30,-15)); ) and each of this threads must be handled by a seperate process.
To do this, i wrote some code with an ProcessBuilder. This is within one class.
Second not correctly working one:
public void execute(Class class1, int a, String str, String team, String x,
String y) {
ProcessBuilder builder;
String javaHome = System.getProperty("java.home");
String javaBin = javaHome + File.separator + "bin" + File.separator
+ "java";
String classpath = System.getProperty("java.class.path");
String className = class1.getCanonicalName();
builder = new ProcessBuilder(javaBin, "-cp", classpath,
className, ip, port, team, str, x, y);
try {
process[a] = builder.start();
} catch (Exception e) {
e.printStackTrace();
System.out.println("failed");
}
public void loadPlayers() {
process = new Process[4];
for (int i = 0; i < 4; i++) {
try {
execute(processes.StartProcesses.class, i,
playerCombo[i].getSelectedItem().toString(),
playerTeam[i].getText(), playerStartX[i].getText(),
playerStartY[i].getText());
} catch (Exception e) {
System.out.println("Failed to create processes for players");
e.printStackTrace();
}
}
}
These are the functions i wrote, to execute the class(es) who is/are starting the thread(s).
Following class is executed:
public class StartProcesses{
public static void main(String[] args) {
Thread t = null;
t = new Thread(new HexagonRunner("127.0.0.1",6000,"UnitedTestors",-30,-15));
t.start();
JOptionPane.showMessageDialog(null, "should've started");
}
}
In my second try, the parameters which are given to the StartProcesses Class are containing some information like IP-Adresses, Portnumbers, Playerpositons and stuff like this. Anyway i was trying to execute the class with "hard" information, just to be sure it is working like in my first codepart.
The connections to the server are correctly established in both attempts, but in the first one the thread keeps working. In my second try it seems like the thread is dead after the connection is established. The process is still alive, since the connection to the server is still there.
This is a bit of code, but what i want to tell, is that the Thread is working correctly when executed manually, but it is not working correctly if i try to start the class automatically with the use of ProcessBuilders.
I really really hope you guys could understand what i am trying to say. Hopefully someone has a working solution for me.
Cheers.
EDIT: Add Code for HexagonRunner:
public class HexagonRunner extends GeneralAgent {
// Bunch of Methods
// Important Method:
#Override
protected void simulationCycle() {
turnAgent(40);
}
}
The simulationCycle() method, is the method that is going to be go through over and over again.
Since the class HexagonRunner is inherited from the class GeneralAgent, i'm going to post the relevant stuff of this class here as well:
public class GeneralAgent implements Runnable, UdpListener{
// Attributes, getters, setters, methods..
#Override
public final void run() {
// giving the worker threads the signal to start
mServerConnection.start();
mParser.start();
// waiting for the first information to be parsed, so that the
// simulation loop can run
try{
System.out.println("GeneralAgent-Run: waiting for latch");
mLogger.info("Run method: waiting for CountDownLatch");
mFirstDone.await();
}
catch(InterruptedException e){
System.out.println("GeneralAgent-Run: InterruptedException");
mLogger.info("Run method error: " + e.getMessage());
}
mLogger.info("Run method: CountDownLatch now 0, continue");
// setting the initial position
moveToPostion(mXStartCoord, mYStartCoord);
// the simulation loop
while (true){
simulationCycle();
// Shutdown if the game is over
if (mGameworld.getTime() >= 6000){ // later 6000+
System.out.println("Yeah, 50 runs completed -> Shutdown");
mLogger.info("General Agent - Time > 50, programm should terminate");
shutdown();
break;
}
// waiting for the threads to signal that they are
// ready (e.g. have new server information)
// for another round of the simulation loop
mPhaser.arriveAndAwaitAdvance();
}
}
I hope things get clearer now. I still have no idea where my code fails.
You could build somethin a lot simpler by using Executors. It' part of the comcurrent package introduced in Java 1.5. It basically works as follows:
// create a pool with 10 threads
ExecutorService executorService = Executors.newFixedThreadPool(10);
//loop as long as you need to detach your threads
for (int i = 0; i < 4; i++) {
// this actually contains the thread bit, will be executed in parallel
executorService.execute(new Runnable() {
public void run() {
// this is where your code is
new HexagonRunner("127.0.0.1",6000,"UnitedTestors",-30,-15)
}
});
}
// clean up when you're done to prevent leaks
executorService.shutdown();
That's it, much simple and you don't need to spawn different JVMs through ProcessBuilder, which is a lot slower.
I have the following piece of code:
public class Test {
List<Future> future = new ArrayList<Future>();
public static void main(String args[]) throws Exception {
Adapter b1 = new Adapter();
final ExecutorService threadPool = Executors.newCachedThreadPool();
for(//iterate for number of files) {
while(data exists in file) {
//Call a function to process and update values in db
future.add(threadPool.submit(new Xyz(b1)));
//read next set of data in file;
}
}
try {
for(Future f: future) {
f.get();
}
}
catch(Exception e) {
throw e;
}
}
}
class Xyz implements Runnable {
private Adapter a1;
public Xyz(Adapter al) {
this.a1=a1;
}
#Override
public void run() {
try {
a1.abc();
} catch (Exception e) {
throw new RuntimeException(e);
}
}
}
When the number of files is 1 (for loop runs for 1 time), the code runs fine.
But, when the number of files increases, the code never returns back from future.get() method.
just out of curiosity.. do i need to shutdown the executor somewhere ??
Yes, and this is likely the problem. Each Future.get() will block until the corresponding task is complete, then once all the tasks are complete your main thread will exit. But your java process will not exit because the thread pool threads are still active in the background. You should shut down the executor once you have finished with it, most likely as the last thing in your main method.
I also note that you're submitting many tasks that wrap the same Adapter instance and all call its abc() method - check that there's nothing in there that will deadlock when called simultaneously in more than one thread.
Your Callable::call / Runable::run does not return. Otherwise the corresponding future would not block.
Additional executor.shutdown or future.cancel will thow an InterruptedException to stop the thread processing the object you submitted but it is up to you if to catch it or not. Your are responsible for making the jobs you submitted stop.
When you submit thousands Callables/Runnables to a CachedExecutor that it might spawn so many threads that your machine gets so slow that you think it takes forever. But you would have noticed that.
When dealing with an undefined number of parallelizable tasks i suggest to use a FixedThreadPool with not much more threads that there are cpu cores.
Edit: Therefore when you set a breakpoints at a1.abc(); and step forward you will probably find out that it never returns.
I am trying to figure out how to use the types from the java.util.concurrent package to parallelize processing of all the files in a directory.
I am familiar with the multiprocessing package in Python, which is very simple to use, so ideally I am looking for something similar:
public interface FictionalFunctor<T>{
void handle(T arg);
}
public class FictionalThreadPool {
public FictionalThreadPool(int threadCount){
...
}
public <T> FictionalThreadPoolMapResult<T> map(FictionalFunctor<T> functor, List<T> args){
// Executes the given functor on each and every arg from args in parallel. Returns, when
// all the parallel branches return.
// FictionalThreadPoolMapResult allows to abort the whole mapping process, at the least.
}
}
dir = getDirectoryToProcess();
pool = new FictionalThreadPool(10); // 10 threads in the pool
pool.map(new FictionalFunctor<File>(){
#Override
public void handle(File file){
// process the file
}
}, dir.listFiles());
I have a feeling that the types in java.util.concurrent allow me to do something similar, but I have absolutely no idea where to start.
Any ideas?
Thanks.
EDIT 1
Following the advices given in the answers, I have written something like this:
public void processAllFiles() throws IOException {
ExecutorService exec = Executors.newFixedThreadPool(6);
BlockingQueue<Runnable> tasks = new LinkedBlockingQueue<Runnable>(5); // Figured we can keep the contents of 6 files simultaneously.
exec.submit(new MyCoordinator(exec, tasks));
for (File file : dir.listFiles(getMyFilter()) {
try {
tasks.add(new MyTask(file));
} catch (IOException exc) {
System.err.println(String.format("Failed to read %s - %s", file.getName(), exc.getMessage()));
}
}
}
public class MyTask implements Runnable {
private final byte[] m_buffer;
private final String m_name;
public MyTask(File file) throws IOException {
m_name = file.getName();
m_buffer = Files.toByteArray(file);
}
#Override
public void run() {
// Process the file contents
}
}
private class MyCoordinator implements Runnable {
private final ExecutorService m_exec;
private final BlockingQueue<Runnable> m_tasks;
public MyCoordinator(ExecutorService exec, BlockingQueue<Runnable> tasks) {
m_exec = exec;
m_tasks = tasks;
}
#Override
public void run() {
while (true) {
Runnable task = m_tasks.remove();
m_exec.submit(task);
}
}
}
How I thought the code works is:
The files are read one after another.
A file contents are saved in a dedicated MyTask instance.
A blocking queue with the capacity of 5 to hold the tasks. I count on the ability of the server to keep the contents of at most 6 files at one time - 5 in the queue and another fully initialized task waiting to enter the queue.
A special MyCoordinator task fetches the file tasks from the queue and dispatches them to the same pool.
OK, so there is a bug - more than 6 tasks can be created. Some will be submitted, even though all the pool threads are busy. I've planned to solve it later.
The problem is that it does not work at all. The MyCoordinator thread blocks on the first remove - this is fine. But it never unblocks, even though new tasks were placed in the queue. Can anyone tell me what am I doing wrong?
The thread pool you are looking for is the ExecutorService class. You can create a fixed-size thread pool using newFixedThreadPool. This allows you to easily implement a producer-consumer pattern, with the pool encapsulating all the queue and worker functionality for you:
ExecutorService exec = Executors.newFixedThreadPool(10);
You can then submit tasks in the form of objects whose type implements Runnable (or Callable if you want to also get a result):
class ThreadTask implements Runnable {
public void run() {
// task code
}
}
...
exec.submit(new ThreadTask());
// alternatively, using an anonymous type
exec.submit(new Runnable() {
public void run() {
// task code
}
});
A big word of advice on processing multiple files in parallel: if you have a single mechanical disk holding the files it's wise to use a single thread to read them one-by-one and submit each file to a thread pool task as above, for processing. Do not do the actual reading in parallel as it will degrade performance.
A simpler solution than using ExecuterService is to implement your own producer-consumer scheme. Have a thread that create tasks and submits to a LinkedBlockingQueue or ArrayBlockingQueue and have worker threads that check this queue to retrieve the tasks and do them. You may need a special kind of tasks name ExitTask that forces the workers to exit. So at the end of the jobs if you have n workers you need to add n ExitTasks into the queue.
Basically, what #Tudor said, use an ExecutorService, but I wanted to expand on his code and I always feel strange editing other people's posts. Here's a sksleton of what you would submit to the ExecutorService:
public class MyFileTask implements Runnable {
final File fileToProcess;
public MyFileTask(File file) {
fileToProcess = file;
}
public void run() {
// your code goes here, e.g.
handle(fileToProcess);
// if you prefer, implement Callable instead
}
}
See also my blog post here for some more details if you get stuck
Since processing Files often leads to IOExceptions, I'd prefer a Callable (which can throw a checked Exception) to a Runnable, but YMMV.
I am using ThreadPoolExecutor for implmenting multithreading .
Basically each thread is assigned a file , which needs to be uploaded to the server.
After each successfull upload , a notification is issued from the server.
The following code , generates the thread & assigns the file to them.
Random random = new Random();
ExecutorService executor = new ThreadPoolExecutor(5, 5, 50000L, TimeUnit.MILLISECONDS, new ArrayBlockingQueue<Runnable>(400));
int waitTime = 100;
while (it.hasNext()) {
int time = random.nextInt(1000);
waitTime += time;
newFile = new File((String) it.next());
executor.execute(new Runnable() {
#Override
public void run() {
processFile(newFile);
}
});
try {
Thread.sleep(waitTime);
} catch (Exception e) {
}
}
try {
Thread.sleep(waitTime);
executor.shutdown();
executor.awaitTermination(waitTime, TimeUnit.MILLISECONDS);
} catch (InterruptedException e) {
}
I have created a Java Applet for rendering the UI.
After each notification , I am updating the file status on the Java Applet window where the updateUI() is called from the processFile().
Before using ExecutorService (recommended for handling multithreading on Java v5.0 & above) , I was using a Thread class for creating the threads & wait-notify for the file upload functionality . The UI displaying each file update status gets properly rendered on using the Thread class but on using the ExecutorService , all the files get uploaded(functionality working) but the UI hangs .
After each file is uploaded successfully , the file upload status for each of the files needs to be updated.
Any suggestions/hints welcome.
when wanting to update the UI from a non EDT thread (the event thread that calls the event handlers and such) you need to use SwingUtilities.invokeLater(Runnable)
and you should NEVER sleep on the EDT or block it as that is the one making sure everything is responsive
however it would be better to use a SwingWorker as this implements several things specific to using background threads that need to update the gui
with swingWorker your code will be
while (it.hasNext()) {
final File newFile = new File((String) it.next());
SwingWorker<Void,Void> work = new SwingWorker<Void,Void>(){
#Override
public Void doInBackground() {
processFile(newFile);//do **not** call updateUI() in here
return null;
}
#Override
protected void done(){//this gets called after processFile is done on the EDT
updateUI();
}
}).execute();
}