I have an application running on a cluster/grid where I need to run N tasks that do not have to communicate. I only need to collect the result of each task. So I have a Master distributing the tasks to some Slaves (possibly running on different hosts) and combining all the results at the end.
As the cluster is controlled by a batch system the configuration of my nodes changes for each run and I get a list of nodes that have been assigned to me for my job.
I'm looking for a library (pure Java) to help me with this. I looked at the following:
MPJ - doesn't work for me because of the way that MPJ runs when there are multiple processors available on the same machine. It uses custom class loaders and this gives me problems with a native library that I'm loading (it's loaded multiple times because the custom class loaders load the class multiple times).
Hazelcast - works in principle but it's not really made for this (I can distribute jobs with a queue and put the results back in another queue but it seems like a bit of an overkill). What I like is that it's easy to set up the group of nodes (in principle just one needs to be specified and the other nodes can just connect to it).
Simon/RMI - I guess I could let each slave register with the master and then let the master distribute jobs to each slave. Or let each slave request a queue where the jobs are queued and a queue where the results should be stored from the master.
Cajo - would in principle work but I don't want to have multicast on the grid network and there seems to be no way around this for Cajo.
RabbitMQ - I don't like to have an extra server running and it's not pure Java. Same for ZeroMQ.
Akka - Seems to be overkill as well. And a lot of configuration to set up the group of nodes.
Hadoop - Like Akka seems to be an overkill, especially the configuration to set up the group of nodes.
JPPF - Seems to be more suited for setting up a long running cluster of servers and nodes. After my application finishes I need to stop all servers and nodes. Also it seems to rely on Serialization of the Tasks which is not an option for me (see further below)
So I would stick with either Hazelcast or Simon. Which one is better suited for this kind of application? Does anyone know another library (not too heavy, not too much configuration). Any other suggestions?
Hazelcasts ExecutorService is not an option btw. because I'm using some JNI and so the serialization would be a pain.
I finally settled with MPJ. The problem with custom class loaders can simply be circumvented by not using the scripts included in MPJ but instead calling the java program directly with the following parameters:
java class rank mpj-config niodev [additional arguments for the application]
The rank, mpj-config and niodev arguments will be removed by the MPI_Init call.
mpj-config is a file listing number of ranks, a switching threshold for the message protocol and a list of hosts with corresponding port number and rank. niodev specifies the communication mechanism (see MPJ-Express documentation for more details). The config file could look like this:
3
131072
a6444#20000#0
a6444#20002#1
a6413#20000#2
It is important to seperate the port numbers on the same host by 2, because MPJ uses the specified port + the next one (so e.g. 20000 and 20001).
Simon and Hazelcast were also good solutions but they were a little bit slower than MPJ. Especially the initialization for both is quite a bit slower.
Let me know if this solution doesn't work.
Hazelcast provides a multi node task execution with Executor Service.
So you'll get the list of nodes that you want a task to be executed.
And then
HazelcastInstance h = Hazelcast.newHazelcastInstance();
Set<Member> members = h.getCluster().getMembers();//or any subset given your requirement
MultiTask<Long> multitask = new MultiTask<Long>(new MyCallableTask("default"), members);
ExecutorService es = h1.getExecutorService();
es.execute(multitask);
Collection<Long> results = multitask.get();
The only thing you need to do is to have the class of MyCallableTask in the classpath of all nodes.
Related
I have a Java application named 'X'. In Windows environment, at a given point of time there might be more than one instance of the application.
I want a common piece of code to be executed sequentially in the Application 'X' no matter how many instances of the application are running. Is that something possible and can be achieved ? Any suggestions will help.
Example :- I have a class named Executor where a method execute() will be invoked. Assuming there might be two or more instances of the application at any given point of time, how can i have the method execute() run sequential from different instances ?
Is there something like a lock which can be accessed from two instances and see if the lock is currently active or not ? Any help ?
I think what you are looking for is a distributed lock (i.e. a lock which is visible and controllable from many processes). There are quite a few 3rd party libraries that have been developed with this in mind and some of them are discussed on this page.
Distributed Lock Service
There are also some other suggestions in this post which use a file on the underlying system as a synchornization mechanism.
Cross process synchronization in Java
To my knowledge, you cannot do this that easily. You could implement TCP calls between processes... but well I wouldn't advice it.
You should better create an external process in charge of executing the task and a request all the the tasks to execute by sending a message to a JMS queue that your executor process would consume.
...Or maybe you don't really need to have several processes running in the same time but what you might require is just an application that would have several threads performing things in the same time and having one thread dedicated to the Executor. That way, synchronizing the execute()method (or the whole Executor) would be enough and spare you some time.
You cannot achieve this with Executors or anything like that because Java virtual machines will be separate.
If you really need to synchronize between multiple independent instances, one of the approaches would be to dedicate internal port and implement a simple internal server within the application. Look into ServerSocket or RMI is full blown solution if you need extensive communications. First instance binds to the dedicated application port and becomes the master node. All later instances find the application port taken but then can use it to make HTTP (or just TCP/IP) call to the master node reporting about activities they need to do.
As you only need to execute some action sequentially, any slave node may ask master to do this rather than executing itself.
A potential problem with this approach is that if the user shuts down the master node, it may be complex to implement approach how another running node could take its place. If only one node is active at any time (receiving input from the user), it may take a role of the master node after discovering that the master is not responding and then the port is not occupied.
A distributed queue, could be used for this type of load-balancing. You put one or more 'request messages' into a queue, and the next available consumer application picks it up and processes it. Each such request message could describe your task to process.
This type of queue could be implemented as JMS queue (e.g. using ActiveMQ http://activemq.apache.org/), or on Windows there is also MSMQ: https://msdn.microsoft.com/en-us/library/ms711472(v=vs.85).aspx.
If performance is an issue and you can have C/C++ develepors, also the 'shared memory queue' could be interesting: shmemq API
I have 2 java processes, Process1 is responsible for importing some external data to the database, Process2 is running the rest of the application using the same database, i.e. it hosts the web module the everything else. Process1 would normally import data once a day.
What I require is when Process1 has finished it's work it should notify the Process2 about it, so that it can perform some subsequent tasks. That is it, this will be their limit of interaction with each other. No other data has to be shared later.
No I know I can do this in one of the following ways:
Have the Process1 write an entry in the database when it has finished its execution and have a demon thread in Process2 looking for that entry. Once this entry is read, complete the task in Process2. Even though this might be the easiest to implement in the existing ecosystem, I think having a thread loop the database just for one notification looks kind of ugly. However, it could be optimised by starting the thread only when the import job starts and killing it after the notification is received.
Use a socket. I have never worked with sockets before, so this might be an interesting learning curve. But after my initial readings I am afraid it might be an overkill.
Use RMI
I would like to hear from people who have worked on similar problems, and what approach they choose and why and also would like to know what will be an appropriate solution for my problem.
Edit.
I went through this but found that for a starter in interprocess communication it lacks basic examples. That is what I am looking in this post.
I would say take a look at Chronicle-Queue
It uses a memory mapped file and saves data off-heap (so no problem with GC). Also, Provides TCP replication for failover scenarios.
It scales pretty well and supports distributed processing when more than one machine is available.
Sorry if the question is too open-ended or otherwise not suitable, but this is due to my lack of understanding about several pieces of technology/software, and I'm quite lost. I have a project where I have an existing java swing GUI, which runs MPI jobs on a local machine. However, it is desired to support running MPI jobs on HPC clusters (let's assume linux cluster with ssh access). To be more specific, the main backend executable (linux and windows) that I need to, erm, execute uses a very simple master-slave system where all relevant output is performed by the master node only. Currently, to run my backend executable on multiple machines, I would simply need to copy all necessary files to the machines (assuming no shared filespace) and call "mpiexec" or "mpirun" as is usual practice. The output produced by the master needs to be read in (or partially read in) by my GUI.
The main problem as I see things is this: Where to run the GUI? Several options:
Local machine - potential problem is needing to read data from cluster back to local machine (and also reading stdout/stderr of the cluster processes) to display current progress to user.
Login node - obvious problem of hogging precious resources, and in many cases will be banned.
Compute node - sounds pretty dodgy - especially if the cluster has a queuing system (slurm, sun grid, etc)! Also possibly banned.
Of these three options, the first seems the most reasonable, and also seems least likely to upset any HPC admin people, but is also the hardest to implement! There are multiple problems associated with that setup:
Passing data from cluster to local machine - because we're using a cluster - by definition we probably will generate large amounts of data, which the user wants to see at least part of! Also, how should this be done? I can see how to execute commands on remote machine via ssh using jsch or similar, but if i'm currently logged in on the remote machine - how do I communicate information back to the local machine?
Displaying stdout/stderr of backend in local machine. Similar to above.
Dealing with peculiar aspects of individual clusters - the only way I see around that is to allow the user to write custom slurm scripts or such like.
How to detect if backend computations have finished/failed - this problem interacts with any custom slurm scripts written by user.
Hopefully it should be clear from the above that I'm quite confused. I've had a look at apache camel, jsch, ganemede ssh, apache mina, netty, slurm, Sun Grid, open mpi, mpich, pmi, but there's so much information that I think I need to ask for some help and advice. I would greatly appreciate any comments regarding these problems!
Thanks
================================
Edit
Actually, I just came across this: link which seems to suggest that if the cluster allows an "interactive"-mode job, then you can run a GUI from a compute node. However, I don't know much about this, nor do I know if this is common. I would be grateful for comments on this aspect.
You may be able to leverage the approach shown here: a ProcessBuilder is used to execute a command in the background of a SwingWorker, while the command's output is displayed in a suitable component. In the example, ls -l would become ssh username#host 'ls -l'. Use JPasswordField as required.
I am designing a server application, that is supposed to crunch a lot of data continuously and present results on demand using web interface.
The operating scheme goes roughly like this:
An electronic sensor array constantly spills data into ramdisk through USB
A "flusher" application processes data as fast as it can and loads it into db (staging area)
Using triggers, db performs calculations on data and stores results in another schema (data area)
Client webapp can display processed data in graphs/reports etc. on demand
The solution would ideally look like this:
Database server - PostgreSQL
Have an administration web interface, that can monitor the flusher (i.e. records processed per hour or something like that) and if implemented as separate daemon, control it.
Flusher and Client applications written in Java, ideally using J2EE
Now the problem that keeps bugging me and I can't find the answer: How to go about writing the flusher component, i.e. a process that constantly runs in background in J2EE.
By scouring the web, basically three possibilities emerged:
a) Write the flusher as message driven bean and control it from master application using JMS. However: I don't like the idea of having a MDB running constantly, I'm not even sure that that's possible
b) Write the flusher as EJB and control it using Timer/Scheduling service. However: the events are not really timed, it just needs to run in infinite loop until told not to do so, just seems wrong usage of the technology.
c) Write the flusher as separate java application, run it as OS service (Linux or Windows) and control using startup scripts through ProcessBuilder invoked from EJB. To monitor it's status, use JMS. However: this just seems to me as overly complicated solution, platform dependent and maybe even unreliable and as EJB should not spawn/manage it's own threads, which ProcessBuilder basically does, it just seem wrong.
Basically, none of these look right to me and I cannot figure out, what would we the right solution in the Java/J2EE world.
Thank you
Thomas
I would write the "Flusher" app as a stand alone Java process. Perhaps use something like Java Service Wrapper to turn it into a service for your OS. I'm not very familiar with the options for interfacing with a RAM disk via Java, but you're either going to end up with an InputStream which you can keep open for the life of the process and continually read from, or you're going to continually poll from inside a while loop. It's perfectly ok to do something like the following:
private volotile boolean stopFlag;
...
while(!stopFlag) {
processNextInput();
}
Then you would have some other mechanism in another thread that could set stopFlag to true when you wanted to terminate the process.
As for monitoring the flusher JMX seems like a good solution. That's exactly what it was intended for. You would create an MBean that would expose any kind of status or statistics you wanted and then other processes could connect to that MBean and query for that data.
The "Client" app would then be a simple servlet application which does reporting on your database and provides a pretty front end for the MBean from your flusher. Alternatively you could just monitor the flusher using a JMX console and not even involve the client with that piece of the system.
I don't think EJBs really make sense for this system. I'm somewhat biased against EJBs, so take my advice with a grain of salt, but to me I don't really see a need for them in this application.
My team built a Java application using the Hadoop libraries to transform a bunch of input files into useful output.
Given the current load a single multicore server will do fine for the coming year or so. We do not (yet) have the need to go for a multiserver Hadoop cluster, yet we chose to start this project "being prepared".
When I run this app on the command-line (or in eclipse or netbeans) I have not yet been able to convince it to use more that one map and/or reduce thread at a time.
Given the fact that the tool is very CPU intensive this "single threadedness" is my current bottleneck.
When running it in the netbeans profiler I do see that the app starts several threads for various purposes, but only a single map/reduce is running at the same moment.
The input data consists of several input files so Hadoop should at least be able to run 1 thread per input file at the same time for the map phase.
What do I do to at least have 2 or even 4 active threads running (which should be possible for most of the processing time of this application)?
I'm expecting this to be something very silly that I've overlooked.
I just found this: https://issues.apache.org/jira/browse/MAPREDUCE-1367
This implements the feature I was looking for in Hadoop 0.21
It introduces the flag mapreduce.local.map.tasks.maximum to control it.
For now I've also found the solution described here in this question.
I'm not sure if I'm correct, but when you are running tasks in local mode, you can't have multiple mappers/reducers.
Anyway, to set maximum number of running mappers and reducers use configuration options mapred.tasktracker.map.tasks.maximum and mapred.tasktracker.reduce.tasks.maximum by default those options are set to 2, so I might be right.
Finally, if you want to be prepared for multinode cluster go straight with running this in fully-distributed way, but have all servers (namenode, datanode, tasktracker, jobtracker, ...) run on a single machine
Just for clarification...
If hadoop runs in local mode you don't have parallel execution on a task level (except you're running >= hadoop 0.21 (MAPREDUCE-1367)). Though you can submit multiple jobs at once and these getting executed in parallel then.
All those
mapred.tasktracker.{map|reduce}.tasks.maximum
properties do only apply to the hadoop running in distributed mode!
HTH
Joahnnes
According to this thread on the hadoop.core-user email list, you'll want to change the mapred.tasktracker.tasks.maximum setting to the max number of tasks you would like your machine to handle (which would be the number of cores).
This (and other properties you may want to configure) is also documented in the main documentation on how to setup your cluster/daemons.
What you want to do is run Hadoop in "pseudo-distributed" mode. One machine, but, running task trackers and name nodes as if it were a real cluster. Then it will (potentially) run several workers.
Note that if your input is small Hadoop will decide it's not worth parallelizing. You may have to coax it by changing its default split size.
In my experience, "typical" Hadoop jobs are I/O bound, sometimes memory-bound, way before they are CPU-bound. You may find it impossible to fully utilize all the cores on one machine for this reason.