I have 2 java processes, Process1 is responsible for importing some external data to the database, Process2 is running the rest of the application using the same database, i.e. it hosts the web module the everything else. Process1 would normally import data once a day.
What I require is when Process1 has finished it's work it should notify the Process2 about it, so that it can perform some subsequent tasks. That is it, this will be their limit of interaction with each other. No other data has to be shared later.
No I know I can do this in one of the following ways:
Have the Process1 write an entry in the database when it has finished its execution and have a demon thread in Process2 looking for that entry. Once this entry is read, complete the task in Process2. Even though this might be the easiest to implement in the existing ecosystem, I think having a thread loop the database just for one notification looks kind of ugly. However, it could be optimised by starting the thread only when the import job starts and killing it after the notification is received.
Use a socket. I have never worked with sockets before, so this might be an interesting learning curve. But after my initial readings I am afraid it might be an overkill.
Use RMI
I would like to hear from people who have worked on similar problems, and what approach they choose and why and also would like to know what will be an appropriate solution for my problem.
Edit.
I went through this but found that for a starter in interprocess communication it lacks basic examples. That is what I am looking in this post.
I would say take a look at Chronicle-Queue
It uses a memory mapped file and saves data off-heap (so no problem with GC). Also, Provides TCP replication for failover scenarios.
It scales pretty well and supports distributed processing when more than one machine is available.
Related
I have troubles with understanding how I can work with Vowpal Wabbit (in this context, it doesn't matter. It can be something like one).
There are several steps of working with this program:
prepare data
train a model
?
profit
What step will be number 3?
I have found 2 ways of working with vowpal from my java app.
One of them is creating a vowpal process with necessary parameters like a path to a created model. But there is a problem. I am not sure, that this approach is good in a concurrent environment. Of course, I can run a stuck of processes for each thread, but it will be not ok.
Another way is running a vowpal daemon and connect to it via a socket connection. I see problems here too. For instance, I have to create a socket connection and send some data to a daemon. Then, I have to wait for a result, but I don't know when a result will be ready. Also, when I will receive data from a daemon, I don't know which chunk of data is the last. A result of work is just a string and its format doesn't let me process an output stream correctly.
Maybe are there other ways of working with vowpal wabbit, which are more productive and more reliable?
I have a Java application named 'X'. In Windows environment, at a given point of time there might be more than one instance of the application.
I want a common piece of code to be executed sequentially in the Application 'X' no matter how many instances of the application are running. Is that something possible and can be achieved ? Any suggestions will help.
Example :- I have a class named Executor where a method execute() will be invoked. Assuming there might be two or more instances of the application at any given point of time, how can i have the method execute() run sequential from different instances ?
Is there something like a lock which can be accessed from two instances and see if the lock is currently active or not ? Any help ?
I think what you are looking for is a distributed lock (i.e. a lock which is visible and controllable from many processes). There are quite a few 3rd party libraries that have been developed with this in mind and some of them are discussed on this page.
Distributed Lock Service
There are also some other suggestions in this post which use a file on the underlying system as a synchornization mechanism.
Cross process synchronization in Java
To my knowledge, you cannot do this that easily. You could implement TCP calls between processes... but well I wouldn't advice it.
You should better create an external process in charge of executing the task and a request all the the tasks to execute by sending a message to a JMS queue that your executor process would consume.
...Or maybe you don't really need to have several processes running in the same time but what you might require is just an application that would have several threads performing things in the same time and having one thread dedicated to the Executor. That way, synchronizing the execute()method (or the whole Executor) would be enough and spare you some time.
You cannot achieve this with Executors or anything like that because Java virtual machines will be separate.
If you really need to synchronize between multiple independent instances, one of the approaches would be to dedicate internal port and implement a simple internal server within the application. Look into ServerSocket or RMI is full blown solution if you need extensive communications. First instance binds to the dedicated application port and becomes the master node. All later instances find the application port taken but then can use it to make HTTP (or just TCP/IP) call to the master node reporting about activities they need to do.
As you only need to execute some action sequentially, any slave node may ask master to do this rather than executing itself.
A potential problem with this approach is that if the user shuts down the master node, it may be complex to implement approach how another running node could take its place. If only one node is active at any time (receiving input from the user), it may take a role of the master node after discovering that the master is not responding and then the port is not occupied.
A distributed queue, could be used for this type of load-balancing. You put one or more 'request messages' into a queue, and the next available consumer application picks it up and processes it. Each such request message could describe your task to process.
This type of queue could be implemented as JMS queue (e.g. using ActiveMQ http://activemq.apache.org/), or on Windows there is also MSMQ: https://msdn.microsoft.com/en-us/library/ms711472(v=vs.85).aspx.
If performance is an issue and you can have C/C++ develepors, also the 'shared memory queue' could be interesting: shmemq API
I;m looking for a very basis IPC mechanism between Java programs. I prefer not to make use of sockets because my 'agent' is spawning new JVM's and setting up sockets in such an environments is a bit more complicated.
I was thinking about having 2 files per spawned JVM: in and out. On the in, the agent sends commands to the worker. And on the out, the worker sends back a response back to the agent.
The big problem is that till so far I didn't manage to get the communication up and running. Just creating ObjectOutputStream/ObjectInputStream doesn't work out of the box, because the readObject method isn't blocking. It will throw an EOFException when there is no content instead instead of blocking. Luckily that was easy to fix, by adding a delay and trying again a bit later.
So I got my POC up and running, but eventually I ran into a stream corruption issue. So apparently, even in append only mode, you still can run into corruption issue. So I started to look at the FileLock, but I'm running now into a ""main" java.lang.Error: java.io.IOException: Bad file descriptor".
So till so far the 'lets do the simple file thing' has been quite an undertaking and I'm not sure if I'm in the right path at all. I don't want to introduce a heavy weight solution like JMS or a less heavyweight solution like sockets. Does anyone know something extremely simple that solves this particular problem? My preference is still for a file based approach.
I am designing a server application, that is supposed to crunch a lot of data continuously and present results on demand using web interface.
The operating scheme goes roughly like this:
An electronic sensor array constantly spills data into ramdisk through USB
A "flusher" application processes data as fast as it can and loads it into db (staging area)
Using triggers, db performs calculations on data and stores results in another schema (data area)
Client webapp can display processed data in graphs/reports etc. on demand
The solution would ideally look like this:
Database server - PostgreSQL
Have an administration web interface, that can monitor the flusher (i.e. records processed per hour or something like that) and if implemented as separate daemon, control it.
Flusher and Client applications written in Java, ideally using J2EE
Now the problem that keeps bugging me and I can't find the answer: How to go about writing the flusher component, i.e. a process that constantly runs in background in J2EE.
By scouring the web, basically three possibilities emerged:
a) Write the flusher as message driven bean and control it from master application using JMS. However: I don't like the idea of having a MDB running constantly, I'm not even sure that that's possible
b) Write the flusher as EJB and control it using Timer/Scheduling service. However: the events are not really timed, it just needs to run in infinite loop until told not to do so, just seems wrong usage of the technology.
c) Write the flusher as separate java application, run it as OS service (Linux or Windows) and control using startup scripts through ProcessBuilder invoked from EJB. To monitor it's status, use JMS. However: this just seems to me as overly complicated solution, platform dependent and maybe even unreliable and as EJB should not spawn/manage it's own threads, which ProcessBuilder basically does, it just seem wrong.
Basically, none of these look right to me and I cannot figure out, what would we the right solution in the Java/J2EE world.
Thank you
Thomas
I would write the "Flusher" app as a stand alone Java process. Perhaps use something like Java Service Wrapper to turn it into a service for your OS. I'm not very familiar with the options for interfacing with a RAM disk via Java, but you're either going to end up with an InputStream which you can keep open for the life of the process and continually read from, or you're going to continually poll from inside a while loop. It's perfectly ok to do something like the following:
private volotile boolean stopFlag;
...
while(!stopFlag) {
processNextInput();
}
Then you would have some other mechanism in another thread that could set stopFlag to true when you wanted to terminate the process.
As for monitoring the flusher JMX seems like a good solution. That's exactly what it was intended for. You would create an MBean that would expose any kind of status or statistics you wanted and then other processes could connect to that MBean and query for that data.
The "Client" app would then be a simple servlet application which does reporting on your database and provides a pretty front end for the MBean from your flusher. Alternatively you could just monitor the flusher using a JMX console and not even involve the client with that piece of the system.
I don't think EJBs really make sense for this system. I'm somewhat biased against EJBs, so take my advice with a grain of salt, but to me I don't really see a need for them in this application.
I'm trying to develop an application that just before quit has to run a new daemon process to execute the main method of a class.
I require that after the main application quits the daemon process must still be in execution.
It is a Java Stored Procedure running on Oracle DB so I can't use Runtime.exec because I can't locate the java class from the Operating System Shell because it's defined in database structures instead of file system files.
In particular the desired behavior should be that during a remote database session I should be able to
call the first java method that runs the daemon process and quits leaving the daemon process in execution state
and then (having the daemon process up and the session control, because the last call terminated) consequentially
call a method that communicates with the daemon process (that finally quits at the end of the communication)
Is this possible?
Thanks
Update
My exact need is to create and load (reaching the best performances) a big text file into the database supposing that the host doesn't have file transfer services from a Java JDK6 client application connecting to Oracle 11gR1 DB using JDBC-11G oci driver.
I already developed a working solution by calling a procedure that stores into a file the LOB(large database object) given as input, but such a method uses too many intermediate structures that I want to avoid.
So I thought about creating a ServerSocket on the DB with a first call and later connect to it and establish the data transfer with a direct and fast communication.
The problem I encountered comes out because the java procedure that creates the ServerSocket can't quit and leave an executing Thread/Process listening on that Socket and the client, to be sure that the ServerSocket has been created, can't run a separate Thread to handle the rest of the job.
Hope to be clear
I'd be surprised if this was possible. In effect you'd be able to saturate the DB Server machine with an indefinite number of daemon processes.
If such a thing is possible the technique is likely to be Oracle-specific.
Perhaps you could achieve your desired effect using database triggers, or other such event driven Database capabilities.
I'd recommend explaining the exact problem you are trying to solve, why do you need a daemon? My instict is that trying to manage your daemon's life is going to get horribly complex. You may well need to deal with problems such as preventing two instances being launched, unexpected termination of the daemon, taking daemon down when maintenance is needed. This sort of stuff can get really messy.
If, for example, you want to run some Java code every hour then almost certanly there are simpler ways to achieve that effect. Operating systems and databases tend to have nice methods for initiating work at desired times. So having a stored procedure called when you need it is probably a capability already present in your environment. Hence all you need to do is put your desired code in the stored procedure. No need for you to hand craft the shceduling, initiation and management. One quite significant advantage of this approach is that you end up using a tehcnique that other folks in your environment already understand.
Writing the kind of code you're considering is very intersting and great fun, but in commercial environments is often a waste of effort.
Make another jar for your other Main class and within your main application call the jar using the Runtime.getRuntime().exec() method which should run an external program (another JVM) running your other Main class.
The way you start subprocesses in Java is Runtime.exec() (or its more convenient wrapper, ProcessBuilder). If that doesn't work, you're SOL unless you can use native code to implement equivalent functionality (ask another question here to learn how to start subprocesses at the C++ level) but that would be at least as error-prone as using the standard methods.
I'd be startled if an application server like Oracle allowed you access to either the functionality of starting subprocesses or of loading native code; both can cause tremendous mischief so untrusted code is barred from them. Looking over your edit, your best approach is going to be to rethink how you tackle your real problem, e.g., by using NIO to manage the sockets in a more efficient fashion (and try to not create extra files on disk; you'll just have to put in extra elaborate code to clean them up…)