I;m looking for a very basis IPC mechanism between Java programs. I prefer not to make use of sockets because my 'agent' is spawning new JVM's and setting up sockets in such an environments is a bit more complicated.
I was thinking about having 2 files per spawned JVM: in and out. On the in, the agent sends commands to the worker. And on the out, the worker sends back a response back to the agent.
The big problem is that till so far I didn't manage to get the communication up and running. Just creating ObjectOutputStream/ObjectInputStream doesn't work out of the box, because the readObject method isn't blocking. It will throw an EOFException when there is no content instead instead of blocking. Luckily that was easy to fix, by adding a delay and trying again a bit later.
So I got my POC up and running, but eventually I ran into a stream corruption issue. So apparently, even in append only mode, you still can run into corruption issue. So I started to look at the FileLock, but I'm running now into a ""main" java.lang.Error: java.io.IOException: Bad file descriptor".
So till so far the 'lets do the simple file thing' has been quite an undertaking and I'm not sure if I'm in the right path at all. I don't want to introduce a heavy weight solution like JMS or a less heavyweight solution like sockets. Does anyone know something extremely simple that solves this particular problem? My preference is still for a file based approach.
Related
Can I base on "started" text in its STDOUT? Is there a standard way for this?
I'm basically trying to find a way to know it without pinging the WebApp and without having a timeout. I'm trying to find a reliable way.
If you can't modify the webapp, and it doesn't give you any other notification, then there's going to be no better way than pinging it and seeing if it's up every second or so. It might feel a bit inelegant, but in reality there's nothing wrong with that approach. I'm assuming here that it's going to start up within a few seconds - if it's going to be in the order of minutes / hours before it's up, then it might be worth combining this with reading the "started" text of stdout (though this approach seems a bit more fragile.)
If you could modify it (or build a wrapper around it), and don't like the "keep pinging it" approach, then the only other way would be to use some kind of inter process communication (such as over a localhost socket) so that you could connect to it, and it could push out a message to you when it had spun up. Worth pointing out though that in most scenarios, the additional effort of this approach won't be warranted.
I have 2 java processes, Process1 is responsible for importing some external data to the database, Process2 is running the rest of the application using the same database, i.e. it hosts the web module the everything else. Process1 would normally import data once a day.
What I require is when Process1 has finished it's work it should notify the Process2 about it, so that it can perform some subsequent tasks. That is it, this will be their limit of interaction with each other. No other data has to be shared later.
No I know I can do this in one of the following ways:
Have the Process1 write an entry in the database when it has finished its execution and have a demon thread in Process2 looking for that entry. Once this entry is read, complete the task in Process2. Even though this might be the easiest to implement in the existing ecosystem, I think having a thread loop the database just for one notification looks kind of ugly. However, it could be optimised by starting the thread only when the import job starts and killing it after the notification is received.
Use a socket. I have never worked with sockets before, so this might be an interesting learning curve. But after my initial readings I am afraid it might be an overkill.
Use RMI
I would like to hear from people who have worked on similar problems, and what approach they choose and why and also would like to know what will be an appropriate solution for my problem.
Edit.
I went through this but found that for a starter in interprocess communication it lacks basic examples. That is what I am looking in this post.
I would say take a look at Chronicle-Queue
It uses a memory mapped file and saves data off-heap (so no problem with GC). Also, Provides TCP replication for failover scenarios.
It scales pretty well and supports distributed processing when more than one machine is available.
I was just wondering if it's possible to dump a running Java program into a file, and later on restart it (same machine)
It's sounds a bit weird, but who knows
--- update -------
Yes, this is the hibernate feature for a process instead of a full system. But google 'hibernate jvm process' and you'll understand my pain.
There is a question for linux on this subject (here). Quickly, it's possible to hibernate a process (far from 100% reliable) with CryoPID.
A similar question was raised in stackoverflow some years ago.
With a JVM my educated guess is that hibernating should be a lot easier, not always possible and not reliable at 100% (e.g. UI and files).
Serializing a persistent state of the application is an option but it is not an answer to the question.
This may me a bit overkill but one thing you can do is run something like VirtualBox and halt/save the machine.
There is also:
- JavaFlow from Apache that should do just that even though I haven't personally tried
it.
- Brakes that may be exactly what you're looking for
There are a lot restrictions any solution to your problem will have: all external connections might or might not survive your attempt to freeze and awake them. Think of timeouts on the other side, or even stopped communication partners - anything from a web server to a database or even local files.
You are asking for a generic solution, without any internal knowledge of your program, that you would like to hibernate. What you can always do, is serialize that part of the state of your program, that you need to restart your program. It is, or at least was common wisdom to implement restart point in long running computations (think of days or weeks). So, when you hit a bug in your program after it run for a week, you could fix the bug and save some computation days.
The state of a program could be surprisingly small, compared to the complete memory size used.
You asked "if it's possible to dump a running Java program into a file, and later on restart it." - Yes it is, but I would not suggest a generic and automatic solution that has to handle your program as a black box, but I suggest that you externalize the important part of your programs state and program restart points.
Hope that helps - even if it's more complicated than what you might have hoped for.
I believe what the OP is asking is what the Smalltalk guys have been doing for decades - store the whole programming/execution environment in an image file, and work on it.
AFAIK there is no way to do the same thing in Java.
There has been some research in "persisting" the execution state of the JVM and then move it to another JVM and start it again. Saw something demonstrated once but don't remember which one. Don't think it has been standardized in the JVM specs though...
Found the presentation/demo I was thinking about, it was at OOPSLA 2005 that they were talking about squawk
Good luck!
Other links of interest:
Merpati
Aglets
M-JavaMPI
How about using SpringBatch framework?
As far as I understood from your question you need some reliable and resumable java task, if so, I believe that Spring Batch will do the magic, because you can split your task (job) to several steps while each step (and also the entire job) has its own execution context persisted to a storage you choose to work with.
In case of crash you can recover by analyzing previous run of specific job and resume it from exact point where the failure occurred.
You can also pause and restart your job programmatically if the job was configured as restartable and the ExecutionContext for this job already exists.
Good luck!
I believe :
1- the only generic way is to implement serialization.
2- a good way to restore a running system is OS virtualization
3- now you are asking something like single process serialization.
The problem are IOs.
Says your process uses a temporary file which gets deleted by the system after
'hybernation', but your program does not know it. You will have an IOException
somewhere.
So word is , if the program is not designed to be interrupted at random , it won't work.
Thats a risky and unmaintable solution so i believe only 1,2 make sense.
I guess IDE supports debugging in such a way. It is not impossible, though i don't know how. May be you will get details if you contact some eclipse or netbeans contributer.
First off you need to design your app to use the Memento pattern or any other pattern that allows you to save state of your application. Observer pattern may also be a possibility. Once your code is structured in a way that saving state is possible, you can use Java serialization to actually write out all the objects etc to a file rather than putting it in a DB.
Just by 2 cents.
What you want is impossible from the very nature of computer architecture.
Every Java program gets compiled into Java intermediate code and this code is then interpreted into into native platform code (when run). The native code is quite different from what you see in Java files, because it depends on underlining platform and JVM version. Every platform has different instruction set, memory management, driver system, etc... So imagine that you hibernated your program on Windows and then run it on Linux, Mac or any other device with JRE, such as mobile phone, car, card reader, etc... All hell would break loose.
You solution is to serialize every important object into files and then close the program gracefully. When "unhibernating", you deserialize these instances from these files and your program can continue. The number of "important" instances can be quite small, you only need to save the "business data", everything else can be reconstructed from these data. You can use Hibernate or any other ORM framework to automatize this serialization on top of a SQL database.
Probably Terracotta can this: http://www.terracotta.org
I am not sure but they are supporting server failures. If all servers stop, the process should saved to disk and wait I think.
Otherwise you should refactor your application to hold state explicitly. For example, if you implement something like runnable and make it Serializable, you will be able to save it.
I'm trying to develop an application that just before quit has to run a new daemon process to execute the main method of a class.
I require that after the main application quits the daemon process must still be in execution.
It is a Java Stored Procedure running on Oracle DB so I can't use Runtime.exec because I can't locate the java class from the Operating System Shell because it's defined in database structures instead of file system files.
In particular the desired behavior should be that during a remote database session I should be able to
call the first java method that runs the daemon process and quits leaving the daemon process in execution state
and then (having the daemon process up and the session control, because the last call terminated) consequentially
call a method that communicates with the daemon process (that finally quits at the end of the communication)
Is this possible?
Thanks
Update
My exact need is to create and load (reaching the best performances) a big text file into the database supposing that the host doesn't have file transfer services from a Java JDK6 client application connecting to Oracle 11gR1 DB using JDBC-11G oci driver.
I already developed a working solution by calling a procedure that stores into a file the LOB(large database object) given as input, but such a method uses too many intermediate structures that I want to avoid.
So I thought about creating a ServerSocket on the DB with a first call and later connect to it and establish the data transfer with a direct and fast communication.
The problem I encountered comes out because the java procedure that creates the ServerSocket can't quit and leave an executing Thread/Process listening on that Socket and the client, to be sure that the ServerSocket has been created, can't run a separate Thread to handle the rest of the job.
Hope to be clear
I'd be surprised if this was possible. In effect you'd be able to saturate the DB Server machine with an indefinite number of daemon processes.
If such a thing is possible the technique is likely to be Oracle-specific.
Perhaps you could achieve your desired effect using database triggers, or other such event driven Database capabilities.
I'd recommend explaining the exact problem you are trying to solve, why do you need a daemon? My instict is that trying to manage your daemon's life is going to get horribly complex. You may well need to deal with problems such as preventing two instances being launched, unexpected termination of the daemon, taking daemon down when maintenance is needed. This sort of stuff can get really messy.
If, for example, you want to run some Java code every hour then almost certanly there are simpler ways to achieve that effect. Operating systems and databases tend to have nice methods for initiating work at desired times. So having a stored procedure called when you need it is probably a capability already present in your environment. Hence all you need to do is put your desired code in the stored procedure. No need for you to hand craft the shceduling, initiation and management. One quite significant advantage of this approach is that you end up using a tehcnique that other folks in your environment already understand.
Writing the kind of code you're considering is very intersting and great fun, but in commercial environments is often a waste of effort.
Make another jar for your other Main class and within your main application call the jar using the Runtime.getRuntime().exec() method which should run an external program (another JVM) running your other Main class.
The way you start subprocesses in Java is Runtime.exec() (or its more convenient wrapper, ProcessBuilder). If that doesn't work, you're SOL unless you can use native code to implement equivalent functionality (ask another question here to learn how to start subprocesses at the C++ level) but that would be at least as error-prone as using the standard methods.
I'd be startled if an application server like Oracle allowed you access to either the functionality of starting subprocesses or of loading native code; both can cause tremendous mischief so untrusted code is barred from them. Looking over your edit, your best approach is going to be to rethink how you tackle your real problem, e.g., by using NIO to manage the sockets in a more efficient fashion (and try to not create extra files on disk; you'll just have to put in extra elaborate code to clean them up…)
I have a dotnet process that through calls to an unmanaged dll is communicating with a Java process.
Under some circumstances, the Java process appears to be crashing and taking my dotnet process down with it. No exceptions are raised, the process just dies. Upon crashing, java is creating a log file with names like "hs_err_pid3228" etc.
Not having received any satisfaction from the vendor that is providing the unmanaged dll and the java process, I am reduced to trying to mitigate the problem which would necessitate ensuring the calls into the java process, if they crash, don't take down my process.
Having read various articles, appdomains seem a likely candidate to use - my theory being I can with a bit of work separate my functionality that calls the java process and run it in a separate appdomain, which will hopefully allow me to if not catch the appdomain going down, at least detect that it has happened and restart that functionality.
Has anyone had a similar sort of issue? Does this approach seem reasonable to those of you with more experience of appdomain?
To make it even more fun, the Java crash is not really reproducible - it seems very random and I'm still battling with how I'm going to TEST that separating into the appdomain
This is a reasonable use of AppDomains, and what you propose will work.
In a similar vein, I once used AppDomains to create a single application that watched for itself crashing for exception reporting purposes. The application started itself up, created a new AppDomain, then re-executed itself in the new AppDomain, which then detected it was running in an AppDomain and executed normally. When an exception happened in that AppDomain, the original process is notified, it tears down the child domain reports to the user that an error occured, asks whether they want to report it or not, then picked itself up and tried it all over again.
EDIT: To give you a headstart, if you want to look at the Program.cs for that project, I've uploaded a stripped down version here. (It's pretty long, so I didn't think I should post it here.)
Yep, leveraging AppDomains make a lot of sense here.
I've recently reworked my Windows service to load its various WCF services as plug-ins that operate within their own AppDomain. I've got a few cases in the bootstrapping process where I'm using MarshalByRefObject objects to get things up and running, but once the plug-ins are loaded, communication between the AppDomains is extremely easy using WCF.