I am designing a server application, that is supposed to crunch a lot of data continuously and present results on demand using web interface.
The operating scheme goes roughly like this:
An electronic sensor array constantly spills data into ramdisk through USB
A "flusher" application processes data as fast as it can and loads it into db (staging area)
Using triggers, db performs calculations on data and stores results in another schema (data area)
Client webapp can display processed data in graphs/reports etc. on demand
The solution would ideally look like this:
Database server - PostgreSQL
Have an administration web interface, that can monitor the flusher (i.e. records processed per hour or something like that) and if implemented as separate daemon, control it.
Flusher and Client applications written in Java, ideally using J2EE
Now the problem that keeps bugging me and I can't find the answer: How to go about writing the flusher component, i.e. a process that constantly runs in background in J2EE.
By scouring the web, basically three possibilities emerged:
a) Write the flusher as message driven bean and control it from master application using JMS. However: I don't like the idea of having a MDB running constantly, I'm not even sure that that's possible
b) Write the flusher as EJB and control it using Timer/Scheduling service. However: the events are not really timed, it just needs to run in infinite loop until told not to do so, just seems wrong usage of the technology.
c) Write the flusher as separate java application, run it as OS service (Linux or Windows) and control using startup scripts through ProcessBuilder invoked from EJB. To monitor it's status, use JMS. However: this just seems to me as overly complicated solution, platform dependent and maybe even unreliable and as EJB should not spawn/manage it's own threads, which ProcessBuilder basically does, it just seem wrong.
Basically, none of these look right to me and I cannot figure out, what would we the right solution in the Java/J2EE world.
Thank you
Thomas
I would write the "Flusher" app as a stand alone Java process. Perhaps use something like Java Service Wrapper to turn it into a service for your OS. I'm not very familiar with the options for interfacing with a RAM disk via Java, but you're either going to end up with an InputStream which you can keep open for the life of the process and continually read from, or you're going to continually poll from inside a while loop. It's perfectly ok to do something like the following:
private volotile boolean stopFlag;
...
while(!stopFlag) {
processNextInput();
}
Then you would have some other mechanism in another thread that could set stopFlag to true when you wanted to terminate the process.
As for monitoring the flusher JMX seems like a good solution. That's exactly what it was intended for. You would create an MBean that would expose any kind of status or statistics you wanted and then other processes could connect to that MBean and query for that data.
The "Client" app would then be a simple servlet application which does reporting on your database and provides a pretty front end for the MBean from your flusher. Alternatively you could just monitor the flusher using a JMX console and not even involve the client with that piece of the system.
I don't think EJBs really make sense for this system. I'm somewhat biased against EJBs, so take my advice with a grain of salt, but to me I don't really see a need for them in this application.
Related
I have 2 java processes, Process1 is responsible for importing some external data to the database, Process2 is running the rest of the application using the same database, i.e. it hosts the web module the everything else. Process1 would normally import data once a day.
What I require is when Process1 has finished it's work it should notify the Process2 about it, so that it can perform some subsequent tasks. That is it, this will be their limit of interaction with each other. No other data has to be shared later.
No I know I can do this in one of the following ways:
Have the Process1 write an entry in the database when it has finished its execution and have a demon thread in Process2 looking for that entry. Once this entry is read, complete the task in Process2. Even though this might be the easiest to implement in the existing ecosystem, I think having a thread loop the database just for one notification looks kind of ugly. However, it could be optimised by starting the thread only when the import job starts and killing it after the notification is received.
Use a socket. I have never worked with sockets before, so this might be an interesting learning curve. But after my initial readings I am afraid it might be an overkill.
Use RMI
I would like to hear from people who have worked on similar problems, and what approach they choose and why and also would like to know what will be an appropriate solution for my problem.
Edit.
I went through this but found that for a starter in interprocess communication it lacks basic examples. That is what I am looking in this post.
I would say take a look at Chronicle-Queue
It uses a memory mapped file and saves data off-heap (so no problem with GC). Also, Provides TCP replication for failover scenarios.
It scales pretty well and supports distributed processing when more than one machine is available.
I have a JavaFX application, when the user closes the window, I want to destroy all of the JavaFX related resources and only have a tray icon, where the user can then reopen the application.
I have many background threads running, which should stay running when the GUI is closed. I have tried using Platform.exit() however it has no impact on the RAM usage of the program.
What is the best way to accomplish this? My goal is to reduce the impact on the system from my program as much as possible when the application is closed, but still running all of the background threads.
One option is to run the application as a separate process, launching the process when you want to create the application and exiting the process when the application is no longer needed (so completing a full application lifecycle). That way you will be absolutely sure that the application is not consuming any resources when it is not being used, because it won't be running.
How you would accomplish the launching and any communication between your tray service and the application would be up to you. You can research various mechanisms and, if you decide to go this route, ask some new follow up questions on accomplishing certain aspects of the task.
Some example routes you could look at are the ProcessBuilder, which is admittedly a pretty finicky and horrible API or the new Process API updates that will be available with Java 9. If wish to ensure at most a single instance of the application process is ever used, there are solutions for that. If you need to send a signal to the running application process, you could use something like RMI or run a basic HTTP REST server in your application and send messages into that.
As an aside, years ago there used to be some ongoing work on building multi-process JVMs, but there was never any wide uptake of the idea for Java. Though most modern browser clients, such as Chrome and Firefox, are multi-process architectures, the linked articles give some insight into this architecture, some of the potential implications of it and why it used for those applications.
Before going such a route, I would advise you to ensure that such an approach is truly necessary for your application (as pointed out by user npace in comments).
I am developing a server application in java (SE) + some open source libraries. This is a game, so I think it will have to be updated sometimes. As long as the app is tracking the state of the clients, using it, and also supports client sessions (in a form of dedicated threads), and singletons, which store references to these player instances, I would like to reload the JVM process (installing a new version of jar file for example), so that some of the classes and instances are not erased by GC, but hooked into the new process started.
Probable Situation:
I have a game version 1.0 running on the server. I need to add some new features in 1.2. But 100 players are fighting each other. I need to install version-1.2, players might feel some lag, but they should be able to continue with their fight even though the version of the jar has changed.
How can that be done?
Take a look at JRebel authors' nice summary on the topic:
http://zeroturnaround.com/blog/reloading_java_classes_401_hotswap_jrebel/#!/
You could look at things like OSGI and class-loading tricks and see if you can achieve what you need, however maybe there is a simpler approach which is a bit safer too.
You could have 2 servers running behind a simple load balancer and when you need to do such maintenance you start routing all new games to Server A only. When all the games on server B finish (and no new games have started because all are going to server A) you stop server B and do whatever maintenance you need to do together with any tests to verify everything is OK before you introduce it back into the 'cluster'.
You then do the same thing and route new games to server B, so that when all games on server A finish you cans switch that off too and do the maintenance.
You will need to develop this simple load balancer which just acts as a connection router but is aware of your game sessions. Maybe you also need to have some information synchronisation between each server instance (if there is any need of communicating lists of players online, chat messages etc., depends what features you have).
First thing to note is that your current game state might be inconsistent with new version. So not all modifications could be installed like this. You can use remote caching system with master/slave configuration. So that all session is stored on remote machine and while one server is stopped, second server can continue working with the same state. First you put down slave, update it and start. Then switch roles and put down second server for update.
I'm trying to develop an application that just before quit has to run a new daemon process to execute the main method of a class.
I require that after the main application quits the daemon process must still be in execution.
It is a Java Stored Procedure running on Oracle DB so I can't use Runtime.exec because I can't locate the java class from the Operating System Shell because it's defined in database structures instead of file system files.
In particular the desired behavior should be that during a remote database session I should be able to
call the first java method that runs the daemon process and quits leaving the daemon process in execution state
and then (having the daemon process up and the session control, because the last call terminated) consequentially
call a method that communicates with the daemon process (that finally quits at the end of the communication)
Is this possible?
Thanks
Update
My exact need is to create and load (reaching the best performances) a big text file into the database supposing that the host doesn't have file transfer services from a Java JDK6 client application connecting to Oracle 11gR1 DB using JDBC-11G oci driver.
I already developed a working solution by calling a procedure that stores into a file the LOB(large database object) given as input, but such a method uses too many intermediate structures that I want to avoid.
So I thought about creating a ServerSocket on the DB with a first call and later connect to it and establish the data transfer with a direct and fast communication.
The problem I encountered comes out because the java procedure that creates the ServerSocket can't quit and leave an executing Thread/Process listening on that Socket and the client, to be sure that the ServerSocket has been created, can't run a separate Thread to handle the rest of the job.
Hope to be clear
I'd be surprised if this was possible. In effect you'd be able to saturate the DB Server machine with an indefinite number of daemon processes.
If such a thing is possible the technique is likely to be Oracle-specific.
Perhaps you could achieve your desired effect using database triggers, or other such event driven Database capabilities.
I'd recommend explaining the exact problem you are trying to solve, why do you need a daemon? My instict is that trying to manage your daemon's life is going to get horribly complex. You may well need to deal with problems such as preventing two instances being launched, unexpected termination of the daemon, taking daemon down when maintenance is needed. This sort of stuff can get really messy.
If, for example, you want to run some Java code every hour then almost certanly there are simpler ways to achieve that effect. Operating systems and databases tend to have nice methods for initiating work at desired times. So having a stored procedure called when you need it is probably a capability already present in your environment. Hence all you need to do is put your desired code in the stored procedure. No need for you to hand craft the shceduling, initiation and management. One quite significant advantage of this approach is that you end up using a tehcnique that other folks in your environment already understand.
Writing the kind of code you're considering is very intersting and great fun, but in commercial environments is often a waste of effort.
Make another jar for your other Main class and within your main application call the jar using the Runtime.getRuntime().exec() method which should run an external program (another JVM) running your other Main class.
The way you start subprocesses in Java is Runtime.exec() (or its more convenient wrapper, ProcessBuilder). If that doesn't work, you're SOL unless you can use native code to implement equivalent functionality (ask another question here to learn how to start subprocesses at the C++ level) but that would be at least as error-prone as using the standard methods.
I'd be startled if an application server like Oracle allowed you access to either the functionality of starting subprocesses or of loading native code; both can cause tremendous mischief so untrusted code is barred from them. Looking over your edit, your best approach is going to be to rethink how you tackle your real problem, e.g., by using NIO to manage the sockets in a more efficient fashion (and try to not create extra files on disk; you'll just have to put in extra elaborate code to clean them up…)
We have a application that should always be running. Does anyone know of a way to create an automated way of monitoring to see if this application is running (possibly using a batch file)? If it is not running, then send an email notification and start the application?
Nagios is generally what's used by systems administrators that I've come across. You can script it to do whatever check you need and alert based on a variety of conditions. Works well with cacti so you can graph stuff too :)
If you want to ensure that your service always restarts should it die you could use supervise from daemontools.
Alternative to Nagios is zabbix
You don't mention an OS but if you're looking for something on Windows, Application Monitor might be a good start.
If you're on Linux, monit look pretty useful.
Most monitoring systems have a built-in test which watches the process list to check that everything that should be running is running.
We use Hobbit, it has a configurable table of processes which should be running (and the number of instances, red/yellow alert etc).
We are now heading to release our service that can do some monitoring tasks that usually are hard to handle by Nagios or other similar tools. We provide instant notifications (email, SMS) when:
a) your application/service does not respond for some time
b) some conditions are met (e.g. time of execution of some part of logic > X, number of emails sent < Y or whatever you want)
This is absoltely easy to use when compared to Nagios or others and it does not require installation. We spent a lot of time to make it user-friendly
As I mentioned this will be released very soon (will come back and give you the information). If you are interested in our approach we invite you to beta tests of our application (there will be some promotion for participants).