We Keep Coding

stop java execution if condition is met

So the idea is a kind of virtual classroom (a website) where students uploads uncompiled .java files, our server will compile it and execute it through C# or PHP, the language doesn't matter, creating a .bat file and get the feedback of the console if the program compiled correctly or not and if the execution was correct based on some pre-maded test, so far our tests did work but we have completely no control on what's inside the .java file so we want to stop the execution if some criterias did happen, i.e. an user input, infite loop, sockets instances, etc... I've digging on internet if there's a way to configure the java environment to avoid this but so far can't find anything, and we don't want our backend language to go through the file to check this things cause will be a completly mess up
Thanks for the help

You could configure a security manager, but it doesn't have a very good track record of stopping a determined attacker, and doesn't do resource limiting anyways.
You could load the untrusted code with a dedicated class loader that only sees white-listed classes.
Or you could use something like docker to isolate the process at the operating system level. This could also limit its cpu and memory consumption.
I'd probably combine these approaches, but some risk will remain in either case.
(Yes, I realize that is complex, but safely sandboxing arbitrary java code is a hard problem.)

Technology to transfer data with external system

We have an interface with an external system in which we get flat files from them and process those files. At present we run a job a few times a day that checks if the file is at the ftp location and then processes if it exists.
I recently read that it is a bad idea to make use of file systems as a message broker which is why I am putting in this question. Can someone clarify if a situation like this one is a right fitment for the use of some other tool and if so which one?
Ours is a java based application.

The first question you should ask is "is it working?".
If the answer to that is yes, then you should be circumspect about change just because you read it was a bad idea. I've read that chocolate may be bad for you but I'm not giving it up :-)
There are potential problems that you can run into, such as files being deleted without your knowledge, or trying to process files that are only half-transferred (though there are ways to mitigate both of those, such as permissions in the former case, or the use of sentinel files or content checking in the latter case).
Myself, I would prefer a message queueing system such as IBM's MQ or JMS (since that's what they're built for, and they do make life a little easier) but, as per the second paragraph above, only if either:
problems appear or become evident with the current solution; or
you have some spare time and money lying around for unnecessary rework.
The last bullet needs expansion. While the work may be unnecessary (in terms of fixing a non-existent problem), that doesn't necessarily make it useless, especially if it can improve performance or security, or reduce the maintenance effort.

I would use a database to synchronize your files. Have a database that points to the file locations. Put an entry into the database only when the files have been fully transferred. This would ensure that you are picking up completed files. You can poll the database to check if new entries are present instead of polling the file system. A very easy simple set up for a polling mechanism. If you would like to be told when a new file appears on the folder, then you would need to go in for a Message Queue.

How can two programs talk to each other in Java?

I want to ‪reduce‬ the CPU usage/ROM usage/RAM usage - generally‬, all system resources that my app uses - who doesn't? :)
For this reason I want to split the preferences window from the rest of the application,
and let the preferences window to run as ‪independent‬ program.
The preferences program ‪should‬ write to a Property file(not a problem at all) and to send a "update signal" to the main program - which means it should call the update method (that i wrote) that found in the Main class.
How can I call the update method in the Main program from the preferences program?
To put it another way, is a way to build preferences window that take system resources just when the window appears?
Is this approach - of separating programs and let them talk to each other (somehow) - the right approach for speeding up my programs?

What you're describing sounds like Premature Optimisation. If you're writing something other than a toy application, it's important to be confident that your optimisations are actually addressing a real problem. Is your program running slowly? If so, have you run it through a profiler or otherwise identified where the poor performance is happening?
If you have identified that what you want to do will address your performance issue, I suggest you look at running the components concurrently in different threads, not different processes. Then your components can avoid blocking each other, you will be able to take advantage of multi-core processors and you do not take on the complexity and performance overhead of inter-process communication over network sockets and the like.

You can communicate back and forth using sockets. Here's a tutorial of how to do something similar..
Unfortunately, I don't think this is going to help you minimize CPU usage, RAM, etc... If anything it might increase the CPU usage, RAM usage etc, because you need to run two JVM's instead of one. Unless you have some incredibly complicated preferences window, it is not likely taking that many resources that you need to worry about it. By adding the network communication, you are just adding more complexity without adding any benefit.
Edit:
If you have read the book Filthy Rich Clients, one of the main points of the book is that Rich Effects do not need to be resource intensive. Most of the book is devoted to showing how to add cool effects to an app with out taking a lot of resources. Throughout the book they are very careful to time everything to show what takes a long time and what doesn't. This is crucial when making your app less resource hungry. Write your app, see what feels slow, add timing code to those particular items that are slow, and speed up those particular parts of the code. Check with your timing code to see if it is actually faster. Rinse and repeat. Otherwise you are doing optimization that may not make any difference. Without timing your code you don't know if code needs to be sped up even if you've sped up the code after doing your optimizing.
Others have mentioned loading the properties window in a separate thread. It's important to remember that Swing has only one thread called the EDT that does all of the painting of pixels to the screen. Any code that causes pixels on the screen to change should be called from the EDT and thus should not be called from a separate thread. So, if you have something that may take a while to run (perhaps a web service call or some expensive computation), you would launch a separate thread off of the EDT, and when it finishes run code on the EDT to do the UI update. There are libraries such as SwingWorker to make this easier. If you are setting a dialog to be visible, this should not be on a separate thread, but it may make sense to build the data structures in a separate thread if it is time consuming to build these data structures.
Using Swing Worker is one of many valuable ideas in Filthy Rich Clients for making UI's feel more responsive. Using the ideas in this book I have taken some fairly resource intensive UI's and made them so the UI was hardly using any resources at all.

You could create a ServerSocket in the main window and have the preferences app connect to that with a regular Socket the protocol to use may be extremely simple, but... I think you should really look for the second approach: to build preferences window that take system resources just when it's appear?
To do that, you have to build the window and all it resources until the user performs the Preferences action, save your file ( or pass the content to the main app ) and dispose all the resources of the preference window by making all of its reference non accessible. The garbage collector will handle the rest.

Maybe you could use some sort of directory watcher like this or maybe implement some sort of semaphore.
Honestly, I think that you should be able to solve the problem if you have some sort of menu item that the user can access. Once that the user saves the preferences, these are written to a file. The application then loads the values from the file whenever it needs them.
If your system is operating slowly, or hanging, you might consider the use of threads, or increase the number of threads.

Actually, as others have explained, you can use socket for inter-process communication.
However, that won't reduce your overall CPU / RAM usage at all. (might even slightly worsen your resources usage)
For your case, you can launch the Perference window in a different Thread rather than a different Process.
Thread is lighter for OS to handle and poses no additional complexity for inter-process communications.

Nobody seems to have mentioned the DBUS - available to developers on a Linux system. I guess that's no good if you're trying to make a Windows/Cross Platform application, but the DBUS is a ready-made application-communication platform. It helps address issues such as:
Someone else might already be using the port you're trying. There's no way for you client application (The "Preferences" window I guess) to know whether the thing listening on that port is your main application, or just something else that happens to be there, so you'll have to do some sort of handshake, and implement a conflict-resolution mechanism
It's not going to be obvious to either the future you, or anyone who comes to maintain your app why you're on the port you are. This might not seem important, but communicating on Socket 5574 just doesn't seem as neat to me as communicating on channel org.yourorganisation.someapp .
Firewalls (as I think someone's already said) can be a little over-zealous
Also, it's worth getting your hand in on DBUS - it's useful for communicating with a whole bunch of other applications such as the little popup notification thing you'll find in recent Ubuntu distributions, or certain instant messaging clients, etc.
You can read up on what I'm talking about (and maybe correct me on some of the things I've said) here: http://www.freedesktop.org/wiki/Software/dbus . It looks like they're working on making it happen on Windows too, which is nice.

Is it likely appdomains will help in this scenario?

I have a dotnet process that through calls to an unmanaged dll is communicating with a Java process.
Under some circumstances, the Java process appears to be crashing and taking my dotnet process down with it. No exceptions are raised, the process just dies. Upon crashing, java is creating a log file with names like "hs_err_pid3228" etc.
Not having received any satisfaction from the vendor that is providing the unmanaged dll and the java process, I am reduced to trying to mitigate the problem which would necessitate ensuring the calls into the java process, if they crash, don't take down my process.
Having read various articles, appdomains seem a likely candidate to use - my theory being I can with a bit of work separate my functionality that calls the java process and run it in a separate appdomain, which will hopefully allow me to if not catch the appdomain going down, at least detect that it has happened and restart that functionality.
Has anyone had a similar sort of issue? Does this approach seem reasonable to those of you with more experience of appdomain?
To make it even more fun, the Java crash is not really reproducible - it seems very random and I'm still battling with how I'm going to TEST that separating into the appdomain

This is a reasonable use of AppDomains, and what you propose will work.
In a similar vein, I once used AppDomains to create a single application that watched for itself crashing for exception reporting purposes. The application started itself up, created a new AppDomain, then re-executed itself in the new AppDomain, which then detected it was running in an AppDomain and executed normally. When an exception happened in that AppDomain, the original process is notified, it tears down the child domain reports to the user that an error occured, asks whether they want to report it or not, then picked itself up and tried it all over again.
EDIT: To give you a headstart, if you want to look at the Program.cs for that project, I've uploaded a stripped down version here. (It's pretty long, so I didn't think I should post it here.)

Yep, leveraging AppDomains make a lot of sense here.
I've recently reworked my Windows service to load its various WCF services as plug-ins that operate within their own AppDomain. I've got a few cases in the bootstrapping process where I'm using MarshalByRefObject objects to get things up and running, but once the plug-ins are loaded, communication between the AppDomains is extremely easy using WCF.

Are there any Java VMs which can save their state to a file and then reload that state?

Are there any Java VMs which can save their state to a file and then reload that state?
If so, which ones?

Another option, which may or may not be relevant in your case, is to run the JVM (any JVM) inside a virtual machine. Most virtual machines offer the option to store and resume state, so you should be able to restart your PC, fire up the VM when it comes back up and have the Java process pick up from where it was.
I use VMWare Player for testing on IE at work, and this works as noted above when I close and later reopen it. I don't generally do this when apps are doing anything of note in the VM, but as long as they aren't accessing any external resources (e.g. network sockets), I would expect it to work as if the VM was never shut down.

Continuations are probably be what you are looking for:
[...] first class continuations, which are constructs
that give a programming language the
ability to save the execution state at
any point and return to that point at
a later point in the program.
There are at least two continuation libraries for Java: RIFE continuations and javaflow. I know that javaflow at least allows serializing state to disk:
A Continuation can be serialized if
all objects it captured is also
serializable. In other words, all the
local variables (including all this
objects) need to be marked as
Serializable. In this example, you
need to mark the MyRunnable class as
Serializable . A serialized
continuation can be sent over to
another machine or used later. - Javaflow Tutorial

You should serialize relevant domain-specific objects which can be de-serialized by another JVM run-time.
I'm not aware of any tools persisting an entire JVM. The closest I got to doing this was creating a core dump from a running JVM process using gcore, then using jsadebugd, jmap or jstack to debug it.
For instance:
$ jps # get JVM process ID XXX
$ gcore -o core XXX
$ jsadebugd $JAVA_HOME/bin/java core.XXX
UPDATE
I don't think you're going to find a solution that's portable between architectures just yet.

It is worth noting that many objects cannot be serialized as they have state outside the java context.
e.g. Sockets, Threads, Open files, Database connections.
For this reason, it is difficult to to save the state of a useful application in a generic way.

I'm not aware of JVM's that can store state. Depending on your exact needs, you can maybe consider using Terracotta. Terracotta is essentially able to share heap state between JVM's, and store this state to disk.
This can be used to cluster applications, and/or make the heapstate persistent. In effect, you can use it to start the JVM up and pick up where you left off. For more information check out:
http://www.infoq.com/articles/open-terracotta-intro
Hope this helps.

I've worked on an embedded Java project which used this approach to start up quickly.
The JVM was from Wind River, running on top of VxWorks.

Sun has done some research on "orthogonal persistence", which provides "persistence for the full computational model that is definedby the Java Language Specification":
http://research.sun.com/forest/COM.Sun.Labs.Forest.doc.opjspec.abs.html
PJama is a prototype implementation:
http://research.sun.com/forest/opj.main.html

To my knowledge, there is nothing to capture JVM state and restore it, but people are trying to serialize/deserialize the Thread class to achieve something similar. The closest thing to a working implementation I found was brakes, but you may find more when you google for "thread serialization".

I take it you want to be able to resume from where the snapshot was stored, as if nothing thereafter had happened.
I wonder how many framework components and libraries such functionality would break. Suddenly, you are reviving a JVM state from storage; in the meantime, the clock has mysteriously skipped forward by 23 hours, network connections are no longer valid, GUI objects no longer have any underlying O/S handles... I'd say this is nontrivial, and impossible in the general case without modifying the framework extensively.

If you can get away with just storing the state of your in-memory objects, then something like Prevaylor might work for you. It uses a combination of journalling changes to business objects with a serialized snapshot to record the state of your objects, which you can then reload later.
However, it doesn't store the full JVM state (call stack, GC status etc). If you really need that level of detail, then a specialized JVM might be needed.

The answer at this time is no, there are no JVMs that can 'hibernate' like your operating system can or like VMWare et al can.
You could get half-way there, depending on the complexity of your app, by just serializing state out when the program closes and serializing it back in, but that won't do stuff like pause executing some business logic when you close and then continue when you open it again.