I'm running a shell script on the university's server. In this shell script, I will execute java, c, c++, python and perl programs. Because every program will be executed many many times(I'm a teaching assistant and will test the students' programs with many different inputs). The server always gives me an error: "running out of system resource". I guess this is due to I do not release the resource.
I heard that running a program in the shell script one time will active one process. So I think maybe there are so many processes that the system recourse allocated for me has been run out.
Is there any way to figure this problem out?
I pose part of my shell code as following:
# maxconnect4 is the compiled c code
for ((i = 1; i <= 21; i++))
do
maxconnect4 input1.txt
done
Thanks
Zhong
Since you are automatically running students' programs then it may be that their programs are badly written and using more RAM than similar programs written by more skilled programmers would require. Even Java and Python programs can be written in such a way as to leak memory (think about a stack that never gets anything popped off of it, only more things pushed on).
You should test your setup with known good implementations of the assignments you are about to grade as a sanity check.
You should also look at the source code for the students' work. Especially if you get the error on their assignment.
You may also just have an overloaded system, and may need to run these tests on another machine. Using a machine that does not have other users is a good idea for this type of thing, since things outside of your and the program you are testing aren't likely to mess up your tests.
You may also want to keep top running on that machine on another terminal while you run the test to monitor resource usage.
You seem to be running maxconnect4, then waitng for it to finish before starting the next run, so I don't think your shell script itself is the isuue. The big question is what maxconnect4 is doing. It could be very hungry for resources, or it itself could start child processes and return to your script.
I would try a few experiments such as by hand start maxconnect4 a few times, do you se the resource error?
I would also use system tools to invetsigate. For example use ps to see whether there are lots of processes running. Use vmstat to look at CPU and memory usage.
Related
So the idea is a kind of virtual classroom (a website) where students uploads uncompiled .java files, our server will compile it and execute it through C# or PHP, the language doesn't matter, creating a .bat file and get the feedback of the console if the program compiled correctly or not and if the execution was correct based on some pre-maded test, so far our tests did work but we have completely no control on what's inside the .java file so we want to stop the execution if some criterias did happen, i.e. an user input, infite loop, sockets instances, etc... I've digging on internet if there's a way to configure the java environment to avoid this but so far can't find anything, and we don't want our backend language to go through the file to check this things cause will be a completly mess up
Thanks for the help
You could configure a security manager, but it doesn't have a very good track record of stopping a determined attacker, and doesn't do resource limiting anyways.
You could load the untrusted code with a dedicated class loader that only sees white-listed classes.
Or you could use something like docker to isolate the process at the operating system level. This could also limit its cpu and memory consumption.
I'd probably combine these approaches, but some risk will remain in either case.
(Yes, I realize that is complex, but safely sandboxing arbitrary java code is a hard problem.)
I'm fairly new at this stuff, but essentially: there are programs and there are processes. A program is a file that spawns a process, when executed.
You can't delete a program if there is a process still associated to it. The process needs to be killed first.
This seems to be the case for Java programs too. However I'm curious as to why - isn't the entire thing loaded into the JVM anyway?
"Deleted file" involves som OS-semantics. Under Unix/Linux a file may be deleted and all open file handles stay valid. When the last open file handle vanishes, the space occupied by the deleted file is returned to the pool of free space.
Under Windows there may be other mechanisms.
The JVM works as a Just-In-Time (JIT) compiler. There are many sources of information on JIT compilation, but basically as a java program is running it will encounter parts of the program that are needed, these pieces of the program are in .class files. These .class files is just an intermediate form of Java code (it's not quite Java code, but not quite machine code, yet). Obviously, compiling at runtime (JIT) takes resources (CPU cycles) and, thus, time. So, the JVM only loads pieces of the program that it needs to minimize wasted CPU cycles.
But yes, your understanding of process/programs is correct. To sum up: A process is a running instance of a program. This running program, then can spawn even more processes or threads to perform work.
Here's my issue. I have an existing .jar file that I must use in my program. The program, however, is written in Python.
Since my program is taking a long time to run (a named entity tagger on a large development corpus) I profiled it using cProfiler and lined profiled it using line_profiler. It seems that 92% of the time is spent on this task.
I am currently using the following code:
import subprocess as sub
sub.call(["java", "-jar", "-Xmx512m", "MyFile.jar",
featuresFileName, numIterations, featureCutOff])
I read somewhere about subprocess vs Popen and other bits and pieces, but couldn't find a good solution that does not require subprocess or os calls (of course, there may not be any).
I'd really appreciate some advice on the fastest way to run a .jar file from within a Python script. Note, however, that I cannot modify the Java code nor do I have access to speak to the developer of that code.
Alternatively, and I don't know if this will help or if I'm simply grasping at straws here, but perhaps there is a way to keep the process called in sub.call() above in the background, somehow keeping the JVM running (?) so that I can simply invoke the jar file. Maybe that can help reduce startup costs? BTW I am a total Java newbie (mostly C++,C#,Python experience) so my question could make no sense whatsoever - I apologize in advance...
You could try porting your Python to Jython, and then run it all natively in the same JVM (that may or may not work). That way you have effectively zero start up time, and the JVM has enough time to leverage its JIT over time to ideally give you better performance overall.
That indicates that most of the time is spent in this process. It may not be the startup time which is the problem. It may be what it does once it has started.
The only way around this I can think of is to run the process in the background, multiple time concurrently if that is an option. (concurrently rather than running one after another)
Try with "-client" option. It should reduce JVM startup time.
By analysing the manifest file of the jar file you can find out the class name of the jar file which is used. So then you could in principle write your own small java daemon which is listening for new arguments to arrive and calls the main() function of the appropriate class. But it is really worth the effort only if startup costs are the issue.
I assume the latest update version of java would provide better performance.
I am looking for a way to implement isolation of software components from endless loops or memory leaks. Android isolates each app in it's own process, Google Chrome isolates each tab in it's own process.
My primary drawback is that java takes so long to start and also I would like to reduce memory consumption.
Is there any alternate build or more controlled startup that will accomplish this?
If quick startup is your goal, Java on a PC may not be your best bet. It's going to take a few seconds because that's how long it takes to load the VM from disk.
If you want your app to start more quickly it's easy to get a splash screen up, just create a module that only loads your splash screen, waits for it to fully display then uses reflection to link to your "Real" main module.
(Use reflection because otherwise it will pull in your entire program through references before it starts the main one--at least that's how it used to work).
If you're talking about run-time performance, you won't get quicker by changing languages, Java's about as fast as you can get. You MIGHT be able to get a boost by converting to C/C++ and rewriting it to suit those platforms (Less OO, stack allocations instead of heap, etc), but otherwise none of the other languages in general usage are close to Java in speed.
If you really need the quick startup, depending on what you are doing there may be some tricks. I've seen projects that try to keep a Java VM running in your toolbar and allow you to make requests (tell it to start an app). This was faster but made additional requirements of the user (Loading this additional tool)
Another possibility--if you are constantly starting up/shutting down small tasks and that's the reason the startup bothers you then you can definitely speed it up by keeping it running invisibly. Just have your Java app open a socket and listen for commands then create a little .EXE or shell script that can start your program if it's not running or send commands to that socket if it is. This would completely eliminate startups after the first run.
In general, Java has a much longer startup time than other languages. If you are sticking with Java on a desktop app, a lot of stuff like startup time is determined by the JRE installed on the client's computer, which you can't control.
As to "endless memory leaks"... Java doesn't leak memory. If your program does, fix it.
This is a second answer because it's completely different and my other got too long :)
Try compiling it--I think GCC can compile it. This could almost completely eliminate your startup. I believe Jikes used to be a windows java compiler by IBM, but I don't know if it's still maintained.
Note that compiled code will probably run slower than JVM code for long-running apps.
I have an application that has a license for a set number of cpus and I want to be able to set the number of cpus that java runs in to 1 before the check is done. I am running Solaris and have looked at pbind but thought that if I started the application and then used pbind it would have checked the license before it had set the number of CPUs that java could use.
Does anyone know a way of starting an application with a set number of CPUs on Solaris?
It is a workaround, but using Solaris 10 you could set up a zone with a single CPU available and then run the application inside that zone.
If you want to do testing without running the full application, this bit of Java is most likely what they are using to get the number of CPU's:
Runtime runtime = Runtime.getRuntime();
int nrOfProcessors = runtime.availableProcessors();
A full example here.
This isn't a complete solution, but might be enough to develop into one. There's definitely a point at which the java process exists (and thus can be controlled by pbind) and at which point it hasn't yet run the code to do the processor check. If you could pause the launch of the application itself until pbind had done its work, this should be OK (assuming that the pbind idea will work from the CPU-checking point of view).
One way to do this that should definitely pause the JVM at an appropriate place is the socket attach for remote debuggers and starting with suspend mode. If you pass the following arguments to the java invocation:
-Xdebug -Xrunjdwp:transport=dt_socket,address=8000,suspend=y,server=y
then the JVM will pause after starting the java process but before executing the main class, until a debugger/agent is attached to port 8000.
So perhaps it would be possible to use a wrapper script to start the program in the background with these parameters, sleep for a second or so, use pbind to set the number of processors to one for the java process, then attach and detach some agent to port 8000 (which will be enough to get Java to proceed with execution).
Flaws or potential hiccoughs in this idea would be whether running in debug mode would notably affect performance of your app (it doesn't seem to have a big impact in general), whether you can control some kind of no-op JDWP agent from the command line, and whether you're able to open ports on the machine. It's not something I've tried to automate before (though I've used something broadly similar in a manual way to increase the niceness of a Java process before letting it loose), so there might be other issues I've overlooked.
I think the most direct answer to your question is to use pbind to bind the running shell process, and then start Java from that shell. According to the man page the effects of pbind are inherited by processes that are created from a bound process. Try this:
% pbind -b 0 $$
% java ...
Googling over, I found that you are right, pbind binds processes to processors.
More info and examples at: http://docs.sun.com/app/docs/doc/816-5166/pbind-1m?a=view