We Keep Coding

Determining if a Java app is malware

I am curious about what automatic methods may be used to determine if a Java app running on a Windows or PC is malware. (I don't really even know what exploits are available to such an app. Is there someplace I can learn about the risks?) If I have the source code, are there specific packages or classes that could be used more harmfully than others? Perhaps they could suggest malware?
Update: Thanks for the replies. I was interested in knowing if this would be possible, and it basically sounds totally infeasible. Good to know.

If it's not even possible to automatically determine whether a program terminates, I don't think you'll get much leverage in automatically determining whether an app does "naughty stuff".
Part of the problem of course is defining what constitutes malware, but the majority is simply that deducing proofs about the behaviour of other programs is surprisingly difficult/impossible. You may have some luck spotting particular patterns, but on the whole you can't be confident (and I suspect it's provably impossible) that you've caught all possible attack vectors.
And in the general sphere, catching 95% of vectors isn't really worthwhile when the attackers simply concentrate on the remaining 5%.

Well, there's always the fundamental philosophical question: what is a malware? It's code that was intended to do damage, or at least code that doesn't do what it claims to. How do you plan to judge intent based on libraries it uses?
Having said that, if you at least roughly know what the program is supposed to do, you can indeed find suspicious packages, things the program wouldn't normally need to access. Like network connections when the program is meant to run as a desktop app. But then the network connection could just be part of an autoupdate feature. (Is autoupdate itself a malware? Sometimes it feels like it is.)
Another indicator is if a program that ostensibly doesn't need any special privileges, refuses to run in a sandbox. And the biggest threat is if it tries to load a native library when it shouldn't need one.
But all these only make sense if you know what the code is supposed to do. An antivirus package might use very similar techniques to viruses, the only difference is what's on the label.

Here is a general outline for how you can bound the possible actions your java application can take. Basically you are testing to see if the java application is 'inert' (can't take harmful actions) and thus it probably not mallware.
This won't necessarily tell you mallware or not, as others have pointed out. The app could still do annoying things like pop-up windows. Perhaps the best indication, is to see if the application is digitally signed by an author you trust; if not -- be afraid.
You can disassemble the class files to determine which Java APIs the application uses; you are looking for points where the java app uses the OS. Since java uses a virtual machine, there are well defined points where a java application could take potentially harmful actions -- these are the 'gateways' to various OS calls (for example opening a socket or reading a file).
Its difficult to enumerate all the APIs, different functions which execute the same OS action should require the same Permission. But java's docs don't provide an exhaustive list.
Does the java app use any native libraries -- if so its a big red flag.
The JVM does not offer the ability to run arbitrary code, or use native system APIs; in particular it does not offer the ability to modify the registry (a typical action of PC mallware). The only way a java application can do this is via native libraries. Typically there is no need for a normal application written in java to use native code (unless it needs to use devices).
Check for System.loadLibrary() or System.load() or Runtime.loadLibrary() or Runtime.load(). This is how the VM loads native libraries.
Does it use the network or file system?
Look for use of java.io, java.net.
Does it make system calls (via Runtime.exec())
You can check for the use of java.lang.Runtime.exec() or ProcessBuilder.exec().
Does it try to control the keyboard / mouse?
You could also run the application in a restricted policy JVM (the instructions/tools for doing this are not as simple as they should be) and see what fails (see Oracle's security tutorial) -- note that disassembly is the only way to be sure, just because the app doesn't do anything harmful once, doesn't mean it won't in the future.
This definitely is not easy, and I was surprised to find how many places one needs to look at (for example several java functions load native libraries, not just one).

What are common Java vulnerabilities?

What are common Java vulnerabilities that can be exploited to gain some sort of access to a system? I have been thinking about it recently, and havent been able to come up with much of anything - integer overflow - maybe? race condition - what does it give you?
I am not looking for things like "sql injection in a web app". I am looking for a relationship similar to buffer overflow - c/c++.
Any security experts out there that can help out? Thanks.

Malicious Code injection.
Because Java (or any language using an interpreter at runtime), performs linkage at runtime, it is possible to replace the expected JARs (the equivalent of DLLs and SOs) with malicious ones at runtime.
This is a vulnerability, which is combated since the first release of Java, using various mechanisms.
There are protections in places in the classloaders to ensure that java.* classes cannot be loaded from outside rt.jar (the runtime jar).
Additionally, security policies can be put in place to ensure that classes loaded from different sources are restricted to performing only a certain set of actions - the most obvious example is that of applets. Applets are constrained by the Java security policy model from reading or writing the file system etc; signed applets can request for certain permissions.
JARs can also be signed, and these signatures can be verified at runtime when they're loaded.
Packages can also be sealed to ensure that they come from the same codesource. This prevents an attacker from placing classes into your package, but capable of performing 'malicious' operations.
If you want to know why all of this is important, imagine a JDBC driver injected into the classpath that is capable of transmitting all SQL statements and their results to a remote third party. Well, I assume you get the picture now.

After reading most of the responses I think your question has been answered in an indirect way. I just wanted to point this out directly. Java doesn't suffer from the same problems you see in C/C++ because it protects the developer from these types of memory attacks (buffer overflow, heap overflow, etc). Those things can't happen. Because there is this fundamental protection in the language security vulnerabilities have moved up the stack.
They're now occurring at a higher level. SQL injection, XSS, DOS, etc. You could figure out a way to get Java to remotely load malicious code, but to do that would mean you'd need to exploit some other vulnerability at the services layer to remotely push code into a directory then trigger Java to load through a classloader. Remote attacks are theoretically possible, but with Java it's more complicated to exploit. And often if you can exploit some other vulnerability then why not just go after and cut java out of the loop. World writable directories where java code is loaded from could be used against you. But at this point is it really Java that's the problem or your sys admin or the vendor of some other service that is exploitable?
The only vulnerabilities that pose remote code potential I've seen in Java over the years have been from native code the VM loads. The libzip vulnerability, the gif file parsing, etc. And that's only been a handful of problems. Maybe one every 2-3 years. And again the vuln is native code loaded by the JVM not in Java code.
As a language Java is very secure. Even these issues I discussed that can be theoretically attacked have hooks in the platform to prevent them. Signing code thwarts most of this. However, very few Java programs run with a Security Manager installed. Mainly because of performance, usability, but mainly because these vulns are very limited in scope at best. Remote code loading in Java hasn't risen to epidemic levels that buffer overflows did in the late 90s/2000s for C/C++.
Java isn't bullet proof as a platform, but it's harder to exploit than the other fruit on the tree. And hackers are opportunistic and go for that low hanging fruit.

I'm not a security expert, but there are some modules in our company that we can't code in java because it is so easy to de-compile java bytecode. We looked at obfuscation but if you want real obfuscation it comes only with a lot of problems (performance hit/loss of debug information).
One could steal our logics, replace the module with a modified version that will return incorrect results etc...
So compared to C/C++, I guess this is one "vulnerability" that stands out.
We also have a software license mechanism built-in in our java modules, but this can also be easily hacked by de-compiling and modifying the code.

Including third party class files and calling upon them basically means you are running unsecure code. That code can do anything it wants if you don't have security turned on.

SAFE levels in JRuby

Are SAFE levels supported in JRuby? If not, is there other ways of safely running user supplied code in server?

AFAIK, they aren't supported.
The main problem is that they are very badly documented, so how are the JRuby developers supposed to provide a compatible implementation if nobody knows what a compatible implementation is?
Another reason not to waste time implementing $SAFE levels in JRuby, is that the JVM's security mechanisms provide better protection anyway. Which is also the answer to your second question: from the point of view of the JVM, your Ruby script is just another Java program and it can be sandboxed and controlled just like any other Java program.
How to do that, however, is a question for a Java expert. I'm just a lowly Ruby hacker …

Does Java have a built-in Antivirus? Is it true?

Does Java have a built-in Antivirus?
One of my friends told me there is in the JVM itself - it's called the "sandbox". Is it true?

Java does have a security-related concept called "sandbox", but it works very differently from typical anti-virus products. The latter usually try to catch viruses via signatures or code analysis before they are executed.
The Java sandbox on the other hand allows you to run Java code while witholding from it access to system resources that could be used to to bad things, e.g. no access to any files.
However, only Java applets and Java Web Start applications run in a sandbox per default. Regular java applications have full access to your system.

Doubtful. Perhaps he was referring to the fact that the JVM (somewhat) sandboxes execution of a Java program, to help prevent it from damaging the host OS.

No they do not have a built-in antivirus. Did he tell you this on April 1st?
To clear your doubt, sandbox is not an antivirus.

does the java have an in-built antivirus?
No.

Java has a security model built-in that allows it to execute untrusted code. This model is called "the sandbox model".
It is not a virus-scanner. Instead, it limits the possibilities of untrusted code so that applets on a webpage do not have access to files on your computer's hard drive.
You can read more about Java's Security Architecture.

java uses a class called SecurityManager to determine what a program can or cannot do, so in some sense it implements anti-exploit code, but not specifically anti-virus.
http://java.sun.com/j2se/1.4.2/docs/api/java/lang/SecurityManager.html
anti-virus in the usual sense of the word detects viruses in files and removes them. this is not built in to java.

No. What it does is running the program in an environment that is (somewhat) separated from the operating system, which should, in most cases, prevent malicious code from doing any damage. Sort of like running VMware - virii and other malware have no influence on the host OS.

I heard garbage collection also acts as a handy anti-bacterial, making your applications 99.99% free from germs.
Wash after every use.

The closest thing in the JRE to literal "anti-virus" is the blacklisting feature for signed jars. If a signed jar is found to cause a security issue, it can be blocked. This has been designed for accidental security flaws rather than blocking deliberately malicious code. Also it is possible to revoke a certificate using a CRL (Certificate Revocation List) or OCSP (Online Certificate Status Protocol) if enabled. Conventional anti-virus is left to specialist anti-virus products, rather than trying to produce a half-baked alternative.
(Today's anti-virus products do more than just check for known viruses.)

Limitations of Java desktop applications?

I come from a C/C++ background and now do a lot of C# stuff.
Lately I have become interested in doing some projects in Java since playing around with the Android SDK.
I know that Java apps run in a sandbox that can limit their access to the system.
In a desktop/server application environment what kind of things are restricted?

Java applications are much in a sandbox as .NET applications are in a sandbox. They both run on their respective virtual machines, and do have some limitations as to what they can do, but for the most part, they have a good deal of access to the system, including access to native code through certain calls.
You may be thinking about Java applets, which run inside a browser, and generally will be in a security sandbox that prevents access to the system resources such as local files. (This restriction can be circumvented by specifically granting access to the system to certain applets.)
Here's a section on Security Restrictions for applets from The Java Tutorials, which includes a list of restrictions placed on applets.

Typically desktop and server application run with security disabled. However, Java and the JVM still have a robust type system, so you can't for instance cast to types that an object was not created with, cannot access freed memory and can't run off the end of buffers.

For normal desktop and server apps, the limitations are not related to the sandbox concept (though you could use it to apply very fine-grained restrictions to e.g. user-submitted code) but to the platform-independant nature of Java. Basically, OS-specific stuff and hardware access usually can't be done in pure JAVA unless specifically adressed by the API library.
Examples are:
Windows registry
Windows system tray
Bluetooth
WLAN configuration

I think the main limitation you might see, is the ability to easily use the native system API's if you needed, for example if you needed to use a user32 or kernel32 API from java I think it is possible, however it is not an easy task to do, however in C# it is fairly easy thing to do.
Also if you have some legacy C/C++ dll's you can still use them in a C# application, while in java is still hard to do especially that in the worst case when your native code api has to use pointers, you can use unsafe mode in C# application to pass pointers and allocate fixed memory on stack ... etc.
but as mentioned above Java & C# in general are very much have the same limitations especially if you are targetting being platfrom independent.