We Keep Coding

Java Testing that Includes Multiple Virtual Machines

I have a large piece of Java software that I am looking to add tests to for a new module. The new module involves running external applications and handling the results. The external applications are not something I can modify. In order to get results, these external applications need to be able to access and modify 2+ remote machines. Once they make those modifications, that machine is no longer in a valid state for more testing. My current application already includes JUnit tests, so it would be nice if I could write the new tests in JUnit as well.
My question: Is there a know technique for this use case? I know I could hack together some kind of solution using Vagrant, where machines are started up and torn down in the setup and teardown blocks of JUnit, but that seems like it could be hard to maintain, would require the tests to be run serially, and would be very, very slow.
EDIT: Removed the word "unit" because it was significantly sidetracking the question.

Pluggable cluster management system

Problem:
We have a cluster of servers, running on Java (Tomcat). Each server exposes a lot of information via JMX. It takes a lot of time to go through JMX to make sure the cluster is in valid state.
In some cases it's enough to check a certain status in every node. In other cases the data is spread across the nodes, so some logical analysis is required. Sometimes too much data needs to be analyzed and it cannot be done manually.
The question:
Is there a cluster management system that would provide a platform for automating such tests?
Requirements:
It has to be able to be extended by plugins (preferably written in Java), so for some complex customized test all we need is to develop a plugin with Business Logic.
It has to provide some JMX client platform
GUI/JMX interface for running tests and seeing results
Scheduling
SNMP monitoring

This does not satisfy all of your requirements, but you should take a look at JGroups
(A Toolkit for Reliable Multicast Communication)
Essentially, it is a java library for creating clustered nodes that can be kept in synch with each other via various protocols using mutlicast or unicast. It includes a rich set of Building Blocks to help you build the functional stack you need. Your customization of the stack can be implemented using the Building Blocks and/or your own custom state management or cluster invokable "business methods". By that I mean, you could define a business method called int getOpenPortCount() which you could then invoke, not on a single node, but on the cluster. Each attached node would then invoke the method locally and return the result of the invocation, resulting in your cluster invocation returning [effectively] a value of int[] whose length is the number of nodes on your JGroups cluster.
It has to provide some JMX client platform
I am not completely sure what you mean by this, but there is no built in JMX-Connector as such. However, you may not need it since you can communicate with individual nodes directly, or all nodes through the cluster using the JGroups API.
GUI/JMX interface for running tests and seeing results
I don't think you'll find anything like this, but, since you would essentially be using pure Java, you could use a combination of the JGroups API, the JMX API, jUnit (or TestNG) and an eclipse based test-runner which provides a fairly decent testing harness and visualizing UI.
Scheduling
You can schedule events to be executed across the cluster using the JGroups TimeScheduler, Quartz or simply use ScheduledThreadPoolExecutor.
SNMP monitoring
JGroups supports JMX for monitoring, which in turn can be bridged through the JVM's SNMP agent. This jboss specific link will give you an idea of how this can be implemented.

Testing a client-server based app in Java

I have written a client server based application in Java, where the client continually (after every 30 seconds) sends some data, and a panel on server is redrawn according to the incoming message from client. There are multiple threads (one for redrawing, one for reading incoming messages, and one for outgoing messages) running on the server. For GUI I have used Swing.
Now I am a complete newbie in testing, and I would like to know various methods, tricks and gotchas for testing my application. Any web resource or good texts on the same will be very helpful. Thanks in advance.

Take a look to Maveryx test automation framework and its documentation. It contains several examples to create and run automated scripts for testing Java (swing-based) applications.

You will, at least, have to do testing at two levels.
You will need to first unit test your java code, on both server and client, by writing e.g. JUnit tests.
You should then test the server and the client components individually by simulating the other side. Something like JMock may be able to help you simulate the parts of the code that you are not testing. For testing you GUI, you may refer to What is the best testing tool for Swing-based applications?
There is a lot of good material available on this topic online!

If you're looking to do unit testing, one of the most common ways of doing this in Java is with JUnit. The JUnit Cookbook has some decent information to get started with writing your unit tests.
JUnit Cookbook

Invoking a service on other java application running on the same machine

I created a command line interface on a small java application I created for personal use.
For the moment the cli is resided in the same project as the original application but I'm planning to extract it into it's own project, effectively building 2 separate executable jars enabling me to start the cli as needed and query the other running program for information.
I'm trying to figure out the easiest and most lightweight solution to call a remote service, on the same machine.
I looked at spring remoting but many of the provided solutions such as HttpInvoker, Hessian/Burlap, JAX RPC web services are based on HTTP or SOAP and therefore not suited for the job.
JMS also seems like overkill.
This leaves me with RMI, which looks rather heavyweight, and possibly JMX?
Suggestions?

JMX would use RMI underneath for remote access. JMX is meant for exposing admin apis (monitoring / management) - not intended as a general purpose remoting api.
RMI with the spring remoting support is fairly lightweight from a development point of view. Even runtime it is the option that adds least overhead compared to the other options you have listed.
Also with spring remoting support you can easily switch over to a different option if required later.
Take a look at this artcile, that compares / benchmarks performance of the above options.

I'd say it depends very much on where the project/functionality is heading. JMX is easy enough to set up, and you can make use of existing clients/guis to query and set parameters - this may save you a lot of work. It may also allow you system to integrate with monitoring tools out there.
If, on the other hand, the functionality has little to do with managment/monitoring, and more along the lines of pumping data in and out, one option may be Apache MINA. I've used it in the past with great results. But you'll effectively be creating your own protocol ! I doubt that MINA will end up being "less heavyweight" than simple RMI though.

In an app for personal use, I'd go with JMX because it should be the path of least resistance. I've had great experiences with this in the past. You'll be able to get it up and running in minutes, and you won't have to think about what message format to move data in (as long as your beans are Serializable, that is).
Put an interface in front of the remote call, so that later you can drop in another implementation later if JMX turns out to be inadequate.

Spring / Java, good method for remotely interacting with command line Java app on another server?

I am working on a Spring web application where I have a need to interact with a remotely based command-line java application to run a simple search query on this application and get back the results. I initially had integrated this into my Spring app but my app is, itself, needing a lot of memory (its an app that involves huge amounts of data) and I don't think they can coexist on one server anymore.
I am running everything on Amazon ec2 so the latency between the servers should be really low. I figure I could use a direct SSH connection but am not so sure if this is the best approach. I'd like to keep the command-line app I am interacting with as simple as possible (would rather not make it into a web-service if I don't have to). I'm still fairly new to Java so sorry if this sounds like a basic question.

You have several options other than a web service. Some of them are:
Protocol Buffers
JMS
Simple socket based client/server Java
Thrift

Assuming you have or can have spring on both ends -
Exposing service objects and consuming them from a different process becomes extremely trivial using Spring's remoting support. (RmiServiceExporter may be most appropriate and least trivial to setup in this case)
It really does away with all the boiler plate code and let's you focus on your business/service logic.
You can write a hello world service and consume it from another Java program in less than twenty minutes. Once you have this "infrastructure" setup, you are free to focus on your actual business logic.
(You absolutely don't have to know rmi to get this working though rmi working knowledge may help if you run into problems. But then, what is SO community for? ;))