We Keep Coding

How to effectively manage a bunch of jar files and their plumbing?

This is a rather high-level question so apologies if it's off-topic. I'm new to the enterprise Java world.
Suppose I have written some individual Java packages that do things like parse data feeds and store the parsed information to a queue. Another package might read from that queue and ingest those entries into a rules engine package. Tripped alerts get fed into another queue, which is polled by an alerting service (assume it's written in Python) that reads from the queue and issues emails.
As it stands I have to manually run each jar file and stick it in the background. While I could probably daemonize some or all of these services for resiliency or write some kind of service manager to do the same, this strikes me as being very amateur. Especially since I'd have to start a dozen services for this single workflow at boot.
I feel like I'm missing something, but I don't know what I don't know. Short of writing one giant, monolithic application, what should I be looking into to help me manage all these discrete components and be able to (conceptually) deliver a holistic application? I'd like to end up with some sort of hypervisor where I can click one button, it starts/stops all the above services, provides me some visibility into their status and makes sure the services are running when they should.
Is this where frameworks come into play? I see a number of them but don't know if that's just overkill, especially if I'm not actively developing a solution for that framework.

It seems you architected a system with a lot of components, and then after some time you decided to aggregate some of them because they happen to share the same programming language: Java. So, first a warning: this is not the best way to wire components together.
Also, it seems you don't know Java very well because you mix terms like package, jar and executable that are totally unrelated and distinct concepts.
However, let's assume that the current state of the art is the best possible and is immutable. Your current requirement is building a graphical interface (I guess HTTP/HTML based) to manage all the distinct components of the system written in Java. I suggest you use a single JVM, writing your components as EJB (essentially a start(), stop() and a method to query the component state that returns a custom object), and finally wire everything up with the Spring framework, that has a nice annotation-driven configuration for #Bean's.
SpringBoot also has an actuator package that simplify exposing objects. You may also find it useful to register your beans as Managed beans, and using the Hawtio framework to administer them (via a Jolokia agent).

I am not sure if you're actually using J2EE (i.e. Java Enterprise Edition). It is possible to write enterprise software also in J2SE. J2SE is not having too much available off the shelf for this, but in contrast has a lot of micro-frameworks such as Ninja, or full stack frameworks such as Play framework which work quite well, much easier to program, and performs much better than J2EE.
If you're not using J2EE, then you can go as simple as:
make one new Java project
add all the jars as dependency to that project (see the comment on Maven above by NimChimpsky)
start the classes in the jars by simply calling their constructor
This is quite a naive approach, but can serve you at this point. Of course, if you're aiming for a scalable platform, there is a lot more you need to learn first. For scalability, I suggest the Play! framework as a good start. Alternatively you can use Vert.x which has its own message queue implementation as well as support for high performance distributed caches.
The standard J2EE approach is doable (and considered "de-facto" in many oldschool enterprises) but has fundamental -flaws- or "differences" which makes a very steep learning curve and a very much non-scalable application.

It seems like you're writing your application in a microservice architecture.
You need an orchestrator.
If you are running everything in a single machine, a simple orchestrator that you probably is already running is systemd. You write systemd service description, and systemd will maintain your services according to your services description. You can specify the order the services should be brought up based on dependencies between services, restart policy if your service goes down unexpectedly, logging for stdout/stderr, etc. Note that this is the same systemd that runs the startup sequence of most modern Linux distros.
If you're running multiple machines, you can still keep using single machine orchestrator like systemd, but usually the requirement for the orchestrator will also become more complex. With multiple machines, you now have to take into account things like moving services between machines, phased roll out, etc. For these setups, there are software that adapts systemd for multi machine orchestration, like CoreOS's fleetd; and there are also standalone multi machine orchestrator like Kubernetes. Both uses docker as application container mechanism.
None of what I've described here is Java specific, which means you can use the same orchestration for Java as you used for Python or other languages or architecture.

You have to choose, As Raffaele suggested you can choose to write all your requirements into one app/service. Seems like a possible mission, using java Ejb's or using spring integration - ampqTemplate ( can write to a queue with ampqTemplate and receive the message with a dedicated listener (example).
Or choosing implementation with microservices architecture. write a service that will push to the queue another one that will contain the listener etc. a task that can be done easily with spring boot.
"One button to control them all" - in the case of a monolithic app - it's easy.
In case that you choose microservices architecture. It depends what are you needs. if its just the "start" "stop" operation I guess that that start and stop of your tomcat/other server will do. For other metrics, there is a variety of solutions. again, it depends on your needs.

Fork/Join for JavaEE app?

I know that spinning off threads in a JavaEE application is a big no-no. However, I have an application that is a perfect candidate for Java's fork/join mechanism. But, since threads are not supposed to be created within an application, is there any way to make use of this within my EJB? I know that on WebSphere applications the async bean functionality provides this. However, my application is deployed on JBoss EAP 6.1, so this is not an option.
Is there a "legal" way to accomplish fork/join within a JavaEE application?

The best answer now is the Concurrency Utils API in the Java EE 7 specification. You have ManagedExecutors and ManagedThreadPools. Since these managed threads features are and managed tasks are controlled by the application server then ensure your fork join computation uses these resources then you can ensure that threads are contained and not orphaned.
Finally you probably have to write a version of ForkJoinPool that is 'Managed' to get the optimal solution. However it should be possible because one would replace the thread pool executor with the managed version as a first step.
PS: Java EE 8 must resolve this when Java SE 8 is released!

JBoss EAP 6.1 actually supports async EJBs as well. But AFAIK async EJBs only really help you when you don't need to wait for the results of the subtasks (eg. you only need the fork part, not the join part).
If you use java.util.concurrent.ForkJoinPool there isn't a legal way of using it in Java EE and Java EE 7 / JSR-236 does not help (I raised this point with the EG but they couldn't be bothered). ForkJoinPool spwans threads which is illegal and ManagedThreadFactory from EE 7 / JSR-236 is not a ForkJoinWorkerThreadFactory.
In JDK 8 there is a default ForkJoinPool which can be configured to not spawn any threads and run everything in the caller thread (probably through the "java.util.concurrent.ForkJoinPool.common.parallelism" system property). This makes it legal but won't give you any parallelism.
On a more general note fork join tasks should be compute bound and not IO bound. In theory spawning thread threads in Java EE is safe as long as you don't use any Java EE features (and let them terminate when your application undeploys). For example:
transactions
JDBC
JPA (including lazy loading)
security
EJBs
remoting
class loading
JNDI
…
These features also generally make tasks IO bound instead of CPU bound.
And yes the same issues apply to JDK 8 parallel streams.

You can create a Singleton ejb, which is similar to a "service" (i.e. a JMX service). within the context of this special service, you can control threading and synchronization. so, you can create a singleton ejb which encapsulates the job execution with the fork/join logic, and your standard ejbs/mdbs can utilize this service.

In JavaEE the container controls the threads. Just imagine what would happen if every programmer decided to create his own threads? In any case the use of threads in the parallel bulk operations for Java8 have been rejected. SEE HERE You can make any assumption you want, yes I wrote the article.

How to delay processing reasonably in Java EE context?

Within a Java EE 5 environment I have the problem to ensure the existence of some data written by another part before continue processing my own data.
Historically (J2EE time), it was done by putting the data object to be processed into an internal JMS queue after waiting for e.g. 500ms via Thread.sleep.
But this does not feel like the best way to handle that problem, so I have 2 questions:
Is there any problem with using the sleep method within an Java EE context?
What is a reasonable solution to delaying some processing within an Java EE 5 application?
Edit:
I should have mentioned, that my processing takes place while handling objects from a JMS queue via an MDB.
And it may be the case, that the data for which I'm waiting never shows up, so there must be some sort of timeout, after which I can do some special processing with my data.

You can use EJB TimerService feature. Using threads in a managed environment should be avoided.

I agree with #dkaustubh about timers and avoiding threads manipulation in JavaEE.
Another possibility is to use JMS queue with delayed delivery. Although it is not a part of JavaEE API, most of messaging systems vendors supports it. check here.

I think, its possible with some advanced Threading approach. More than thinking on manual synchronizations and thread management, you can always use the Java Concurrent package.
Future can be one of the ways to do this. Please refer to Java Concurrent package.

Use notifications and Object#wait() / Object#notifyAll()
i.e. Multithreaded, the producer notifies the consumer.

Invoking a service on other java application running on the same machine

I created a command line interface on a small java application I created for personal use.
For the moment the cli is resided in the same project as the original application but I'm planning to extract it into it's own project, effectively building 2 separate executable jars enabling me to start the cli as needed and query the other running program for information.
I'm trying to figure out the easiest and most lightweight solution to call a remote service, on the same machine.
I looked at spring remoting but many of the provided solutions such as HttpInvoker, Hessian/Burlap, JAX RPC web services are based on HTTP or SOAP and therefore not suited for the job.
JMS also seems like overkill.
This leaves me with RMI, which looks rather heavyweight, and possibly JMX?
Suggestions?

JMX would use RMI underneath for remote access. JMX is meant for exposing admin apis (monitoring / management) - not intended as a general purpose remoting api.
RMI with the spring remoting support is fairly lightweight from a development point of view. Even runtime it is the option that adds least overhead compared to the other options you have listed.
Also with spring remoting support you can easily switch over to a different option if required later.
Take a look at this artcile, that compares / benchmarks performance of the above options.

I'd say it depends very much on where the project/functionality is heading. JMX is easy enough to set up, and you can make use of existing clients/guis to query and set parameters - this may save you a lot of work. It may also allow you system to integrate with monitoring tools out there.
If, on the other hand, the functionality has little to do with managment/monitoring, and more along the lines of pumping data in and out, one option may be Apache MINA. I've used it in the past with great results. But you'll effectively be creating your own protocol ! I doubt that MINA will end up being "less heavyweight" than simple RMI though.

In an app for personal use, I'd go with JMX because it should be the path of least resistance. I've had great experiences with this in the past. You'll be able to get it up and running in minutes, and you won't have to think about what message format to move data in (as long as your beans are Serializable, that is).
Put an interface in front of the remote call, so that later you can drop in another implementation later if JMX turns out to be inadequate.

Why and when to use EJB-based web services?

To transfer data from one system to another, through data interface, by web services, we normally get a result set by SQL query, and format them as a web service endpoint, and allow it to be retrieved by another side.
With EJB 3.0, it seems we can replace the result set by stateless session bean. So are there any advantages over the SQL-based web services? And when should we use it?

This is a very broad question on the system architect level. I will try to answer with my best knowledge without starting a flame war (FYI, I have used both ejb and spring).
As you know, building a stable/robust software application requires many building blocks, such as logging, connection pool, etc. Usually, you can find libraries of these building blocks, but not all of them have common api, so they may require integration. In the worst case, you may have to lock into some vendors. The main idea of EJB 3 (or Java EE) is to provide a more complete set of building blocks (via API, annotation or config), so developers can start working on the core business logic right away with an industry standard API/spec/config without training on the proprietary APIs. Additionally, you can change vendor without changing your codes since API/config are really the industry standard (well, your mileage may vary a lot in the real life. hopefully, the new Java EE will fix it).
Your application may already have some of the main elements that EJB 3 already provides. However, EJB 3 promises to provide more such as ORM mapping, RMI, Load balancing, failover, transactions, dynamic redeployment, logging, system management, thread managing, resource pooling (db connection), security, caching.
As you have an working application already, you can really consider if it is worth of your efford to migrate your codes to a standard system to gain more functionality vs integrate new functionality individually. Additionally, EJB 3.0 (or Java EE) is not really the framework that you can pick. You can also look into other framework, such as Spring.
My suggestion is to really figure what your system requirements, and then pick the right technologies instead of picking up the coolest technologies first.
Good luck