I've recently worked on refactoring a system that processes bundles of client data. The system executes a series of steps, each of which consumes files from previous steps (and sometime in-memory data), and produces its own output, in the form of files or data. Sometimes the output data for a particular step is already available. I have to be careful to make sure that, when one step fails, we continue to run all possible steps (ones that don't depend on the failed step), so that the final output is as complete as possible. Furthermore, not all steps have to be run in all situations.
Previously, the relationships were all implicit in the structure of the code. For instance:
void processClientData() {
try {
processA();
} catch(Exception e) {
log.log(Level.SEVERE, "exception occured in step A", e);
processC(); // C doesn't depend on A, so we can still run it.
throw e;
}
processB();
processC();
//etc... for ~20 steps
}
I changed this to make the dependencies explicit, the error handling uniform, etc, by introducing Tasks:
public interface Task {
List<Task> getDependencies();
void execute(); //only called after all dependencies have been executed
}
public class TaskRunner {
public void run(Set<Task> targets) {
// run the dependencies and targets ala ANT
// make sure to run all possible tasks on the "road" to targets
// ...
}
}
This starts to feel a lot like a very watered-down version of a build system with dependency management (ANT, being most familiar to me). I don't want to pull in ANT for this kind of thing, and I certainly don't want to write out the XML.
I have my system up and running (mostly), but it still feels a bit hacked together, and I have since reflected on how much I hate to be reinventing the wheel. I would expect that this is a fairly common problem - one that has been solved many times over by people smarter than me. Alas, a few hours of googling turned up nothing
Is there a library that implements this sort of thing, without being a really heavy-weight build system? I'd also appreciate any pointers, including libraries in other languages (or even novel systems) that I should take inspiration from.
EDIT: I appreciate the suggestions (and I will give them due consideration), but I'm really NOT looking for a "build system" per se. What I am looking for is something more like the kernel of a build system, that I could just call directly from Java and use as a small, low-overhead library for doing said dependency analysis, task execution, and resulting resource management. Like I said, I have existing (working) code in pure Java, and I don't want to bring in XML and all of the baggage that comes with it, without a very compelling reason.
At its core, a build system does 3 things. It manages dependency, it test whether something is "built" or not, and it "builds" the things that aren't built.
Dependency management is little more than a simple topological sort. There rest is iterating through the tasks in dependent order, and processing them.
You can readily create something like:
BuildSystem bs = new BuildSystem();
bs.addTask(new Task1());
bs.addTask(new Task...);
bs.addTask(new TaskN());
bs.build();
public void build() {
List<Task> sortedTasks = topologicalTaskSort(tasks);
for(Task t : sortedTasks) {
if (t.needsBuilding()) {
t.execute();
}
}
}
If you have no need to externalize the list of Tasks, then there's no reason for an XML file or anything.
The topological sort allows you to simply add tasks to the list and let the system sort things out. Not a problem with 4 tasks, more of an issue with dozens of tasks.
The sort fails if it detects a cycle of dependency, so that's where you get that control.
Something like this is "too simple" to need a framework. I don't know how you're doing your dependency management now.
Take a look at jsr166 fork/join framework. It seems to me this is exactly what you're trying to accomplish.
http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ForkJoinTask.html
This is included in JDK7 but is available as a separate jar for 5 and 6. If I wasn't on my tablet I'd write a more comprehensive example. Maybe someone else can expand in the meantime.
public class DependencyTreeTask extends RecursiveAction {
private final List<DependencyTreeTask> dependencies = new ArrayList<Task>();
public void addDependency(DependencyTreeTask t) { dependencies.add(t) }
public void compute() {
invokeAll(dependencies);
}
}
...
// build tree...
DependencyTreeTask root = ...
ForkJoinPool pool = new ForkJoinPool();
pool.invoke(root);
You also have to take care if your graph is unconnected, but there are a well known set of algorithms for determining this.
I would consider writing a Maven plugin, it isn't that hard and much lighter weight because you only have to provide the relevant special logic. All the infrastructure is provided by Maven. Maven 3 would even give you things like parallel builds where your plugin supports it for free, amongst all the other things it provides.
One of the major goals of Maven 3 was a re-write to make it as easy as possible to embed the workflow engine in your own projects.
I've heard of Jenkins being used for this sort of thing in addition to it's primary 'build system' role. I only just started using Jenkins so can't say for sure whether it will do what you need. I'm impressed with it so far. It's relatively easy to use and has a lot of configuration options. There are a large number of plugins for it as well. Just run Jenkins and go to the plugins page to review the list and install them.
Your code reminds me of iwant, a java build engine I've been developing. You can declare your target definitions with dependencies using fluent java, and in addition to using it normally from a commandline (or ant script), you can also embed your build in a java program.
Related
I'm a vim/neovim user and I'm wondering whether I can simplify the task of adding imports to a file java without resorting to use intellij or language server.
So my idea was to get a list of all classes and then use this list in some way to insert an import statement.
To illustrate my idea. Say my cursor is here.
var a = List.of(1);
^
I'm invoking some vimscript and it gives me filterable list of imports that include List
Maybe something similar to hoogle?
You could write a gradle task to output all classes to a file. Eg
task dumpClasses {
inputs.files configurations.compile
outputs.file "$buildDir/dumpClasses.txt"
doLast {
file("$buildDir/dumpClasses.txt").withWriter { writer ->
configurations.compile.each { file ->
FileTree tree = file.name.endsWith('.jar')
? zipTree(file)
: (file.directory ? fileTree(file) : null)
if (tree) {
tree.matching { include '**/*.class' }.each {
writer.println(it.name)
}
}
}
}
}
}
Have a look at vim-javacomplete2. One of its features is to automatically add imports, including static imports and imports of nested classes. It searches class files automatically using Maven, Gradle, or Eclipse's .classpath file.
These days Java IDE's are so smart that using anything else is essentially wasting your time (and therefore indirectly money).
It is very similar to using Notepad instead of Word until (or vim instead of LibreOffice) until you want to have grammar checking and embedded graphics etc. Of course you can do it given enough time or elbow grease but why reinvent all the wheels on the way. Emacs had a quite usable Java development environment back in the early 2000's which almost instantly lost to Eclipse when it came out.
As an example, IntelliJ now knows how to show the type of the current point in a chain of method calls (very frequent with streams) which to say it brief is very helpful when reading the code later.
So, to get what you need you need an IDE anyway which can understand your code. Considered instead telling an IDE to launch vim on your source if you want to use the full power of vim? IntelliJ can pick up changes in the file system without being told.
What would be a good way to let 2 different separate Java programs communicate with each other?
I am not sure if I want to use OSGi (but it seems to be the only one which still gets updates; jpf and jspf are very old). I am very new to the topic Plugin Framework. Am I right that OSGi is just the description of the architecture, plugin structure etc. and that equinox and all this other thing are the real implementation, which i have to use?
I make things more precise: I want to program a Core Tool which should be able to hot-load Plugins, and bidirectionally communicate with them and a JavaFX GUI.
I usually use a class for controlling the GUI and a additional class for algorithms and so on (MVC). But I think this style is no longer helpful to structure a plugin-based Tool.
I want to use a good design pattern from the beginning, else it'll end up a mess.
The simplest solution to this problem is to use ServiceLoader (doc here). It is included in Java, and is fairly simple to use:
Load Jar file(s) at runtime
Detect classes that implement a particular interface (e.g.: your.package.YourService).
Instantiate objects from these classes.
Here is a pretty good post describing how to do so (Note: you should use the second proposal with URLCLassLoader; not extend the classpath dynamically). Also, do not forget to declare your services inside the Jar's META-INF directory:
If com.example.impl.StandardCodecs is an implementation of the
CodecSet service then its jar file also contains a file named
META-INF/services/com.example.CodecSet
This file contains the single line:
com.example.impl.StandardCodecs # Standard codecs
By choosing this approach, your core program will naturally have a handle to your plugins, so will be able to communicate with them easily. To ensure the bi-directional communication (i.e. plugins calling your core program), I would suggest to create an interface that your core program will implement, and pass that interface to your plugins.
Plugin interface:
public interface Plugin {
public void doPluginStuff(Caller caller);
}
Core program Caller interface:
public interface Caller {
public void sendBackResults(Object results);
}
Plugin implementation (in separate Jar file):
public class AParticularPlugin implements Plugin {
public void doPluginStuff(Caller caller){
caller.sendBackResults("Hello world");
}
}
Core program:
public class CoreProgram implements Caller {
public void callPlugin(URL[] urlToJarFiles){
URLClassLoader ucl = new URLClassLoader(urlToJarFiles);
ServiceLoader<Plugin> sl = ServiceLoader.load(Plugin.class, ucl);
Iterator<Plugin> plugins = sl.iterator();
while (plugins.hasNext())
plugins.next().doPluginStuff(this);
}
public void sendBackResults(Object results){
System.out.println(results.toString());
}
}
An little "answer" to your first question. It is more likely some sort of experience report.
I used the Akka Framework in some earlier projects to communicate across programm (and hardware) borders.
Akka is an Framework that uses the actor pattern and messages. It is build in scala but there is a good Java Version, too.
Give it a look: http://akka.io/
I'm still new to this, please don't hesitate to give me advice!
Greetings,
Torsten
I have a requirement where I need to put some logic to calculate the rank of some entity (for example a site user eligible for bonus prize). This logic changes very often depending upon the sales, available products, season etc. Also, different installations of the application will have different logic to calculate this rank. I am not sure where should I put this logic. If I put it in java I would have to go for frequent deployments. Getting it through webservice too doesn't looks that promising.
I heard Drools can be used in such scenarios but I have never used it in past. Any help in this regards is highly appreciated.
You should make the classes that implement this logic as decoupled as possible. Use well defined interfaces, and allow each installation to provide it's own implementation if necessary. A Dependency Injection framework, like Spring could be a great help.
Also, consider making your projects in a way that you can deploy independently API and implementation. Maven could help a lot on this.
You can place set of rules in JSON, XML file.
There will be a set of rules.
Your java program will read tags from that file to perform calculations.
For example:
<CalculateBonus>
<minVal>10</minVal>
<maxVal>30</maxVal>
</CalculateBonus>
Then java will read that file containing that rule and get all required data from it:
Int nonPremiumDiscount = 10;//got from file
Int premiumDiscount = 30;//got from file
if(person.isPremiumMember)
{
calculateDiscount(premiumDiscount,price);
}
else
{
calculateDiscount(nonPremiumDiscount,price);
}
If you design your program carefully you will be able to add new rules modify them without or with small change to the code.
That is just idea and would not work for you. But it is exactly what I am doing right now. I have different rules applying to different kind of objects in my code. So for example if I want to validate some additional objects/classes etc (Hope u get what I mean) I just add new rule to my file without even changing a code.
That is really simplified example. In my case I have more than 10 different files having aprox 100 rules each. All of them are in some way connected. So I have rules for general validation but there I include references to other rules in another file:
GeneralValidation -> ValidateEntities -> Something Else
Hoi!
The problem I discovered is pretty simple to understand but I can't find a solution.
First let me provide this little snippet.
#Deployment
public static Archive<?> createDeployableArchive () {
JavaArchive jar = ShrinkWrap.create(JavaArchive.class, "whoCares.jar");
// enable CDI
jar.addAsManifestResource(EmptyAsset.INSTANCE, ArchivePaths.create("beans.xml");
// Some persistence
jar.addAsManifestResource("test-persistence.xml", "persistence.xml");
// Now the interisting part (simplified):
jar.addClass(RegistrationService.class) // This one should be tested
.addClass(RegistrationException.class) // Will be thrown on error.
.addClass(UserDAO.class) // Used by RegService
.addClass(User.class) // JPA Entity
// ...
// ... This scenario goes without interfaces, inheritance, DTOs, different
// ... types of exceptions for different problem types... That's why the list
// ... is so concise.
// ...
.addClass(RegServiceIntegrationTest.class); // Test class must be included
return jar;
}
Every time you want to test a certain use case, for example registerUser, with arquillian you will have to collect all classes your registration process depends on and put them together in a deployable archive.
Doing this manually will cost time and produce problems and errors, for sure! There are several weak points:
Collecting: Think of a long process with many sub services, exceptions, interfaces, super classes, utility stuff and so on. You will walk over the complete flow to find them all. Honestly, that's a repetitious long term job that will be a pain the... eyes. I had to do it just a few times before I decided to rather start yelling.
Keeping your tests up to date: Imagine you involve a new sub service to your registration chain. You will have to update those damn dependencies and if something goes wrong when you run your integration tests at the end of the day there will be fun digging through sometimes incomplete exception messages (incomplete cause you only know something is missing at some point but not what exactly). If you're lucky a ClassNotFoundException occurs. One change could of course easily affect multiple tests:
1. Let UserDao throw some new fancy runtime exception.
2. Waste your life time.
Problems with adding packages: Adding packages is provided by Shrinkwrap but using it would be a bad Idea. Sometimes, after a long day, you feel lazy and just add complete packages but can you be absolutely sure every class will remain in the same package forever? Another problem is that the term "micro deployment" implies a need for compactness. Whole packages introduce overhead, okay I guess that's the smallest problem here.
How to solve this (Just unqualified thoughts)?
It is kind of banal that all required information already is available in the source code.
The best solution would be something like this:
#Deployment
public static Archive<?> createDeployableArchive () {
JavaArchive jar = ShrinkWrap.create(JavaArchive.class, "whoCares.jar");
// enable CDI
jar.addAsManifestResource(EmptyAsset.INSTANCE, ArchivePaths.create("beans.xml");
// Some persistence
jar.addAsManifestResource("test-persistence.xml", "persistence.xml");
Class<?>[] involved;
involved = Tool.findInvolvedClasses("RegistrationService.java", "registerUser");
jar.addClasses(involved);
return jar;
}
I don't think one can achieve this using reflection as it's necessary to know the exact "flow".
I bet there is a cool tool out there that could be put to an unintended use. There might of course be other ways. Does somebody have an idea? Thanks!
You could try out the Arquillian integration in JBoss Tools. You'l find some information on how to use it, in this GitHub repo; see the README. The screencasts linked there might be useful.
To get started with JBoss Tools, you can install it into an existing Eclipse Kepler (e4.3) or above installation via the JBT update site, with these instructions.
Note, this is considered experimental at the moment so it is not included in JBoss Developer Studio (the distribution that packages Eclipse and some plugins), and is available only in the community bits, i.e. JBoss Tools. We'd appreciate if you file any issues you encounter (or any new feature requests) in the JBoss Tools JIRA (please use the testing-tools component).
Is there a good Java library for taking the legwork out of writing good micro-benchmarks? I'm thinking something which can provide (with a minimum of hassle) provide text (CSV or HTML, take your pick) output of results and maybe graphs summarizing results. Ideally, it should be something that plays nicely with JUnit or equivalent, and should be simple to configure benchmarks with variable parameters.
I've looked at japex, but found it too heavyweight (25 MB of libraries to include?!) and frankly it was just a pain to work with. Virtually nonexistent documentation, mucking about with ant, XML, and paths... etc.
A few of us from the Google Collections team are in the early days of building something that satisfies your needs. Here's the code to measure how long foo() takes:
public class Benchmark1 extends SimpleBenchmark {
public void timeFoo(int reps) {
for (int i = 0; i < reps; i++) {
foo();
}
}
}
Neither the API nor the tool itself is particularly stable. We aren't even ready to receive bug reports or feature requests! If I haven't scared you off yet, I invite you to take Caliper for a spin.
Oracle now has JMH. Not only is it written by members of the JIT team (who will take out much of the legwork of writing good micro-benchmarks), but it also has other neat features like pluggable profilers (including those that will print assembly of your hotspots with per-line cpu time).
It prints out tables. Not sure about graphs. The benchmarks can be configured with variable parameters. The documentation is fairly good.
It is easy to set up and get going. I've got it integrated with JUnit, but the developers provide a Maven archetype to get started.