I have to implement a Tool which reads a config file and runs application/programs specified in the configuration file. A sort of automated runner for tests. I have implemented a program which does that, however I have hit the dependency wall. In my current design Tool parses the config file and retrieves a Map of ProgramType and list of Jobs. Based on ProgramType, a Runner class is chosen and initialized, runner class is a part of Program src.
//this is just a pseudo code
pkg org.Tool
class Tool {
//after parsing
runJobs(map) {
if(map.get() == ProgramType)
org.Tool.JobStats = org.ProgramType.Runner.run(Job)
}
}
pkg org.ProgramType
class Runner {
org.Tool.JobStats run(Job) {
if(Job = "certain job")
return CertainJob.run(Job)
}
}
To build the Tool I need compiled org.ProgramType.*; to build the ProgramType I need org.Tool.Job and org.Tool.JobStats. The "dependency hell" I have created is obviously very bad design. I came with a solution of simply invoking ProgramType jar and storing the JobStats in a jobStats.txt file and once the jar execution finishes, read the file and process it. This solution is not acceptable as Jobs can be run multiple times with many configurations etc, simply to much *.txt files to handle. I think I saw once a compile solution for my problem, something like "partial-compile" Tool, compile ProgramType, recompile Tool. However I can't find it, also it would be wise to get rid of the "dependency hell" anti-pattern. Thus my question "How I should have designed this".
(I hope that the explanation is clear, if not just ask)
SOLVED
As I wrote in the comment #aviad I was looking for a design pattern, to solve my problem. I found one its called "Dependency Inversion Principle". Here I am linking a pdf document describing the pattern, its worth reading (http://www.objectmentor.com/resources/articles/dip.pdf) (Another good explanation http://java.dzone.com/articles/fun-modules). Thank you all for your help (I really liked the frameworks you have recomended).
I think you still can use most of your code if you can use the SpringBatch framework.
The advantage of using Spring batch is that it is Framework where all the hard job was already done (configuration, multi-threading, persistency etc).
Basically the java programs that you need to run can be executed by running a batch file\shell script that starts:
java -cp %JOB_CLASSPATH% %JOB_MAIN_CLASS_FULLY_QUALIFIED_CLASS_NAME%
So I would use the following design:
The entire solution is Spring -based application.
Controller class (your main class) that reads configuration and runs the show. Controller class populates the job queue with jobs and their configurations. Then every job is retrieved from the queue and Spring batch job is instantiated programmatically and run.
You can also use Commons Exec for executing batch files\shell scripts from java.
Later you can move to run in web container (In order to enable configuration changes and jobs triggering over http) - check Spring Batch Admin
You can also run your solution in sceduled manner if you use Quartz FW
Good luck!
Related
We rely heavily on logic in our Junit runner and currently call into it with mvn test
The logic mainly consists of
Instantiating a RuntimeHook and binding it to Runner.Builder().hook if an appropriate JVM switch is specified on the command line.
Calling Runner.Builder().clientFactory() with a mock, again if an appropriate JVM switch is specified on the command line.
Last setting minor things like Runner.Builder().tags and Runner.Builder().path
All of this works perfect today. Our main gripe however is Maven is slow.
Is there a way to accomplish the above logic and run without the expensive build time of Maven?
Since you are mixing a bit of Java code, I think you are going to depend on Maven for build + dependencies. That said, there are ways to ask Maven to dump all JAR dependencies into a text file. For example:
mvn dependency:build-classpath -Dmdep.includeScope=test -Dmdep.outputFile=classpath.txt
And then you may be able to shape a command that uses the java binary directly and you probably already know that com.intuit.karate.Main supports all the extension things. The bad news is I don't think we support things like the custom HTTP Client yet, you can see this PR where someone did this for the RuntimeHook. Perhaps you can contribute. Or maybe you have logic in Java code anyway, so calling the Karate Runner directly may be the way to go.
I think maybe the solution you will land on is mvnd - I haven't used it yet, but I'm hearing very good things on Twitter from some experienced Java folks I follow.
I've modified existing Even Scheduler code in Apache Storm. Later how do I compile and execute this scheduler on a benchmark program and check if there are any changes in metrics?
What config files have to be changed?
Please provide some helpful documents with step-by-step explanation.
There's a step by step article here https://inside.edited.com/taking-control-of-your-apache-storm-cluster-with-tag-aware-scheduling-b60aaaa5e37e.
Relevant part:
To wrap things up, we should also cover how the class is meant to be
actually used by Storm. There are two things we need to do for that:
Instruct storm to use our scheduler instead of its default scheduler, via config.
Package our class in a .jar file and add it to the $STORM_HOME/lib directory for Storm to be able to access it.
The config necessary is again very simple; similarly to the tag
scheduling metadata, we just have to add this line to
$STORM_HOME/conf/storm.yaml(and make sure the nimbus process is
restarted):
storm.scheduler: com.edited.tagawarescheduler.TagAwareScheduler
So Im currently in a project where we are using Java playframework 2.3.7 with activator.
One of the things I liked about playframework is the hot-reloading feature. I can modify java files save and the changes are compiled and refreshed on runtime.
How do I get that functionality but for testing? I want to be able to run a single test with this hot reloading feature, so that when I save. Tests for the given file (specified by test-only) is re-runned automatically.
There is not such a solution, however you have two choices:
Use IntellJ: To re-run the previous test(s) in IntellJ, you press shift + F10.
Write a watcher: Write a file/directory watcher such as this question/answer here, and then as soon as there are changes, the program, re-runs the test command, such as sbt clean compile test or activator compile test.
Little advice auto running tests: I don't know how complicated your application is, but as soon as you have couple of injections here and there and with additional concurrency; you do not want to run the tests as soon as you put a char in.
Little advice on Test Driven Development: Your approach should be the other way around! You write a test, which fails because there is no implementation; then you leave it alone. You go and write the implementation, then rerun the test to pass it or to get a feedback. Again, you need your cpu/memory power to focus on one thing, you don't want to brute force your implementation. Hope this makes sense!.
Little advice on your Play version: The Play 2.6 is way much better than Play 2.3; you should slowly but surely update your application; at least for the sake of security.
Ok so I found what I was looking for.
For anybody in need of this particular feature in that particular version of play (I'm not sure about other versions) what you need to do is really simple. run activator and put the ~ prefix before test. for example
#activator
[my-cool-project]~test
That will reload your tests when you make a change. if you want to do this for a particular test then you have to do the same but with test-only
#activator
[my-cool-project]~test-only MyCoolTest
hope it helps anyone looking for the same thing
This is great, but what if the class in the uncompiled source should inherit from a project specific class (which is already loaded), and has other project dependencies?
As an example, say I want to give users of my software the ability to customize a class at run-time. I have an abstract public class Customizable and a custom class StatusDetails in my project, and lets say the user writes code in a file that looks like this:
import com.somepackage.util.StatusDetails;
public class Test extends Customizable {
public Test(){
System.out.println("Initializing Test");
}
#Override
public StatusDetails getStatus(Object params){
StatusDetails status = new StatusDetails();
// Populate status based on params
return status;
}
}
How could I take that and instantiate it?
First the class will need to be compiled, which means you will have to ship the required JAR files and provide a classpath. To manage this, I highly recommend Apache's Maven, but the learning curve will set you back a bit. The good news is that after your learning curve, you will have a really good tool for building that is fully portable and ideally suited for managing Java projects.
Then you will need to put the output JAR file / classes on the class path of the running program and restart it.
There are other ways of going about it, each with a bit more difficulty. You can avoid the restart by using a classloader, and have the program dynamically load the class in response to a user action.
You can avoid the externalization of the build chain by putting the build chain in the program itself, by calling the compiler tools interface, compiling the program and then using the custom classloader mentioned above to incorporate it into the running program.
The problems with such an approach (as mentioned above) are many. Basically you're giving everyone a huge security hole to exploit. For example, I could "load" code that trashes the filesystem, or opens network sockets for attacking other machines.
I recommend the separate build chain and manual configuration of the load, with a required restart. It may make the user jump through a few more hoops, but it will keep the security holes from being easily exploited.
I have started an open source Scala project named omniprop. The feature I'm currently exploring is how to allow users of the project to stack JVM-styled property providers. For instance, you may wish to look up properties in java.lang.System, Lift's util.Prop, and Typesafe's Config library. Due to some other constraints/features of omniprop, I need this stack configuration to reside in a known object, so other parts of the library can retrieve properties from this stack.
In order for omniprop to work correctly, this configuration needs to be invoked by the user of the library before any properties are accessed. Hence any project which uses my library will need a bootstrap/initializer which can set up the provider stack. An example of this initial configuration code looks like this:
import com.joescii.omniprop.providers._
PropertyProviders.configure(List(
SystemPropertyProvider,
LiftPropsProvider
))
The challenge I'm facing is with testing in particular. In order for the test code for a project utilizing omniprop to work, it must somehow run the above code before any tests are run. Currently, I don't see a clean way to do this with sbt or any of the testing libraries I am familiar with such as scalatest, scalacheck, or specs2. As it is, one would need to call the above snippet in every test suite, which is certainly not ideal.
Another approach this this problem is what Lift does, in which every project necessarily has a class called bootstrap.liftweb.Boot that the library invokes to set everything up. I find that to be a reasonable approach to a web framework such as Lift, but seems to be too much for a tiny property helper library.
I really have two questions here:
How can I have sbt invoke the above setup code with the correct classloader before all tests run?
More importantly, is this the best design for a library which requires initialization, or is there a better approach?
Using ScalaTest, when I've had to do something similar, I created a trait that extends BeforeAndAfterAll, which I mixed into every suite that needed it.
trait Configure extends Suite with BeforeAndAfterAll {
override def beforeAll() { PropertyProviders.configure(/*...*/); }
override def afterAll() { PropertyProviders.configure(/*...*/); }
}
You just mix it in like any other trait
trait FooSpec extends Spec with Configure {
// ...
}
You could put the initialization code in a trait constructor and have your tests extend from that trait.
Another approach is to have that configuration as the default, and if no configuration is set then it gets used the first time your library code is called. That would cover both testing and non-testing scenarios.
A hybrid approach would be to have your sbt testing configuration set a system property. If that system property is set and no configuration is has been set then the testing config will be used the first time the library code gets called.