I've modified existing Even Scheduler code in Apache Storm. Later how do I compile and execute this scheduler on a benchmark program and check if there are any changes in metrics?
What config files have to be changed?
Please provide some helpful documents with step-by-step explanation.
There's a step by step article here https://inside.edited.com/taking-control-of-your-apache-storm-cluster-with-tag-aware-scheduling-b60aaaa5e37e.
Relevant part:
To wrap things up, we should also cover how the class is meant to be
actually used by Storm. There are two things we need to do for that:
Instruct storm to use our scheduler instead of its default scheduler, via config.
Package our class in a .jar file and add it to the $STORM_HOME/lib directory for Storm to be able to access it.
The config necessary is again very simple; similarly to the tag
scheduling metadata, we just have to add this line to
$STORM_HOME/conf/storm.yaml(and make sure the nimbus process is
restarted):
storm.scheduler: com.edited.tagawarescheduler.TagAwareScheduler
Related
I have a mapreduce job as a 'jar' ,that should be run daily. Also, I need to run this jar from a remote java application. How can I schedule it: i.e, I just want to run job daily from my remote java application.
I read about Oozie, but I dont think it is apt here.
Take a look at Quartz. It enables you to run a standalone java programs or run inside an web or application container (like JBoss or Apache Tomcat). There is a good integration with Spring and Spring batch in particular.
Quartz can be configured outside of the java code - in XML and the syntax is exactly like in crontab. So, I found it very handy.
äSome examples can be found here and here.
I am not clear about your requirement. You can use ssh command execution libraries in your program.
SSH library for Java
If you are running your program in linux environment itself, You can set some crontab for periodic execution.
If the trigger of your jar is your java program, then you should schedule your java program hourly rather than the jar. And if that is separate, then you can schedule your jar in Oozie workflow where you can have the java code execution in step one of oozie workflow and jar execution in the second step.
In oozie, you can pass the parameters from one level to another as well.Hope this helps.
-Dipika Harwani
I am currently developing a simple plugin that retrieves results from a Jenkins build. I am extending Notifier and using build.getResults() to get the information. However, when I upload my plugin, I can't set it as a post-build action.
When I run my builds, they break on build.getResults() since I am trying to get the results while the build is still running.
What can I do to properly get the build result ?
Best thing is to look at existing plugins which use Notifier extension point (click to expand implementing plugins list).
Check that you have the Descriptor implemenation (inner) class, as well as config.jelly. Also check jenkins.out and jenkins.err logs for any exceptions (such as malformed config.jelly).
Edit: Actually, Notifier subclass of this plugin looks really simple as Notifiers go: https://wiki.jenkins-ci.org/display/JENKINS/The+Continuous+Integration+Game+plugin , see especially its GamePublisher.java and corresponding config.jelly, and it's GameDescriptor.java, which has been made a full outer class (often descriptor is inner class). Also if you want options into Jenkins' Global configuration, you need a global.jelly, but if you don't have such options, that is something you can just leave out (unlike config.jelly, which you must have for Notifier even if it is empty, like here).
As a general note, it can be really annoying when things do not work, and you do not get any error, your stuff simply is just not displayed by Jenkins... If you just want to get things to work for you, using Groovy build step might be easier, but if you want to get things to work for others, then doing a decent full plugin reduces support requests.
Since this sounds so simple, are you sure you need a plugin ? Take a look at using a Groovy Postbuild step instead; they're much easier to write. There are some good usage examples in the link. If you decide you really need a plugin, see if you can extend an existing one rather than writing your own; it's an easier way to understand the ins and outs of Jenkins plugin writing.
I am writing an annotation processor and at a place or other I would like to just attach a debugger like in every java program. The usual
-agentlib:jdwp=transport=dt_socket,address=8787,server=y,suspend=n
is rejected. Any idea on alternative ways to attach a debugger?
Take a look on the following resources:
http://code.google.com/p/acris/wiki/AnnotationProcessing_DebuggingEclipse
http://code.google.com/p/acris/wiki/AnnotationProcessing_DebuggingMaven
Although I used to separate the processor to layers: one that is not processor dependent that can be tested using regular unit tests. I did debugging of whole system using log file...
I have to implement a Tool which reads a config file and runs application/programs specified in the configuration file. A sort of automated runner for tests. I have implemented a program which does that, however I have hit the dependency wall. In my current design Tool parses the config file and retrieves a Map of ProgramType and list of Jobs. Based on ProgramType, a Runner class is chosen and initialized, runner class is a part of Program src.
//this is just a pseudo code
pkg org.Tool
class Tool {
//after parsing
runJobs(map) {
if(map.get() == ProgramType)
org.Tool.JobStats = org.ProgramType.Runner.run(Job)
}
}
pkg org.ProgramType
class Runner {
org.Tool.JobStats run(Job) {
if(Job = "certain job")
return CertainJob.run(Job)
}
}
To build the Tool I need compiled org.ProgramType.*; to build the ProgramType I need org.Tool.Job and org.Tool.JobStats. The "dependency hell" I have created is obviously very bad design. I came with a solution of simply invoking ProgramType jar and storing the JobStats in a jobStats.txt file and once the jar execution finishes, read the file and process it. This solution is not acceptable as Jobs can be run multiple times with many configurations etc, simply to much *.txt files to handle. I think I saw once a compile solution for my problem, something like "partial-compile" Tool, compile ProgramType, recompile Tool. However I can't find it, also it would be wise to get rid of the "dependency hell" anti-pattern. Thus my question "How I should have designed this".
(I hope that the explanation is clear, if not just ask)
SOLVED
As I wrote in the comment #aviad I was looking for a design pattern, to solve my problem. I found one its called "Dependency Inversion Principle". Here I am linking a pdf document describing the pattern, its worth reading (http://www.objectmentor.com/resources/articles/dip.pdf) (Another good explanation http://java.dzone.com/articles/fun-modules). Thank you all for your help (I really liked the frameworks you have recomended).
I think you still can use most of your code if you can use the SpringBatch framework.
The advantage of using Spring batch is that it is Framework where all the hard job was already done (configuration, multi-threading, persistency etc).
Basically the java programs that you need to run can be executed by running a batch file\shell script that starts:
java -cp %JOB_CLASSPATH% %JOB_MAIN_CLASS_FULLY_QUALIFIED_CLASS_NAME%
So I would use the following design:
The entire solution is Spring -based application.
Controller class (your main class) that reads configuration and runs the show. Controller class populates the job queue with jobs and their configurations. Then every job is retrieved from the queue and Spring batch job is instantiated programmatically and run.
You can also use Commons Exec for executing batch files\shell scripts from java.
Later you can move to run in web container (In order to enable configuration changes and jobs triggering over http) - check Spring Batch Admin
You can also run your solution in sceduled manner if you use Quartz FW
Good luck!
Using JAVA framework i want to achieve the following task.
hot code(jar) deployment which will perform certain task in an environment
At any time if i update that jar file it should automatically unload old code and load new code
I want to schedule that deployed jar file to perform tasks.
Currently i see Apache karaf/Felix fulfill this requirement but less help available and difficult to manage.
Any alternate framwork i can use instead of using karaf/felix ?
If you aren't going to go the OSGi route, which you basically implied by forgoing Karaf / Felix (and Karaf uses Equinox, by default) then about the best thing I can suggest for you to consider is LiveRebel when it comes out. #Daniel's answer mentioned JRebel, which is outstanding for hot deployment during development but it is not meant as a tool for production systems. Instead you should check out LiveRebel, also made by Zero Turnaround, might be able to fulfill of your needs. Please note that this is a commercial product but they are offering a private beta right now.
[Edit]
Idiotically, I forgot to mention that there's also Knoplerfish, another OSGI runtime which has a BSD style license. Perhaps give that a shot?
Give JRebel a try. It is a great tool.
Note sure what environment you mean (eg. web, desktop, server-side, etc), but...
Working backwards:
3: Scheduled Tasks
You can achieve this in any Java container with the Quartz Scheduler library. This allows you to schedule events in a CRON like fashion.
1-2: Hot Deployment
Then it's a question of where you want to deploy and how to handle hot deployment. Other answers have mentioned JRebel and OSGI which will work. If you want some super quick deployment (eg. save the code and it's available) and have it hosted in a web container ,then use the Play Framework. It uses Quartz do implement Scheduled Jobs in a very nice way.
For example (from the Play docs) :
#Every("1h")
public class Bootstrap extends Job {
public void doJob() {
List<User> newUsers = User.find("newAccount = true").fetch();
for(User user : newUsers) {
Notifier.sayWelcome(user);
}
}
}
JBoss has the hot deploy feature that your describing. However, I'm guessing it's as complicated to configure Karaf. It may be possible to find out how JBoss is achieving it and use the libraries yourself though.
hot code(jar) deployment which will perform certain task in an
environment
At any time if i update that jar file it should automatically unload
old code and load new code
I want to schedule that deployed jar file to perform tasks.
In a nutshell, hot deploy/redeploy is done like that
Use a classloader (java.net.URLClassLoader is a good start), load the jar(s), actually copy the jar somewhere (temp) before loading it
You need some interface implementation, instantiate the class implementing the interface (META-INF in the jar, custom xml, whatever), configure it (props/xml, whatever)
call start() and perform the tasks.
Monitor the jar: some thread to check it each second and compare the last modified time/size
If changed - call stop() and undeploy, may need to wait for threads, etc, start over
There are a lot of frameworks that allow dynamic deploy
The hotdeploy feature of most web containers (like Tomcat or Jetty) allow you to have the behaviour you want, on web applications.
Such an application can be very simple, and essentially just contain your jar.
What is it you need your application to do?