I am new to Java and currently working on a project where a Hadoop job needs to be triggered from Spring MVC application. The manager asked me to use "process" for which I have no clue. I have written a shell script to trigger the job but the client wants it to be triggered directly from the Spring MVC app so that log can be written in local file system.
Can anyone help me how to trigger a Hadoop jar (more specifically Yarn command with different arguments) to be triggered in edge node through Java process?
You can try using ProcessBuilder.
http://docs.oracle.com/javase/7/docs/api/java/lang/ProcessBuilder.html
Related
I have a few Java applications, mostly with Spring Boot, in different servers (Droplets) from Digital Ocean. Previously I upload each new release manually and with some shell scripts created I am able to start/stop the applications. These scripts contains environment variables when it's necessary for test or production environments. To check if the application is running I have a cronjob launched every 5 minutes, this cronjob execute a shell script to start the application if it's not running.
However I switched to automatic deployments using Bitbucket pipelines and now I won't have shell scripts anymore because all the configuration will provided by the Bitbucket pipeline and will be executed over SSH.
For instance cat environment.sh start.sh | ssh xxx#yyy where environment.sh is created on the fly by the pipeline with right configuration in that moment for each environment.
How can I do something to start the application automatically if it's not running? Should I upload the script files in each automatic deployment and use a cronjob?
You should use a wrapper. YAJSW is popular, free and works (in my experience) well in production systems. The wrapper will restart your process if it crashes and can easily be installed as a service over many different OSes without much effort.
NOTE: I am not affiliated or related in any way to YAJSW.
My spring boot application implemented using command line runner (meaning it's not an API) is executed using java -jar with bunch of command line arguments. We have been running our application manually on AWS EC2 so far and now in an attempt to Automate application execution we have started a POC using Lambda. Lambda is chosen because application must trigger at SNS event (which is for file upload event in S3).
I have configured lambda function using Java runtime and attached it to SNS topic. Lambda is successfully triggered and sending a call to my application Jar which is uploaded in lambda function through S3.
Application first step is to download the file from S3, so I am implementing LambdaHandler class using S3 event, as shown below.
public class LambdaHandler implements RequestHandler<S3Event, String> {
// Code to fetch S3 object name here which is needed in application processing logic further
}
I am unable to figure out how to initialize spring boot batch application which is implemented using command line runner, is this even possible?
Would you recommend an alternate approach (Jenkins connecting to EC2 and running a bash script to download file from S3 then wget jar file from artifactory and run java -jar build command to execute)
I have this project that involves both a client and a server. I developped both parts of the application in java and I want to test it in a hadoop cluster, since the server side is a simulation of a cloud, so by using hadoop I want to give my application a real sense of cloud environment. I started by creatin a multi-node Hadoop Cluster but I don"t know what should be the next step.
I would appreciate any enlightening.
The proper way to accomplish this would be to use a restful interface to send the commands.
For instance, on the computer that is the JobTracker, you could host a tomcat rest server. (Make sure that the hadoop dameons are running in the background). You could use a Spring/Hibernate based servlet in order to process the requests to the server. Finally, in the servlet, you could either include the hadoop-jars and call hadoop through the hadoop API, or you can call hadoop through the console (./hadoop runjar blah).
In order to upload files to the Server, you can use an sftp interface, or possibly directly upload files to the hdfs.
If you would like, I can share some code with you, because I have done a similar project.
Also, you can look into Apache Oozie. They host a restful job flow api for hadoop.
I have created a standalone java application using quartz.i want to deploy the standalone program on weblogic so as it could be run as a batch job.
Can you please help me how could i deploy the java application on the weblogic and run the same.
I think one way to do this is create new thread and do your batch job in it.
You can add your class to Weblogic classpath and use it like start or shoutdown class.
Here is link on oracle documentation.
I have a rails app that talks to an api running on the same domain via ajax calls. I want to test this app using cucumber. The api is written in java and packaged as a jar. How can I mount the jar when using cucumber?
There is no way to do it automatically but you can add Before hook into env.rb or put it into separate file and in this method you can load your java extension by issuing shell command, you can store process pid in variable and kill this process in After callbalk. You can configure Capybara to start server on specific port and I think you can tune your application to use specific port too.