i am working on hadoop but i am not able to understand how we can configure to eclipse and which software install to run hadoop program .in single machine in windows 7.
is there any plugin is available to run hadoop program .
can you please suggest me any link detailed information how can i rum hadoop program.
i refer many sides but i am not getting exact information how can i run program .
is there any side getting all information about the hadoop.program running in eclipse
and how can i map and reduce the program .
Related
I have few Selenium Java Projects that I wanted to run through Jenkins on a specific time.
I run Jenkins from a remote desktop machine and I start it using command prompt for creating a project / build using
java -jar jenkins.war
and access Jenkins using localhost.
If I wanted to run my projects every day at 6 AM,
Should I have my Virtual Machine where my Jenkins is residing on?
Should Jenkins be running all the time?
Will Jenkins automatically run the build even if it is not running on the Virtual Machine if I make the Build selecting "Run Periodically"?
Please suggest me possible solutions.
Best possible solution is thru Docker+jenkins
Have your dockerfile ready with Run command and steps, which can be done by pulling image in container with necessary config and files and combine with Jenkins server as well, once the run is complete it cleans up the container and it is easy to maintain. All you need is to play with docker to achieve the above scenario.
I don't understand your question complete. Maybe you should improve it a bit.
You access a VM via remote desktop. On this VM you have a jenkins server running. I assume your VM is running 24/7. So no need to ever stop jenkins. And how you confiure your time, you'll find in this answers.
I have an Azure App Service running Tomcat 8.5 and can't seem to figure out how to do a thread dump. jstack is not recognized in the Console nor the DebugConsole.
On Azure App Service, all popular versions of Java had been installed at the path D:\Program Files (x86)\Java. Take version 1.8.0u73 as example, please see the figure below.
You can command set PATH=D:\Program Files (x86)\Java\jdk1.8.0_73\bin;%PATH% to add Java tools to the environment temporarily via the Kudu console tool. Then, the jstack tool is available in the current Kudu session.
Inspired by the previous answer, I created the following bat script (could be powershell, I suppose) that does the job:
powershell -Command "get-process java |select -expand id" > pid.txt
set /p id= < pid.txt
"%JAVA_HOME%\bin\jstack" -F %id% > out.txt
It uses the java version that you're using in your app (JAVA_HOME) and doesn't require any input from you. It's simple and does the trick. Just create a bat file in your wwwroot, for example, and run it whenever you want.
I know this is an old question but for Windows Apps on Azure App Service, this can be easily done by going to Azure Portal -> Diagnose and Solve problems blade -> Diagnostic Tools and then choosing Collect Java Thread dump from the left menu or the middle pane.
Get the PID of java process from Process Explorer of Kudu (Advanced Tools)
Use this command to collect thread dump:
jcmd Thread.print > D:\home\threaddump.txt
I got stuck in the steps to deploy the play-java application to standalone server. I am using Ubuntu instance running onEc2 machine. After reading the official documentation I followed these steps:
Install typesafe activator to $Home/activator
Copy my project folder to $Home/activator/projects using Jenkins
Run cd $Home/activator/projects/project
Run dist command. It generate the project sh file
Run cd $HOME/activator/projects/sample/target/universal. It shows me three things i.e. sample-1.0.zip , tmp.
Run unzip sample-1.0.zip
Run cd $Home/activator/sample/target/universal/sample-1.0/bin. It shows me two scripts i.e. sample (for mac and linux) and smaple.bat (for windows).
Now, I tried to run this command as documentation says: sample-1.0/bin/sample -Dplay.crypto.secret="abcdefghijk" but it got stuck and shows nothing except a prompt on terminal.
What next steps are required to run the application in daemon mode. So that server will continue to run always?
Another question is why do I need to follow these steps because just after copying the project I can easily run activator run command to sample folder to run the server.
EDIT-1
Now I am getting Error: Could not find or load main class play.core.server.ProdServerStart while running sample -Dplay.crypto.secret="abcdefghijk" command
Play Framework modes
Play like other frameworks can run in different modes because we are usually like to run our program in different situation and these situations need their configuration. Production is one of play modes. In this mode as the name suggested we are ready to deploy our application where user can use.As a result requirements are different.
Some common configuration in Production mode :
We are confining https
Using production data base not testing one
Play doesn't show errors
Play framework run as long term process and if you are log out of your server your application will run
If your app fails Play will clone another application and run that so your app run forever except you turn it off.
and so on (it is based on your Production config file)
Your answer :
If you just want to test your app in remote server and see what happened, it's okay to use activator run this command runes Play app in development mode and application will killed if you exit or log off (error will shows up and so on). But if you want to use your app for Production it is necessary to run all of these steps.
I am using IntelliJ IDEA in Ubuntu 14.04 to test my hadoop program. When I chang the number of reducer, I use the following code:
job.setNumReduceTasks(3)
I use build artifacts in IDEA to build a jar file and input hadoop jar xxx.jar MyClass intput output in linux shell. The output shows 3 files (part-r-00000, part-r-00001, part-r-00002), which is completely my expectation. However, when I runs the program in IDEA for convenience using the arguments input/ output/, the output result only contains one file part-r-00000. So I am wondering where goes wrong.
When you run in local mode only one reducer will be used - there is no parallelism in local mode. Nothing is going wrong with your code here.
Also see https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html#Standalone_Operation:
Standalone Operation
By default, Hadoop is configured to run in a non-distributed mode, as a single Java process. This is useful for debugging.
I have downloaded Hadoop VM from Yahoo site and started Hadoop VM through VMware player. I saw no issues.
As been mentioned in Yahoo, I made use of Eclipse Europa (v 3.3.1) and I downloaded a jar file (hadoop-0.18.3-eclipse-plugin) to make MapReduce plugin on in eclipse.
I started linux VM with hadoop and in eclipse created new DFS location (entered IP of my VM, Map/Reduce master port 9001, DFS port 9000. But in the node I got the error "Error:null."
What I'm doing wrong? Also while configuring Hadoop variables in eclipse, i am not seeing in advanced tab while setting up hadoop location.
I'm using Eclipse Europe 3.3.1 and Hadoop 0.18.0.
Did i went wrong in choosing the version of the plugin?
Thanks for helping.
Spawn the Eclipse from within Cygwin to resolve this issue.
Go to eclipse installed folder in Cygwin and invoke eclipse.exe
Above answer helped me;
What he says is after installing cygwin
open cygwin terminal and go to "cygdrive" ( use cd .. and pwd )
go to your drive and look for eclipse.exe
then "run eclipse"
now it will show folders instead of "error:null"
What I did and worked for me was:
following the instructions in this thread:
edit this file: \workspace.metadata.plugins\org.apache.hadoop.eclipse\locations
and add a property: "hadoop.job.ugi" with value "hadoop-user,ABC"
but also modify the property " mapred.system.dir" to "/hadoop/mapred/system". As you can see in this webpage. I followed all this page for the rest of the configuration.