Maven error while assembling hadoop in stand alone mode - java

I am new to hadoop and maven. I will like to compile the hadoop 2.0.3 from the source and install it. I am following instructions from
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html
So far, i have managed to download hadoop source code and from the source directory issued "mvn clean install -Pnative"
Next i tried to execute mvn assembly:assembly, but i get following error:
Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.3:assembly (default-cli) on project hadoop-main: Error reading assemblies: No assembly descriptors found. -> [Help 1]
Please help so that i can move forward.
Also, the above mentioned install link, does not mention what should be the value of "$HADOOP_COMMON_HOME/$HADOOP_HDFS_HOME"

I compiled 1.0.4 just as an academic exercise. Not sure if it will be valid for 2.0.3
This should be done (on Ubuntu) before you start compilation to make sure that all needed stuff is there:
sudo apt-get -y install maven build-essential protobuf-compiler autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev
I did not had subversion so I did this too:
sudo apt-get install subversion
After that I checked out the code:
svn checkout http://svn.apache.org/repos/asf/hadoop/common/tags/release-1.0.4/ hadoop-common-1.0.4
Then went to newly created folder “hadoop-common-1.0.4″ and gave command:
ant clean package
You can refer to my blog for the whole story:
http://hadoopmagic.wordpress.com/

Related

Maven jar:jar not gathering dependencies

I am trying to use semantic-release within a GitLab CI pipeline. I have the prepare stage working fine, but the publish stage always fails when I use anything other than mvn jar:jar deploy:deploy, but when i use those commands it deploys a jar that is 3kb big instead of a jar that is 10mb. So i can only assume that it is not gathering dependencies. There was a WARNING message about no files being marked for inclusion and the jar being empty. So I tried to package the project before calling deploy. It did not work.
The pipeline fails with no reason as to why. It just show that line as the culprit.
commands I have tried:
mvn clean install
mvn clean package deploy
mvn jar:jar deploy:deploy
mvn clean deploy:deploy
.. you get the idea.
Here is the prepare section that works:
verifyConditions:
- "#semantic-release/changelog"
- "#semantic-release/gitlab"
- "#semantic-release/git"
verifyRelease:
- path: "#semantic-release/exec"
cmd: echo -e "VERSION=${nextRelease.version}\nNEW_RELEASE=true" > RELEASE.env
prepare:
- path: "#semantic-release/exec"
cmd: if [ ! -d ".m2" ]; then mkdir .m2; cd .m2; touch settings.xml; echo $MVN_SETTINGS | base64 -d > 'settings.xml'; cd ..; fi; mvn versions:set -DnewVersion=${nextRelease.version} -B -gs .m2/settings.xml;
- "#semantic-release/changelog"
And here is the publish section that only works with jar:jar deploy:deploy but does not create the correct jar.
publish:
- "#semantic-release/gitlab"
- path: "#semantic-release/exec"
cmd: if [ ! -d ".m2" ]; then mkdir .m2; cd .m2; touch settings.xml; echo $MVN_SETTINGS | base64 -d > 'settings.xml'; cd ..; fi; mvn versions:set -DnewVersion=${nextRelease.version} -DremoveSnapshot=true clean deploy -B -gs .m2/settings.xml;
I'm extremely new to this, and I cannot see why:
1) trying clean deploy is causing this to fail and jar:jar deploy:deploy doesn't
2) how I can get semantic-release to create a jar with all dependencies for upload to our repository.
I should note that both Maven Shade plugin and Maven Deploy plugin are present in my pom.
This is an older run, but they all are formatted like this and tell you nothing about WHY it failed. Just that it did:
stderr: '/bin/sh: line 1: 425 Killed mvn clean deploy -B -gs .m2/settings.xml\n',
failed: true,
signal: null,
cmd: '/bin/sh -c mvn $MAVEN_CLI_OPTS versions:set -DremoveSnapshot; mvn clean deploy -B -gs .m2/settings.xml',
timedOut: false,
killed: false,
pluginName: '#semantic-release/exec' }ERROR: Job failed: command terminated with exit code 1
First of all, for deployment use mvn clean deploy. The other combinations you presented do not produce sensible output.
If you want to package the dependencies into your jar, you need to properly configure the Maven shade plugin (configuring the deploy plugin is usually not necessary). Without your pom.xml, I can only guess, but I would say that the error is probably in that configuration.
BTW: Only put the dependencies into the jar if the jar is meant to run standalone. If, on the other hand, you write a Java library to be used as dependency, don't do it.

CAN'T Build Apache spark with MAVEN

I have downloaded Apache spark and trying to build it with MAVEN as suggested here. http://spark.apache.org/docs/1.0.0/building-with-maven.html
But I am not able to resolve the error after running the command -
build/mvn -DskipTests clean package run , the error is- build/mvn: No such file or directory .
I checked by running mvn -v and also JAVA_HOME is set to the JDK.(screen shot attached ) .
Please help to resolve the problem.command promt output
mvn -DskipTests clean package run
You don't need to use build/mvn. I am assuming you have mvn installed somewhere within the system.

Build multiple maven pom files and log in to a single file

I am working on a Java project. In my project multiple dependency projects are these. So I created a bat file to build all projects one by one. Please see the logic I used to achieve this.
set x=proj1-prx
set y=proj2-prx
set z=proj3-prx
set LIST=(%x% %y% %z% )
echo Checkout and deploy started
for %%G in %LIST% do (
set _module=%%G
set _value=!%%G!
echo Checkout module - %%G
svn checkout %SVNHOST%/%%G/%REPO% %WORKSPACE%\%%G\%REPO% --username %USER% --password %PASSWORD%
echo Install module to AEM - %%G
mvn clean install -Dskiptests -f %WORKSPACE%\%%G\%REPO%\pom.xml -l output.log
#ECHO OFF
)
echo Checkout and deploy finished
This file is executing well , Log file also creating but each time for loop building a project the build result overrides in to log file. I want build result of all project. Please help me friends
I want build result of all project. Please help me friends
You cannot depend on mavens logger implementation. Use the OS stdout/stderr redirection:
at the beginning of your batch file delete the old log content:
echo > output.log
then change the maven call:
mvn clean install -Dskiptests -f %WORKSPACE%\%%G\%REPO%\pom.xml >> output.log 2>&1

gradlew command not found?

I am working on a Java project with Gradle Wrapper (gradlew). I use Ubuntu Linux as my OS. When I run "gradle" it runs, and gives me information. But when I run "gradlew", it outputs as:
No command 'gradlew' found, did you mean:
Command 'gradle' from package 'gradle' (universe)
gradlew: command not found"
I did my research, I have the JDK, and I did sudo apt-get install gradle. How can I fix it?
The error is:
gradlew clean jpackage
Output:
bash: gradlew: command not found...
Linux and macOS
As noted in the comments, just running
./gradlew
worked for me. Adding the ./ tells it to look in the current directory since it isn't in the path.
Windows PowerShell
.\gradlew
The Gradle wrapper needs to be built. Try running gradle wrapper --gradle-version 2.13.
Remember to change 2.13 to your Gradle version number. After running this command, you should see new scripts added to your project folder. You should be able to run the wrapper with ./gradlew build to build your code.
Please refer to this guide for more information: Building Java Projects with Gradle
Running this Bash command works for me by running chmod 755 gradlew as sometimes file properties changed upon moving from one OS to another (Windows, Linux and Mac).
If you are using Mac, try giving root access to gradlew by doing:
chmod +x ./gradlew
From Mac,
Nothing is working except the following command:
chmod 777 gradlew
Then
./gradlew
The same problem occurs to me...
I check the file wrx permissions with:
$ls -l ./gradlew -> -rw-rw-r-- (no execute permission)
So I use command $chmod +x ./gradlew and this problem is solved.
In addition to Suragch's answer:
Linux and macOS
./gradlew clean
Windows PowerShell
.\gradlew clean
Windows cmd
gradlew clean
You must have the Gradle wrapper available locally before using gradlew. To construct that
gradle wrapper # --gradle-version v.xy
Optionally, pass the Gradle version explicitly. This step produces the gradlew binary. And then you should be able to
./gradlew build
For Ubuntu (Linux) users:
Doing bash ./gradlew build works, but ./gradlew build does not work.
For me, the issue was it was on the NTFS file system, and Linux does not let you execute a script from NTFS.
Try moving the code from NTFS to a Linux partition. Then ./gradlew build should work.
If you are using Visual Studio Code with Flutter, you should find it in your app folder, under the android folder:
C:\myappFolder\android
You can run this in the terminal:
./gradlew signingReport
The first thing is you need to run the gradle task that you mentioned for this wrapper. Example:
gradle wrapper
After running this command, check your directory for the gradlew and gradlew.bat files. gradlew is the shell script file, and it can be used in Linux and macOS. gradlew.bat is the batch file for the Windows OS. Then run,
./gradlew build (Linux and Mac). It will work.
Issue: Couldn't find gradlew at path jenkins
In my case, within the Jenkins CI for flutter project, I have to first run the flutter build app command, and then it automatically generated a gradlew file. And the above issue is resolved.
I put this command in my Jenkins file:
flutter build apk
In a Flutter project, don't forget to go to 'android' folder with 'cd android'.
Then you can run a command like './gradlew build' or './gradlew clean' on it (macOS).
If the answer marked as correct does not work, it is because you need to identify yourself as a super user.
sudo gradle wrapper --gradle-version 2.13
It worked for me.
I faced the same issue, but after some tries I found
it was happening because I was trying to create build in Git Bash,
instead of CMD with system administrator access.
If you create build with a command prompt, run as administrator. Then the build will get created.
If you are trying to run this command for a Flutter app, then go to the android folder first by cd android. And then use the command, and it should work.
cd android
./gradlew signingReport
I use IntelliJ IDEA and in Windows in the terminal I type:
gradlew.bat run
It is working for me.
Instead of gradlew assembleRelease, use ./gradlew assembleRelease.
I had to do dos2unix * in my current folder to make it work.
Ubuntu
Error:
Command 'gradlew' not found, did you mean:
command 'gradle' from snap gradle (7.2)
command 'gradle' from deb gradle (4.4.1-13)
See 'snap info ' for additional versions.
Use this command:
./gradlew signingReport

Switch maven version - mvn command does not get bound

I'm trying to switch maven from 2 to 3 (on Linux) using:
sudo update-alternatives --set mvn /path/to/maven3/bin/mvn
Anyway mvn -v still gives version 2, so I always have to execute /path/to/maven3/bin/mvn to use maven. How can I rebind the mvn command to the appropriate maven path?
Run:
sudo update-alternatives --config mvn
Then choose which version you want use.
Also check than which mvn is symbolic link to /etc/alternatives/mvn
which mvn
And then recreate symbolic link to point on new Maven version. And verify than environment variable MAVEN_HOME points to the right directory.
Set system property M2_HOME to path to the maven home.

Categories