Mounting of HDFS to local directory failing - java

I'm currently trying to implement mounting of hdfs to a local directory on ubuntu machine. I'm using hadoop-fuse-dfs package.
So, I'm executing this below command
ubuntu#dev:~$ hadoop-fuse-dfs dfs://localhost:8020 /mnt/hdfs
Output
INFO
/var/lib/jenkins/workspace/generic-package-ubuntu64-12-04/CDH4.5.0-Packaging-Hadoop-2013-11-20_14-31-53/hadoop-2.0.0+1518-1.cdh4.5.0.p0.24~precise/src/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/fuse_options.c:164
Adding FUSE arg /mnt/hdfs
But, when I try to access the mounted hdfs locally, I see the error message (please check the snapshot attached)
ls: cannot access /mnt/hdfs: No such file or directory
total 4.0K
d????????? ? ? ? ? ? hdfs
PS : I've already executed following commands, but still I get same output.
$ sudo adduser ubuntu fuse
$ sudo addgroup ubuntu fuse
Am I missing something ? Please suggest some workaround.

You need to use hostname instead of localhost. I faced the same issue, after changing localhost to hostname which is also defined in hosts file, it got fixed.
hadoop-fuse-dfs dfs://{hostname}:8020 /mnt/hdfs
According to Cloudera
In an HA deployment, use the HDFS nameservice instead of the NameNode URI; that is, use the value of dfs.nameservices in hdfs-site.xml.

This happens at least when hadoop-fuse-dfs can not connect to filesystem metadata operations running by default on port 8020 e.g. due to network configuration issues.
You can test from your host that connection works prior running hadoop-fuse-dfs e.g. by
telnet your-name-node 8020
GET /

Related

Couldn't resolve host within docker container and java process

I am facing an issue running a docker image based on alpine linux that runs a java process (GoCD server). The java process itself tries to run some code to clone a git repository locally however I am getting a Couldn't resolve host exception. I tried to manually clone the git repo from within the container using 'git clone' and had no issue.
In addition, I am able to ping the domain from within the docker container by running ping my-service-url.com with no issues and also from the host machine with the same result. It seems java has some difficulties resolving the name but I am not sure how to fix it.
I tried creating a new image with RUN echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf. I read in another question that would fix it but no luck. The whole Dockerfile looks like the following:
FROM gocd/gocd-server:v17.5.0
RUN echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf
ADD json-config-plugin-0.2.jar /godata/plugins/external/json-config-plugin-0.2.jar
ENTRYPOINT ["/docker-entrypoint.sh"]
I am running on AWS ECS, ipv4 forwarding is set to 1 and network mode is bridge. I am running out of ideas.

Unable to open debugger port through IntelliJ

I've got a server running on DigitalOcean and a JAR file that I want to debug. I first start the JAR on the remote server using
java -jar Server.jar -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005
but on the console I see no output like "listening on port 5005...".
When I press debug in IntelliJ it says
unable to open debugger port (198.xxx.xxx.xx:5005):
java.net.ConnectException "Connection refused"
This is my IntelliJ configuration:
I also tried using -Xdebug but it still didn't work.
If I set suspend=y it should wait until a debugger is connected, but instead, it starts without problems.
The command to start the remote Java process in debug mode looks correct. If you don't see "Listening to Port blah" when you start the server JAR, then it might mean that the debug args are not being picked up. Another way to quickly check this would be to test with a telnet localhost 5005 on the machine where the server JAR is being executed. The telnet will fail if that port is not being used.
I suggest that you try the following, since the order of the parameters might be significant (I'll add some official evidence for this later):
java "agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005" -jar Server.jar
this command worked for me:
export JAVA_OPTS='-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=0.0.0.0:5005'
by default idea remote dialog suggest:
'agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005'
change it to:
'agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=0.0.0.0:5005'
and issues port 5005.
This command worked for me:
java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005 -jar Server.jar
Thanks to suifengpiao14 but I'm going to describe the problem a bit more in detail.
I checked multiple things, and at the end found the reason: actually as like as a restful service we want to be accessible from out of the server we are running it that we should set 0.0.0.0 as the nameserver, here we should do a similar one.
I checked the difference between the server from which I can remotely debug and the one which I can't. using netstat command:
for the server which I was ok with:
tcp 0 0 0.0.0.0:5005 0.0.0.0:* LISTEN 8323/java
for the server which I had problem with:
tcp 0 0 127.0.0.1:5005 0.0.0.0:* LISTEN 8323/java
So, using below jvm options should be helpful:
'agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005'
'agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=0.0.0.0:5005'
In my case it was because under settings -> build, execution, deployment -> debugger I had a built in server running on the same port as which I was trying to attach my debugger to for some reason.
For people like me who sometimes forget to read...
Copy and paste the arguments to the command line when JVM is started
says the Run/Debug Configuration in IntelliJ, directly under:
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005
Means:
Copy this line and go to your docker configuration. Add an environment variable (modify options dropdown). Paste it there with JAVA_OPTS= prepended.
Now when you did every correctly, you will have
Listening for transport dt_socket at address: 5005
and
Command line argument: -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005
This is who I solved it...
I've reproduced similar issue:
Unable to open debugger port (localhost:5005): java.net.ConnectException "Connection refused (Connection refused)"
I had it while running debugger w/ command line arguments for remote JVM using Run/Debug configurations:
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005
by attaching IDE with exposed debug agent server.
Since I've used Kubernetes cluster, the reason was:
once the pod has been created and was running as expected, I needed to proceed to set up port forwarding with the kubectl port-forward command, with the port to expose locally for traffic from the application before running debugger.
Based on kubectl port-forward syntax:
kubectl port-forward <resource-type/resource-name> [local_port]:<pod_port>
In format like:
kubectl port-forward <pod-name> -n <namespace> 5005:5005
or in a shorter form:
kubectl port-forward <pod-name> 5005:5005
The client listens on port 5000 locally and forwards to 5000 in the
pod.
Accordingly entrypoint.sh file of the service was configured w/ the following command for Kubernetes cluster:
java -Djava.security.egd.=file:/dev/./urandom -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -Xmx400m -jar /service.jar
As result, running port-forward before debugger solved the issue.
One of the reason is that the port is not enabled. You can try by enabling the port using the below command
firewall-cmd --zone=public --permanent --add-port=8001/tcp
Once the port is up, restart tomcat & try connecting again.
Hope it helps.
This might help someone That port in JVM debug is not your web app port

Spring boot on Aws ec2 instance

Spring boot demo app tried on localhost and successfully run the application. I even installed
gradle,jdk and other dependencies
easily on aws instance . Even though i ran ./gradlew bootRun of sample application successfully .
Like http:// localhost:8080 aws instane isn't working according to my path like http://myip:8080.
Let me know what exactly i am doing wrong with this and also to make sure i added custom tcp rule such as 8080 .
Besides adding custom tcp rule 8080, try to add the following rule into your security group.
Type: Custom ICMP Rule
Protocol: Echo Request
Port Range: N/A
Source: 0.0.0.0/0
And then try to ping YourIP in order to see if your EC2 machine can be reach from internet. If your EC2 ip is pingable, then I guess the reason why you cannot access http://myip:8080 is your spring boot failed to start in EC2.
You can ssh into your ec2 and investigate your jar. I don't know any clever method but my method is:
After ssh into the ec2 machine,
Find your jar name by ps -A -F . In my case I can find java -jar application.jar
Find your jar path by sudo find / -name application.jar . In my case I can find /var/app/current/application.jar
Then you can cd /var/app/current/ and java -jar application.jar . If you are lucky, you can find some hints from the spring boot log.
In order for this call to work there are a couple of prerequisites.I will try to make a short list, other things could arise depending on the environment.
Make sure the public IP address is the same as the one you know. This changes after each reboot. If you do not want this behavior you can use an elastic IP
The 8080 port is accessible from the Internet. Here are some steps to make sure this is the case: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/authorizing-access-to-an-instance.html

Local JNLP file not read for jenkins slave

I am trying to connect jenkins slave to the master.
We are allowed to have only one port (say 8888) open from the corporate firewall to talk to the master hosted on AWS. The port is being used for connecting to the jnlp process running on the master jenkins machine. Due to this reason we cannot download the slave-agent.jnlp file using http. I have manually downloaded the slave-agent.jnlp file and I am trying to give the following command to start the agent up.
java -jar slave.jar -jnlpUrl file:///oracle/app/jenkins/slave-agent.jnlp -secret 7f079707d5632d2db2501be73f1a6f5f6422b9c9fde806afd8fdd2000d5db123
I get the the following error
Failing to obtain file:/oracle/app/jenkins/slave-agent.jnlp?encrypt=true
java.io.IOException: file:/oracle/app/jenkins/slave-agent.jnlp?encrypt=true doesn't look like a JNLP file; content type was null
at hudson.remoting.Launcher.parseJnlpArguments(Launcher.java:301)
at hudson.remoting.Launcher.run(Launcher.java:218)
at hudson.remoting.Launcher.main(Launcher.java:192)
Waiting 10 seconds before retry
I have tried this as well without any luck.
java -jar slave.jar -jnlpUrl file://localhost/oracle/app/jenkins/slave-agent.jnlp -secret 7f079707d5632d2db2501be73f1a6f5f6422b9c9fde806afd8fdd2000d5db123
The file is present at the location has read write permissions.
Any clue on why the java process is not able to read that file?
Answer by #gareth_bowles is wrong at least today, the url will work as a file url as well. Don't know if this was changed at some point.
For our windows slave it is "file:/D:/jenkins-slave/slave-agent.jnlp", which works.
The jnlpUrl parameter needs to be an http or https link to your Jenkins master, e.g. http://jenkins.yourco.com/computer/slavename/slave-agent.jnlp -it won't work as a file URL.

JBoss AS7 only works from http://127.0.0.1:8080 on Linux

JBoss AS7 only works from 127.0.0.1:8080 on Linux.
I download jboss-as-7.1.1 and untar it.
I then run bin/standalone.sh and it starts and if I try to access it from 127.0.0.1:8080 it works great.
but If I try to access it with my computers IP 10.52.206.226:8080/ it does not work.
My Question is:
How can I give other computers access to it?
How can I tell Jboss to take requested from all address?
You have to bind server to your public address. Start your server with command
bin/standalone.sh -b 10.52.206.226
and binding to all addresses should work too
bin/standalone.sh -b 0.0.0.0
you can change your hosts file as
$ cd /etc
$ sudo gedit hosts
just add your ip address as given below:
127.0.0.1 localhost
127.0.1.1 someone-pc
192.168.18.75 localhost
Use your pc's ip(yourIP) address instead of 192.168.18.75
May it works well.
Now if you try to access
[ http://yourIP:8080]
it will be fine.

Categories