I’m having a bit of trouble with a simple Hadoop install. I’ve downloaded hadoop 2.4.0 and installed on a single CentOS Linux node (Virtual Machine). I’ve configured hadoop for a single node with pseudo distribution as described on the apache site (http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-common/SingleCluster.html). It starts with no issues in the logs and I can read + write files using the “hadoop fs” commands from the command line.
I’m attempting to read a file from the HDFS on a remote machine with the Java API. The machine can connect and list directory contents. It can also determine if a file exists with the code:
Path p=new Path("hdfs://test.server:9000/usr/test/test_file.txt");
FileSystem fs = FileSystem.get(new Configuration());
System.out.println(p.getName() + " exists: " + fs.exists(p));
The system prints “true” indicating it exists. However, when I attempt to read the file with:
BufferedReader br = null;
try {
Path p=new Path("hdfs://test.server:9000/usr/test/test_file.txt");
FileSystem fs = FileSystem.get(CONFIG);
System.out.println(p.getName() + " exists: " + fs.exists(p));
br=new BufferedReader(new InputStreamReader(fs.open(p)));
String line = br.readLine();
while (line != null) {
System.out.println(line);
line=br.readLine();
}
}
finally {
if(br != null) br.close();
}
this code throws the exception:
Exception in thread "main" org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-13917963-127.0.0.1-1398476189167:blk_1073741831_1007 file=/usr/test/test_file.txt
Googling gave some possible tips but all checked out. The data node is connected, active, and has enough space. The admin report from hdfs dfsadmin –report shows:
Configured Capacity: 52844687360 (49.22 GB)
Present Capacity: 48507940864 (45.18 GB)
DFS Remaining: 48507887616 (45.18 GB)
DFS Used: 53248 (52 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Datanodes available: 1 (1 total, 0 dead)
Live datanodes:
Name: 127.0.0.1:50010 (test.server)
Hostname: test.server
Decommission Status : Normal
Configured Capacity: 52844687360 (49.22 GB)
DFS Used: 53248 (52 KB)
Non DFS Used: 4336746496 (4.04 GB)
DFS Remaining: 48507887616 (45.18 GB)
DFS Used%: 0.00%
DFS Remaining%: 91.79%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Last contact: Fri Apr 25 22:16:56 PDT 2014
The client jars were copied directly from the hadoop install so no version mismatch there. I can browse the file system with my Java class and read file attributes. I just can’t read the file contents without getting the exception. If I try to write a file with the code:
FileSystem fs = null;
BufferedWriter br = null;
System.setProperty("HADOOP_USER_NAME", "root");
try {
fs = FileSystem.get(new Configuraion());
//Path p = new Path(dir, file);
Path p = new Path("hdfs://test.server:9000/usr/test/test.txt");
br = new BufferedWriter(new OutputStreamWriter(fs.create(p,true)));
br.write("Hello World");
}
finally {
if(br != null) br.close();
if(fs != null) fs.close();
}
this creates the file but doesn’t write any bytes and throws the exception:
Exception in thread "main" org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /usr/test/test.txt could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
Googling for this indicated a possible space issue but from the dfsadmin report, it seems there is plenty of space. This is a plain vanilla install and I can’t get past this issue.
The environment summary is:
SERVER:
Hadoop 2.4.0 with pseudo-distribution (http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-common/SingleCluster.html)
CentOS 6.5 Virtual Machine 64 bit server
Java 1.7.0_55
CLIENT:
Windows 8 (Virtual Machine)
Java 1.7.0_51
Any help is greatly appreciated.
Hadoop error messages are frustrating. Often they don't say what they mean and have nothing to do with the real issue. I've seen problems like this occur when the client, namenode, and datanode cannot communicate properly. In your case I would pick one of two issues:
Your cluster runs in a VM and its virtualized network access to the client is blocked.
You are not consistently using fully-qualified domain names (FQDN) that resolve identically between the client and host.
The host name "test.server" is very suspicious. Check all of the following:
Is test.server a FQDN?
Is this the name that has been used EVERYWHERE in your conf files?
Can the client and all hosts forward and reverse resolve
"test.server" and its IP address and get the same thing?
Are IP addresses being used instead of FQDN anywhere?
Is "localhost" being used anywhere?
Any inconsistency in the use of FQDN, hostname, numeric IP, and localhost must be removed. Do not ever mix them in your conf files or in your client code. Consistent use of FQDN is preferred. Consistent use of numeric IP usually also works. Use of unqualified hostname, localhost, or 127.0.0.1 cause problems.
We need to make sure to have configuration with fs.default.name space set such as
configuration.set("fs.default.name","hdfs://ourHDFSNameNode:50000");
Below I've put a piece of sample code:
Configuration configuration = new Configuration();
configuration.set("fs.default.name","hdfs://ourHDFSNameNode:50000");
FileSystem fs = pt.getFileSystem(configuration);
BufferedReader br = new BufferedReader(new InputStreamReader(fs.open(pt)));
String line = null;
line = br.readLine
while (line != null) {
try {
line = br.readLine
System.out.println(line);
}
}
The answer above is pointing to the right direction. Allow me to add the following:
Namenode does NOT directly read or write data.
Client (your Java program using Direct access to HDFS) interacts with Namenode to update HDFS namespace and retrieve block locations for reading/writing.
Client interacts directly with Datanode to read/write data.
You were able to list directory contents because hostname:9000was accessible to your client code. You were doing the number 2 above.
To be able to read and write, your client code needs access to the Datanode (number 3). The default port for Datanode DFS data transfer is 50010. Something was blocking your client communication to hostname:50010. Possibly a firewall or SSH tunneling configuration problem.
I was using Hadoop 2.7.2, so maybe you have a different port number setting.
Related
I am trying to run a JMX file from non GUI mode on linux server. Using SFTP protocol trying to upload file to object store. Below is the script and configuration used to run the script.
Test.jmx
import com.jcraft.jsch.*;
import java.io.*;
def jsch = new JSch()
def session = jsch.getSession("user1", "xyz", 4000)
session.setConfig("StrictHostKeyChecking", "no")
session.setPassword("password")
def sftpSession = session.connect()
def channel = session.openChannel("sftp")
channel.connect();
def channelSftp = (ChannelSftp)channel;
log.info("SFTP Connection with host is acquired" + channelSftp)
channelSftp.cd("/0002/test/upload/r9a1");
def f1 = new File("/home/dc-user/Files/test_exact5mb.txt");
channelSftp.put(new java.io.FileInputStream(f1), f1.getName()+ Math.random());
session.disconnect()
Configuration used is
No of Threads - 150
Ramp up period - 1
Loop Count -1
After running the script, out of 150 files sometimes 126 or 129 or 141 files gets uploaded but not 150.
Below is the error i see on terminal
The JVM should have exited but did not.
The following non-daemon threads are still running (DestroyJavaVM is OK):
Thread[Connect thread 10.157.147.242 session,5,main], stackTrace:java.net.SocketInputStream#socketRead0
java.net.SocketInputStream#socketRead at line:116
java.net.SocketInputStream#read at line:171
java.net.SocketInputStream#read at line:141
com.jcraft.jsch.IO#getByte at line:82
com.jcraft.jsch.Session#read at line:908
com.jcraft.jsch.Session#run at line:1378
java.lang.Thread#run at line:74
Please let me know if i am missing any configuration
I fail to see how this code can work as your i is not defined anywhere so it will not even compile
It seems that some thread(s) cannot be finished because they are still reading something, I believe you have some form of SLAs or NFRs which define the maximum acceptable time for the file upload operation and your test seems to be exceeding this time.
So before calling def sftpSession = session.connect() I would recommend placing the Session.setTimeout() function call like:
session.setTimeout(5000)
this way if the operation is not finished in 5 seconds your sampler will fail
If you're not too comfortable with Groovy scripting using SSH SFTP sampler might be much easier, check out Load Testing FTP and SFTP Servers Using JMeter article for more details.
I'm using apache commons-net FTPClient (ver 3.3) and below error was produced on one of my clients machine (I've tried reproducing the error on my dev machine without luck using testing folder with the same requests on the same server with the same login)
I have a process that check's the remote FTP server for new requests in form of an XML-files. After listing all of those files i proceed in loop to check if they're in XML format. If the file is in this format I do first change their name by changing format from *.xml to *.xmlProcessing, retrive them to a input stream, parse them to my object and create a request in my queue and finally change the name and move them to subfolder working as an archive.
After downloading random amount of files I get stuck while calling retrieveFileStream on the next file, without a timeout or IO Exception.
I've managed to get logs from FTP server and it just says it can't open a data connection
05.06.2019 12:45:45 - > RNFR /folder/file.xml
05.06.2019 12:45:45 - > 350 File exists, ready for destination name.
05.06.2019 12:45:45 - > RNTO /folder/file.xmlProcessing
05.06.2019 12:45:45 - > 250 file renamed successfully
05.06.2019 12:45:45 - > PORT *ports*
05.06.2019 12:45:45 - > 200 Port command successful
05.06.2019 12:45:45 - > RETR /folder/file.xmlProcessing
05.06.2019 12:45:45 - > 150 Opening data channel for file download from server of "/folder/file.xmlProcessing"
05.06.2019 12:45:45 - > 425 Can't open data connection for transfer of "/folder/file.xmlProcessing"
I've already tried diffrent FTP modes. Active local, remote, passive etc. (currently stuck with passive local mode).
I've tried the data timeout but it looks like while i finally got stuck on one of the files the method took more than 1 minute on that file despite me setting the timeout on 30s.
ftp = new FTPClient();
ftp.addProtocolCommandListener(new PrintCommandListener(new PrintWriter(System.out)));
ftp.connect(server, port);
int reply = ftp.getReplyCode();
if (!FTPReply.isPositiveCompletion(reply)) {
ftp.disconnect();
throw new IOException("Exception in connecting to FTP Server");
}
ftp.login(user, password);
ftp.enterLocalPassiveMode();
ftp.setKeepAlive(true);
ftp.setDataTimeout(30000);
Collection<String> listOfFiles = listFiles(FOLDER_PATH);
for(String filePath : listOfFiles){
if (filePath != null && filePath.endsWith(".xml")) {
ftp.rnfr(folder + filePath);
ftp.rnto(folder + filePath + "Processing");
InputStream fileInputStream = ftp.retrieveFileStream(folder + filePath + "Processing");
ftp.completePendingCommand();
//Parsing file to an instance of my object and creating request
ftp.rnfr(folder + filePath + "Processing");
ftp.rnto(archiveFolder + filePath);
if(fileInputStream != null){
fileInputStream.close();
}
}
}
Is there's a bigger likehood that this is fault of the FTP Server, Firewall issues or something else ?
I've runed the same code from my dev machine and it processed all files from test folder (there were around 400 of them) i don't know if im being unlucky for error not occuring on my local dev machine or is it actually something wrong with communication of my contractor with the remote server ?
Everytime you do a file transfer or a directory listing with FTP, the server (or client if using an active mode) assigns a random port number out of a configured range to that transfer. The port number is not released immediately, when the transfer completes. There's some cooldown interval. If you do too many file transfers in a short time interval, it can happen that the server runs out of the available ports – Because all ports end up in the cooldown state.
If you can, check the server configuration and configure a larger range of ports.
Or as a workaround, you can try to slow down the transfer rate.
For some background, see:
How many data channel ports do I need for an FTP?
Why does FTP passive mode require a port range as opposed to only one port?
Though this is just a guess, you should check the server's log, as it can show more details.
Another possibility is, that there's simply a limited number of transfers the server allows for a specific user or source address in some time interval.
i need to send some messages from my java web application to some servers using Diameter protocol, in particular CCR-CCA scenario. I had a look at jdiameter opensource project, but my usecase does not require such complexity, since that i just need to send a single request and log the response (actually i don't even need the CER-CEA part).
So i thought i could just have used Seagull running under my webapp. I downloaded Seagull (for Windows), and what i'm trying to do is basically to run the .bat file coming from Seagull for the diameter environment from my java environment.
That's what i've done till now..
1) A simple test to invoke the client.. Here wrapper simply sets working dir and starts the process
public static void main(String[] args) {
List<String> cmd=new ArrayList<>();
cmd.add("cmd.exe");
cmd.add("/c");
cmd.add("my_start_client.bat");
JavaProcessBuilderWrapper wrapper = new JavaProcessBuilderWrapper();
Process p = wrapper.createProcess(RedirectErrorsTo.STDERR,
new HashMap<>(), new File("my_working_dir"), cmd);
BufferedReader reader = new BufferedReader(new InputStreamReader(p.getInputStream()));
StringBuilder output = new StringBuilder();
String line;
try {
while ((line = reader.readLine()) != null) {
output.append(line);
}
System.out.println(line);
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
2) I modified the client's and server's .bat files coming from Seagull to use CCR-CCA protocol.
Running Java main with this configuration caused a
Fatal: Keyboard saved configuration failure error
on my logs.
3) So, as mentioned here i further modified my client's .bat file to run in background mode, adding -bg at the end. Now my client's bat look like this
#ECHO OFF
rem
"Diameter Start Script Sample"
"Local env"
SET RUN_DIR=C:\Program Files\Seagull
set PATH=%PATH%;%RUN_DIR%
set LD_LIBRARY_PATH=%RUN_DIR%
set RUN_DIR=%RUN_DIR%\diameter-env\run
cd %RUN_DIR%
cls
mode 81,25
echo "Seagull Diameter Client Sample Start"
seagull -conf ..\config\conf.client.xml -dico ..\config\base_ro_3gpp.xml -scen ..\scenario\ccr-cca.ro.client.xml -log ..\logs\ccr-cca.client.log -llevel ETM -bg
pause
Since i was facing some troubles, to keep things simple, i just tried to make it work at least via cmd (not using my java method), but i think background mode is messing around, because now when i start my server and then my client in bg mode, sometimes i get a
Fatal: Forking error
but the most of the times, the client send a single message and then on my console i see that my software is causing connection abort (error code -1), and from the log i see that the channel just get closed, and my client does not even receive an answer. (NB for now i left the configuration files untouched)
Has any of you faced this behaviour? Is something else closing the connection (firewall perhaps)? Do i have to provide other configurations to make this work?
Once i can get this working, can i use my java web app (with a method similar to the one i already mentioned) to make diameter calls?
Thanks in advance, any help is really welcomed.
The code below works to two other sites I've tried, but will not work with my domain hosted by 1and1. The return code is always 500 - Permanent Negative Completion reply.
I know I'm connecting because FTPReply.isPositiveCompletion(reply) returns true. I tried it with both reading file off phone storage and sending it from a byte array populated early in the code. This is the byte array. Both return 500. Both work on the other sites.
If I don't use enterLocalPassiveMode() the code stops executing on the storeFile call. No exception, no socket time-out. It just ends there and the async task will not call again in that session. The code does not do that on the other sites.
I've tried both ASCII and BINARY file types. Both return 500. 1and1 site says to use my domain and port 21. I can connect with CoreFTP and read and write using both of the accounts I've set up.
I also tired ftp4j and had the same response with all scenarios so went back to Apache because the code was already written with robust error trapping.
I've tried both mydomain.com and ftp.mydomian.com. 500 on both. I also tried the dot quad I can see in the CoreFTP window, but i get "cannot resolve host name" with the Apache Java code. Maybe not a static IP?
This is what CoreFTP does. It connects on port 21 and then goes in to passive mode and an ASCII data connection.
It's a long shot, but has anyone else ever FTPed to their 1and1 domain using Java in Android Studio?
Greg
Resolving mydomain.com...
Connect socket #5684 to xx.xx.xx.xxx, port 21...
220 Microsoft FTP Service
USER ftp79815757-0
331 Password required for ftp79815757-0.
PASS **********
230 User logged in.
SYST
215 Windows_NT
Keep alive off...
PWD
257 "/ftp79815757-0" is current directory.
PASV
227 Entering Passive Mode (xx,xxx,xx,xxx,xxx,xxx).
LIST
Connect socket #5700 to xx.xx.xx.xx, port 62894...
150 Opening ASCII mode data connection.
226 Transfer complete.
Transferred 51 bytes in 0.094 seconds
FTPClient mFtpClient = new FTPClient();
String ip = "my domain dot com";
String userName = "ftp79815757-0";
String pass = "password";
mFtpClient.connect(InetAddress.getByName(ip));
mFtpClient.login(userName, pass);
int reply = mFtpClient.getReplyCode();
if (FTPReply.isPositiveCompletion(reply)) {
mFtpClient.setFileType(FTP.BINARY_FILE_TYPE);
//one thread said this would do the trick
mFtpClient.enterLocalPassiveMode();
mFtpClient.enterRemotePassiveMode();
InputStream stream = new ByteArrayInputStream(imageData);
//I have two accounts. One points to images_in
/*if (!mFtpClient.changeWorkingDirectory("images_in")) {
Log.e("ChangeDir", String.valueOf(mFtpClient.getReplyCode()));
}*/
if (!mFtpClient.storeFile("remoteName.jpg", stream)) {
Log.e("FTPUpload", String.valueOf(mFtpClient.getReplyCode()));
}
stream.close();
mFtpClient.disconnect();
}
Finally got it. The main problem was that I was using an old version of the Apache library. The Jar I was using was commons-net-1.4.jar. Someone in another thread pointed me to commons-net-3.3.jar.
I commented out both mFtpClient.enterLocalPassiveMode() and mFtpClient.enterRemotePassiveMode(), and with some trail and error it worked with FTP.BINARY_FILE_TYPE and not ASCII_FILE_TYPE. ASCII got the file there, but it was garbage.
I need to copy some files from hdfs:///user/hdfs/path1 to hdfs:///user/hdfs/path2. I wrote a java code to do the job:-
ugi = UserGroupInformation.createRemoteUser("hdfs", AuthMethod.SIMPLE);
System.out.println(ugi.getUserName());
conf = new org.apache.hadoop.conf.Configuration();
// TODO: Change IP
conf.set("fs.defaultFS", URL);
conf.set("hadoop.job.ugi", user);
conf.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
conf.set("fs.file.impl", org.apache.hadoop.fs.LocalFileSystem.class.getName());
// paths = new ArrayList<>();
fs = FileSystem.get(conf);
I am getting all paths for the wild card as
fs.globStatus(new Path(regPath));
and copy as
FileUtil.copy(fs, p, fs, new Path(to + "/" + p.getName()), false, true, conf);
However copying is failing with following message whereas globstatus execute successfully
WARN BlockReaderFactory:682 - I/O error constructing remote block reader.
org.apache.hadoop.net.ConnectTimeoutException: 60000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/10.110.80.177:50010]
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:532)
at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:3044)
at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:744)
at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:659)
at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:327)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:574)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:797)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:844)
at java.io.DataInputStream.read(DataInputStream.java:100)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:78)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:52)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:112)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:366)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338)
Note that I am running code remotely over Internet using port forwarding. I.e.
192.168.1.10[JAVA API] ---> 154.23.56.116:8082[Name Node Public I/P]======10.1.3.4:8082[Name Node private IP]
I guess following is the reason:-
Query is made to namenode for globStatus which is successfully executed by the name node
Copying command is passed to namenode which will return 10.110.80.177:50010 for other data nodes address on other machines, and then Java IP will try to pass the copy commands to these data nodes, since they are not exported to outside world I got this error!
Am I right in this deduction? How to solve the issue? Do I need to create a java server at namenode which will collect copy command and copy the files locally in the cluster.