Java HDF fsDataOutputStream write fails empty file creation - java

I am having a strange issue with write small files on hadoop. below is sample program
public void writeFile(Configuration conf, String message, String filename) throws Exception {
FSDataOutputStream fsDataOutputStream = null;
DistributedFileSystem fs = null;
try {
fs = (DistributedFileSystem) FileSystem.get(URI.create(properties.getHadoop().getRawLocation()), conf);
Path hdfswritepath = new Path(properties.getHadoop().getRawLocation() + "/" + filename + ".json");
fsDataOutputStream = fs.create(hdfswritepath);
fsDataOutputStream.write(message.getBytes());
fsDataOutputStream.close();
fsDataOutputStream.hsync();
} catch (IllegalArgumentException | IOException e) {
System.out.println("Got Exception");
e.printStackTrace();
throw e;
} finally {
fs.close();
System.out.println("clean up done");
}
}
Above code is creating empty file at hadoop location. here are some on item I tried
Firewall is not there between client and hadoop server
Copy from local to hadoop is working.
Issue is Only 0 byte file getting created.
I am getting below exception for this.
09:12:02,129 INFO [org.apache.hadoop.hdfs.DFSClient] (Thread-118) Exception in createBlockOutputStream: java.net.ConnectException: Connection timed out: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1533)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1309)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1262)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:448)

I was able to fix this by
conf.set("dfs.client.use.datanode.hostname", "true");

Related

Unable to connect to FTP using FTP4J

I'm writing a program that has to connect to an FTP server in order to download certain files. In order to do this I'm using the FTP4J library, However I'm running into some trouble.
So far I have:
if ("Dataset FTP location".equals(link.text())) {
String FTPURL = link.attr("href");
FTPClient client = new FTPClient();
try {
client.connect(FTPURL);
} catch (FTPIllegalReplyException e) {
e.printStackTrace();
} catch (FTPException e) {
e.printStackTrace();
}
Where the URL of the FTP is ftp://ftp.pride.ebi.ac.uk/pride/data/archive/2015/10/PXD002829
However If I run the program I get:
Exception in thread "main" java.net.UnknownHostException: ftp://ftp.pride.ebi.ac.uk/pride/data/archive/2015/10/PXD002829
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at it.sauronsoftware.ftp4j.FTPConnector.tcpConnectForCommunicationChannel(FTPConnector.java:208)
at it.sauronsoftware.ftp4j.connectors.DirectConnector.connectForCommunicationChannel(DirectConnector.java:39)
at it.sauronsoftware.ftp4j.FTPClient.connect(FTPClient.java:1036)
at it.sauronsoftware.ftp4j.FTPClient.connect(FTPClient.java:1003)
at Main.main(Main.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Any help would be appreciated.
Also I don't have a log in for the server, it's just a public repository of files. Will this effect how I go about doing things?
You need to split off the path and create a url that looks like:
ftp.pride.ebi.ac.uk
In answer to your comment you need to do something like this:
String ftpPath = "ftp://ftp.pride.ebi.ac.uk/pride/data/archive/2015/10/PXD002829";
URL url = new URL(ftpPath);
String host = url.getHost();
FTPClient client = new FTPClient();
try {
client.connect(host);
client.login("anonymous", "anonymous");
FTPFile[] list = client.list(url.getPath());
for (FTPFile f : list) {
// Instead of printing out the file download it. See
// http://www.sauronsoftware.it/projects/ftp4j/manual.php#14
System.out.println(f);
}
} catch (FTPIllegalReplyException e) {
e.printStackTrace();
} catch (FTPException e) {
e.printStackTrace();
}

Uploading a file in a specific path of an FTP server [duplicate]

This question already has answers here:
FtpClient storeFile always return False
(5 answers)
Closed 8 years ago.
I want to upload a file in a specific path in an ftp server the code is quite simple:
public static void main(String[] args) {
String server = "xx.xx.xx.xx";
String user = "xxx";
String pass = "xxx";
FTPClient ftpClient = new FTPClient();
try {
ftpClient.connect(server);
System.out.println("Connected to " + server + ".");
System.out.print(ftpClient.getReplyString());
ftpClient.login(user, pass);
ftpClient.enterLocalPassiveMode();
ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
// uploads first file using an InputStream
File firstLocalFile = new File("/tmp/PAR.TXT");
String firstRemoteFile = "/DATA/OUTFILES/PAR.TXT";
InputStream inputStream = new FileInputStream(firstLocalFile);
System.out.println("Start uploading first file");
boolean done = ftpClient.storeFile(firstRemoteFile, inputStream);
System.out.println("done:"+done);
inputStream.close();
if (done) {
System.out.println("The file is uploaded successfully.");
}
} catch (IOException ex) {
System.out.println("Error: " + ex.getMessage());
ex.printStackTrace();
} finally {
try {
if (ftpClient.isConnected()) {
ftpClient.logout();
ftpClient.disconnect();
}
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
I always get done = false.
Here's the result:
Connected to xx.xx.xx.xx.
220 "Welcome (logging activated)"
Start uploading file
done:false
I printed the FtpClient#getReplyCode(). and i get this:
500 Illegal PORT command.
You can only access files relative to the root folder of the ftp server. You need to configure your ftp server to add a virtual folder pointing to the path you want.
I passed to PassiveMode and it works now

Store ObjectOutputStream in FTP causes java.net.SocketException: Software caused connection abort: socket write error

I am trying to store an custom defined (ProjectData) Java object in a file in FTP . ProjectData class implements Serializable interface.I am using apache Commons Net FTP client.I have a custom class FtpClient , which has instance of Commons net FTP Client. I connect , upload file, download file, storeFileStream using this class.When I try to call
saveProjectData function, it throws exception while executing save.writeObject(data).
java.net.SocketException: Software caused connection abort: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method) [:1.6.0_23]
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) [:1.6.0_23]
at java.net.SocketOutputStream.write(SocketOutputStream.java:136) [:1.6.0_23]
at org.apache.commons.net.io.SocketOutputStream.write(SocketOutputStream.java:72) [:2.2]
at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1847) [:1.6.0_23]
at java.io.ObjectOutputStream$BlockDataOutputStream.flush(ObjectOutputStream.java:1792) [:1.6.0_23]
at java.io.ObjectOutputStream.flush(ObjectOutputStream.java:699) [:1.6.0_23]
So what is the problem in the code, I ready that connection is already closed, in the meantime of saving object.But I could not find where the bug.
public void saveProjectData(ProjectData data, String filePath) {
ftpClient = new FtpClient(ftpConfig, logger);
ObjectOutputStream save = null;
try {
ftpClient.connect();
save = new ObjectOutputStream(ftpClient.storeFileStream(filePath));
} catch (Exception e) {
// ignore
}
try {
if (save == null) {
ftpClient.reconnect();
save = new ObjectOutputStream(ftpClient.storeFileStream(filePath));
}
save.writeObject(data);
save.flush();
} catch (IOException e) {
logger.error("Error creating file:" + filePath);
throw new FileTransferException("Could not create file:" + filePath, e);
} finally {
if (save != null) {
try {
save.close();
} catch (Exception e) {
logger.warn("Error closing writer of file:" + filePath, e);
}
save = null;
}
if (ftpClient != null) {
try {
ftpClient.disconnect();
} catch (Exception e) {
// ignore
}
ftpClient = null;
}
}
}
/**
* Stores the file in FTP
* #param filePath
* #return
*/
public OutputStream storeFileStream(String filePath) {
try {
client.setFileType(FTP.BINARY_FILE_TYPE);
return client.storeFileStream(filePath);
} catch (Exception ioe) {
log.warn("Could not store file:" + filePath, ioe);
return null;
}
}
I read that connection is already closed
No, you read (or you should have read) that the connection is already closed by the peer.
Possibly you have exceeded an upload limit.

File Retrieval from java code

I am trying to retrieve files from ftp.I am getting following exception.
Sometimes concurrent access to same file can be case.My firewall is turn off.
My question is
how can get ride of this exception, because it can not download necassary files, my program does not make its work
My download code:
public boolean downloadFile(String sourceFilePath, String destinationDirPath, String newFileName)
throws IllegalStateException {
// final FTPClient client = this.getClient();
FileOutputStream output = null;
try {
client.setFileType(FTP.BINARY_FILE_TYPE);
// The remote filename to be downloaded.
String filename = new File(sourceFilePath).getName();
// Downloads the selected file to the C drive
output = new FileOutputStream(destinationDirPath + File.separator + newFileName);
this.cd(new File(sourceFilePath).getParent());
// Download file from FTP server
boolean result = client.retrieveFile(filename, output);
log.trace("File is downloaded from server:" + filename + ", to destination:"+newFileName);
return result;
}
// Indicate that an exception is occurred while downloading file
catch (Exception ioe) {
throw new FileTransferException("Could not download file", ioe);
} finally {
if (output != null) {
try {
// trying to close FileOutputStream
output.close();
} catch (IOException e) {
log.warn("Error closing writer to file:"+newFileName, e);
}
output = null;
}
}
}
And here is the stack trace:
Exception during data transfer, closing data connection socket
java.net.SocketException: Software caused connection abort: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
at org.apache.ftpserver.impl.IODataConnection.transfer(IODataConnection.java:289)
at org.apache.ftpserver.impl.IODataConnection.transferToClient(IODataConnection.java:161)
at org.apache.ftpserver.command.impl.RETR.execute(RETR.java:166)
at org.apache.ftpserver.impl.DefaultFtpHandler.messageReceived(DefaultFtpHandler.java:210)
at org.apache.ftpserver.listener.nio.FtpHandlerAdapter.messageReceived(FtpHandlerAdapter.java:61)
at

Download file through FTP by Quartz scheduler

I tried working Ftp download stand alone application and it works fine. But when I included that into Quartz scheduler in web application, it stucks.
Here is what I did.
public class FtpTransfer implements StatefulJob {
public void execute(JobExecutionContext arg0) throws JobExecutionException {
FTPClient ftp = new FTPClient();
FileOutputStream br = null;
try
{
ftp.connect("localhost");
ftp.login("admin", "admin");
String path = "alfresco/MYPUB/Admin/TMM/Pickup";
ftp.setFileType(FTPClient.BINARY_FILE_TYPE);
ftp.changeWorkingDirectory(path);
System.out.println("After Changing Directory path");
FTPFile[] ftpFile = ftp.listFiles(path);
System.out.println("After getting list of files");
System.out.println("Length : "+ftpFile.length);
System.out.println("----------------- Downloaded -------------");
for(FTPFile tempFtpFiles : ftpFile) {
br = new FileOutputStream("e:\\Downloaded\\"+tempFtpFiles.getName());
ftp.retrieveFile(tempFtpFiles.getName(), br);
System.out.println(tempFtpFiles.getName());
}
System.out.println("------------------------------------------");
}
catch(Exception exception) {
System.out.println("Error : "+exception);
} finally {
try {
if(br!=null){
br.close();
}
ftp.disconnect();
} catch(IOException e) {
e.printStackTrace();
System.out.println("Error : "+e);
}
}
}
}
When I start the server, It prints
After Changing Directory path
After Changing Directory path
After Changing Directory path
Every 10 secs. But It is not downloading the files from the path given. Mailnly the program didn't crossed the line FTPFile[] ftpFile = ftp.listFiles(path). What did I do wrong?
Thanks for your comments. I have found the problem. After included jakarta-oro.jar in lib, its working fine.

Categories