We Keep Coding

FTP using Apache commons not working - Error: 550 Dataset not found (FTPid that i am using is getting appended to filename)

I am trying to ftp the file from Linux VM to an AS400 server. I was able to login to server in passive mode but when trying to use the STOR command to upload the file getting below error:
STOR XX.YY600040.XXXZZZXXX
**550 Dataset not found, DSN=FTPID.XX.YY600040.XXXZZZXXX**
Not sure why the ftpid that i am using is getting prefixed to the filename. Is there any way to avoid it?
Below is the sample code that i am using:
private static String sendFTPFile(String fileName) throws Exception {
StringBuffer ftpMessage = new StringBuffer();
if (SHOW_DEBUG) ftpMessage.append("<ul>");
FTPClient ftp = null;
try {
String server = "****";
String username = "****";
String password = "XXXXX";
String hostDir = "";
String localFileName = fileName;
String localFilePath = "***/**/*";
boolean binaryTransfer = false, error = false;
FTPClient ftp = new FTPClient();
ftp.addProtocolCommandListener(new PrintCommandListener(
new PrintWriter(System.out)));
int reply;
ftp.connect(server)
// After connection attempt, you should check the reply code to verify
// success.
reply = ftp.getReplyCode();
if (!FTPReply.isPositiveCompletion(reply))
{
ftp.disconnect();
error = true;
}
if(!error) {
if (!ftp.login(username, password))
{
ftp.logout();
error = true;
}
if(!error) {
if (binaryTransfer)
ftp.setFileType(FTP.BINARY_FILE_TYPE);
// Use passive mode as default
ftp.enterLocalPassiveMode();
InputStream input;
input = new FileInputStream(localFilePath+localFileName);
boolean ftpSuccess = ftp.storeFile(hostDir+localFileName, input);
input.close();
if (!ftpSuccess) {
throw new Exception("File ftp error");
}else {
if (SHOW_DEBUG) ftpMessage.append("<li>isFtpSuccess()...success").append("</li>");
}
ftp.logout();
if (SHOW_DEBUG) ftpMessage.append("<li>ftp.logout()...success").append("</li>");
}
}
}
catch (Exception ex) {
ex.printStackTrace();
System.out.println("Exception occur while transfering file using ftp"+ ex.toString());
throw new Exception(ftpMessage.toString(), ex);
}
finally {
if (ftp!=null && ftp.isConnected())
{
try
{
ftp.disconnect();
}
catch (Exception e)
{
e.printStackTrace();
System.out.println("Exception occur while transfering file using ftp"+ e.toString());
throw new Exception(ftpMessage.toString(), e);
}
}
if (SHOW_DEBUG) ftpMessage.append("</ul>");
}
return ftpMessage.toString();
}

I did some more debugging on doing ftp from linux to as400. And when doing pwd after logging to ftp server its giving message as:
ftp> pwd
257 "'FTPID.'" is current prefix
And thinking how to remove that prefix so i ran the below command got output as no prefix defined:
ftp> cd ..
200 "" no prefix defined
And after that when i uploaded the file using put command it was uploaded successfully. So i did some research on how to go back one directory using Apache commons.net api's that i am using and found the CDUP method.
When i ran the FTPClient.cdup() method before uploading the file. I was able to successfully FTP the file from java code as well.
ftp.cdup();
input = new FileInputStream(localFilePath+localFileName);
boolean ftpSuccess = ftp.storeFile(hostDir+localFileName, input);

Java HDF fsDataOutputStream write fails empty file creation

I am having a strange issue with write small files on hadoop. below is sample program
public void writeFile(Configuration conf, String message, String filename) throws Exception {
FSDataOutputStream fsDataOutputStream = null;
DistributedFileSystem fs = null;
try {
fs = (DistributedFileSystem) FileSystem.get(URI.create(properties.getHadoop().getRawLocation()), conf);
Path hdfswritepath = new Path(properties.getHadoop().getRawLocation() + "/" + filename + ".json");
fsDataOutputStream = fs.create(hdfswritepath);
fsDataOutputStream.write(message.getBytes());
fsDataOutputStream.close();
fsDataOutputStream.hsync();
} catch (IllegalArgumentException | IOException e) {
System.out.println("Got Exception");
e.printStackTrace();
throw e;
} finally {
fs.close();
System.out.println("clean up done");
}
}
Above code is creating empty file at hadoop location. here are some on item I tried
Firewall is not there between client and hadoop server
Copy from local to hadoop is working.
Issue is Only 0 byte file getting created.
I am getting below exception for this.
09:12:02,129 INFO [org.apache.hadoop.hdfs.DFSClient] (Thread-118) Exception in createBlockOutputStream: java.net.ConnectException: Connection timed out: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1533)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1309)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1262)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:448)

I was able to fix this by
conf.set("dfs.client.use.datanode.hostname", "true");

Uploading a file in a specific path of an FTP server [duplicate]

This question already has answers here:
FtpClient storeFile always return False
(5 answers)
Closed 8 years ago.
I want to upload a file in a specific path in an ftp server the code is quite simple:
public static void main(String[] args) {
String server = "xx.xx.xx.xx";
String user = "xxx";
String pass = "xxx";
FTPClient ftpClient = new FTPClient();
try {
ftpClient.connect(server);
System.out.println("Connected to " + server + ".");
System.out.print(ftpClient.getReplyString());
ftpClient.login(user, pass);
ftpClient.enterLocalPassiveMode();
ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
// uploads first file using an InputStream
File firstLocalFile = new File("/tmp/PAR.TXT");
String firstRemoteFile = "/DATA/OUTFILES/PAR.TXT";
InputStream inputStream = new FileInputStream(firstLocalFile);
System.out.println("Start uploading first file");
boolean done = ftpClient.storeFile(firstRemoteFile, inputStream);
System.out.println("done:"+done);
inputStream.close();
if (done) {
System.out.println("The file is uploaded successfully.");
}
} catch (IOException ex) {
System.out.println("Error: " + ex.getMessage());
ex.printStackTrace();
} finally {
try {
if (ftpClient.isConnected()) {
ftpClient.logout();
ftpClient.disconnect();
}
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
I always get done = false.
Here's the result:
Connected to xx.xx.xx.xx.
220 "Welcome (logging activated)"
Start uploading file
done:false
I printed the FtpClient#getReplyCode(). and i get this:
500 Illegal PORT command.

You can only access files relative to the root folder of the ftp server. You need to configure your ftp server to add a virtual folder pointing to the path you want.

I passed to PassiveMode and it works now

Unable to download files from FTP server

I've created Java function that downloads files from FTP server. It works fine from my local machine. But I need to run it under linux server (means another host and port). And the function gives an error
The collection, array, map, iterator, or enumeration portion of a for statement cannot be null
Caused in a line with the code:
for(String f : ftpNames) {
ftpclient.retrieveFile(f, os); // os is OutputStream
}
So it doesn't see the files...
I added
ftpclient.enterRemotePassiveMode();
And ftpclient.getPassiveHost() returns 227 Entering Passive Mode (x,x,x,x,204,15)
Tried to list and download them via shell - it works.
How should I modify my code to solve the problem? Thanks.
UPD. I got log from FTP server I'm trying to get files from, and there is such string:
425 Cannot open data connection
Full code:
static boolean ftpFilesDownload(String ip, int port, String login, String passwd, String ftpdir, String localdir) throws IOException {
Boolean result = false;
FTPClient client = new FTPClient();
String separator = File.separator;
try {
client.connect(ip, port);
System.out.println(client.getReplyString());
client.login(login, passwd);
System.out.println(client.getReplyString());
client.setControlKeepAliveTimeout(1000*60*5);
client.setControlKeepAliveReplyTimeout(1000*60*5);
client.setFileType(FTP.BINARY_FILE_TYPE);
System.out.println("client setFileType success");
client.changeWorkingDirectory(ftpdir);
System.out.println(client.getReplyString());
client.printWorkingDirectory();
System.out.println("directory changed");
FTPFile[] ftpFiles = client.listFiles();
System.out.println(ftpFiles);
String[] ftpNames = client.listNames();
System.out.println("the files are " + Arrays.toString(ftpNames)); // so null here...
for(String f : ftpNames) {
String localfile = localdir + f;
OutputStream os = new FileOutputStream(localfile);
try {
result = client.retrieveFile(f, os);
System.out.println("DOWNLOADING STARTED);
System.out.println(client.getReplyString());
client.noop();
}
catch(Exception e) {
System.out.println(e);
result = false;
}
finally {
if(os != null)
os.close();
}
}
client.logout();
System.out.println(client.getReplyString());
}
catch(Exception e)
{
System.out.println(e);
result = false;
}
finally
{
try
{
client.disconnect();
}
catch(Exception e)
{
System.out.println(e);
result = false;
}
}
return result;
}

As the error message explains, you're trying to iterate over a null object. You should check for this (or make sure an empty Iterable is used perhaps)
If this is an execptional (error) state, I'd check for this explicitly and throw some kind of runtime exception, e.g.:
if (ftpNames == null) {
throw new IllegalArgumentException("Cannot use a null set of FTP servers");
}
for (String f : ftpNames) {
ftpclient.retrieveFile(f, os); // os is OutputStream
}
Alternatively you could try to continue with no FTP servers, but seems a bit pointless.

Try to use ftpclient.enterLocalActiveMode();

Store ObjectOutputStream in FTP causes java.net.SocketException: Software caused connection abort: socket write error

I am trying to store an custom defined (ProjectData) Java object in a file in FTP . ProjectData class implements Serializable interface.I am using apache Commons Net FTP client.I have a custom class FtpClient , which has instance of Commons net FTP Client. I connect , upload file, download file, storeFileStream using this class.When I try to call
saveProjectData function, it throws exception while executing save.writeObject(data).
java.net.SocketException: Software caused connection abort: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method) [:1.6.0_23]
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) [:1.6.0_23]
at java.net.SocketOutputStream.write(SocketOutputStream.java:136) [:1.6.0_23]
at org.apache.commons.net.io.SocketOutputStream.write(SocketOutputStream.java:72) [:2.2]
at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1847) [:1.6.0_23]
at java.io.ObjectOutputStream$BlockDataOutputStream.flush(ObjectOutputStream.java:1792) [:1.6.0_23]
at java.io.ObjectOutputStream.flush(ObjectOutputStream.java:699) [:1.6.0_23]
So what is the problem in the code, I ready that connection is already closed, in the meantime of saving object.But I could not find where the bug.
public void saveProjectData(ProjectData data, String filePath) {
ftpClient = new FtpClient(ftpConfig, logger);
ObjectOutputStream save = null;
try {
ftpClient.connect();
save = new ObjectOutputStream(ftpClient.storeFileStream(filePath));
} catch (Exception e) {
// ignore
}
try {
if (save == null) {
ftpClient.reconnect();
save = new ObjectOutputStream(ftpClient.storeFileStream(filePath));
}
save.writeObject(data);
save.flush();
} catch (IOException e) {
logger.error("Error creating file:" + filePath);
throw new FileTransferException("Could not create file:" + filePath, e);
} finally {
if (save != null) {
try {
save.close();
} catch (Exception e) {
logger.warn("Error closing writer of file:" + filePath, e);
}
save = null;
}
if (ftpClient != null) {
try {
ftpClient.disconnect();
} catch (Exception e) {
// ignore
}
ftpClient = null;
}
}
}
/**
* Stores the file in FTP
* #param filePath
* #return
*/
public OutputStream storeFileStream(String filePath) {
try {
client.setFileType(FTP.BINARY_FILE_TYPE);
return client.storeFileStream(filePath);
} catch (Exception ioe) {
log.warn("Could not store file:" + filePath, ioe);
return null;
}
}

I read that connection is already closed
No, you read (or you should have read) that the connection is already closed by the peer.
Possibly you have exceeded an upload limit.