I'm writing a program that has to connect to an FTP server in order to download certain files. In order to do this I'm using the FTP4J library, However I'm running into some trouble.
So far I have:
if ("Dataset FTP location".equals(link.text())) {
String FTPURL = link.attr("href");
FTPClient client = new FTPClient();
try {
client.connect(FTPURL);
} catch (FTPIllegalReplyException e) {
e.printStackTrace();
} catch (FTPException e) {
e.printStackTrace();
}
Where the URL of the FTP is ftp://ftp.pride.ebi.ac.uk/pride/data/archive/2015/10/PXD002829
However If I run the program I get:
Exception in thread "main" java.net.UnknownHostException: ftp://ftp.pride.ebi.ac.uk/pride/data/archive/2015/10/PXD002829
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:178)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at it.sauronsoftware.ftp4j.FTPConnector.tcpConnectForCommunicationChannel(FTPConnector.java:208)
at it.sauronsoftware.ftp4j.connectors.DirectConnector.connectForCommunicationChannel(DirectConnector.java:39)
at it.sauronsoftware.ftp4j.FTPClient.connect(FTPClient.java:1036)
at it.sauronsoftware.ftp4j.FTPClient.connect(FTPClient.java:1003)
at Main.main(Main.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Any help would be appreciated.
Also I don't have a log in for the server, it's just a public repository of files. Will this effect how I go about doing things?
You need to split off the path and create a url that looks like:
ftp.pride.ebi.ac.uk
In answer to your comment you need to do something like this:
String ftpPath = "ftp://ftp.pride.ebi.ac.uk/pride/data/archive/2015/10/PXD002829";
URL url = new URL(ftpPath);
String host = url.getHost();
FTPClient client = new FTPClient();
try {
client.connect(host);
client.login("anonymous", "anonymous");
FTPFile[] list = client.list(url.getPath());
for (FTPFile f : list) {
// Instead of printing out the file download it. See
// http://www.sauronsoftware.it/projects/ftp4j/manual.php#14
System.out.println(f);
}
} catch (FTPIllegalReplyException e) {
e.printStackTrace();
} catch (FTPException e) {
e.printStackTrace();
}
Related
I am having a strange issue with write small files on hadoop. below is sample program
public void writeFile(Configuration conf, String message, String filename) throws Exception {
FSDataOutputStream fsDataOutputStream = null;
DistributedFileSystem fs = null;
try {
fs = (DistributedFileSystem) FileSystem.get(URI.create(properties.getHadoop().getRawLocation()), conf);
Path hdfswritepath = new Path(properties.getHadoop().getRawLocation() + "/" + filename + ".json");
fsDataOutputStream = fs.create(hdfswritepath);
fsDataOutputStream.write(message.getBytes());
fsDataOutputStream.close();
fsDataOutputStream.hsync();
} catch (IllegalArgumentException | IOException e) {
System.out.println("Got Exception");
e.printStackTrace();
throw e;
} finally {
fs.close();
System.out.println("clean up done");
}
}
Above code is creating empty file at hadoop location. here are some on item I tried
Firewall is not there between client and hadoop server
Copy from local to hadoop is working.
Issue is Only 0 byte file getting created.
I am getting below exception for this.
09:12:02,129 INFO [org.apache.hadoop.hdfs.DFSClient] (Thread-118) Exception in createBlockOutputStream: java.net.ConnectException: Connection timed out: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1533)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1309)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1262)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:448)
I was able to fix this by
conf.set("dfs.client.use.datanode.hostname", "true");
I am trying to connecting a server with FTPSClient (true implicit), port 990, and it seems the connection is ok, but it says that the file PDF inside cannot be found.
String protocol = "TLS"; // TLS / SSL
boolean isImpicit = true;
int timeoutInMillis = 3000;
FTPSClient client = new FTPSClient(protocol, isImpicit);
client.setDataTimeout(timeoutInMillis);
client.addProtocolCommandListener(new PrintCommandListener(new PrintWriter(System.out)));
try
{
int reply;
client.connect(server, port);
client.login(user, pass);
client.setFileType(FTP.BINARY_FILE_TYPE);
client.execPBSZ(0);
client.execPROT("P");
System.out.println("Connected to " + server + ".");
reply = client.getReplyCode();
if (!FTPReply.isPositiveCompletion(reply))
{
client.disconnect();
System.err.println("FTP server refused connection.");
System.exit(1);
}
client.listFiles();
boolean retrieved = client.retrieveFile(Constantes.DIRECCION_FTP_PDF_FACTURAS + nombre_factura, new FileOutputStream(Constantes.DIRECCION_FTP_LOCAL_DESCARGAS + nombre_factura));
}
catch (Exception e)
{
if (client.isConnected())
{
try
{
client.disconnect();
}
catch (IOException ex)
{
ex.printStackTrace();
}
}
System.err.println("Could not connect to server.");
e.printStackTrace();
return;
}
finally
{
System.out.println("# client disconnected");
client.disconnect();
}
}
The error I got is java.io.FileNotFoundException
I tried writing the full path since C:\ , and without it, but nothing works.
Anybody can help me?
Thanks.
EDIT: IT WORKS NOW!
The path "Program Files" contains a space and maybe FileInputStream does not manage to resolve it properly.
May give it a try to put your folder to "C:/Temp/" and test it again.
Where does this FileNotFoundException happen exactly?
I want to show list of file which is available on ftp server i am working with some demo ftp server but it is not working it is showing some error , I have to get list of all file available on ftp server here is code
public class getFTPfileList {
public static void main(String[] args) {
FTPClient client = new FTPClient();
try {
client.connect("ftp.javacodegeeks.com");
client.login("username", "password");
FTPFile[] files = client.listFiles();
for (FTPFile ftpFile : files) {
if (ftpFile.getType() == FTPFile.FILE_TYPE) {
System.out.println("File: "
+ ftpFile.getName()
+ "size-> "
+ FileUtils.byteCountToDisplaySize(ftpFile
.getSize()));
}
}
client.logout();
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
client.disconnect();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
The error message showing:
java.net.UnknownHostException: ftp.javacodegeeks.com
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(Unknown Source)
at java.net.InetAddress.getAddressesFromNameService(Unknown Source)
at java.net.InetAddress.getAllByName0(Unknown Source)
at java.net.InetAddress.getAllByName(Unknown Source)
at java.net.InetAddress.getAllByName(Unknown Source)
at java.net.InetAddress.getByName(Unknown Source)
at org.apache.commons.net.SocketClient.connect(SocketClient.java:184)
at org.apache.commons.net.SocketClient.connect(SocketClient.java:273)
at com.journaldev.servlet.getFTPfileList.main(getFTPfileList.java:21)
It seems that your FTP host is either not reachable or the host-name is incorrect and not resolved as a host-name.
See here : SocketClient#connect(String hostname)
Note : try it using IP-address of host server and ping it to check whether it is open or not.
I am trying to store an custom defined (ProjectData) Java object in a file in FTP . ProjectData class implements Serializable interface.I am using apache Commons Net FTP client.I have a custom class FtpClient , which has instance of Commons net FTP Client. I connect , upload file, download file, storeFileStream using this class.When I try to call
saveProjectData function, it throws exception while executing save.writeObject(data).
java.net.SocketException: Software caused connection abort: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method) [:1.6.0_23]
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) [:1.6.0_23]
at java.net.SocketOutputStream.write(SocketOutputStream.java:136) [:1.6.0_23]
at org.apache.commons.net.io.SocketOutputStream.write(SocketOutputStream.java:72) [:2.2]
at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1847) [:1.6.0_23]
at java.io.ObjectOutputStream$BlockDataOutputStream.flush(ObjectOutputStream.java:1792) [:1.6.0_23]
at java.io.ObjectOutputStream.flush(ObjectOutputStream.java:699) [:1.6.0_23]
So what is the problem in the code, I ready that connection is already closed, in the meantime of saving object.But I could not find where the bug.
public void saveProjectData(ProjectData data, String filePath) {
ftpClient = new FtpClient(ftpConfig, logger);
ObjectOutputStream save = null;
try {
ftpClient.connect();
save = new ObjectOutputStream(ftpClient.storeFileStream(filePath));
} catch (Exception e) {
// ignore
}
try {
if (save == null) {
ftpClient.reconnect();
save = new ObjectOutputStream(ftpClient.storeFileStream(filePath));
}
save.writeObject(data);
save.flush();
} catch (IOException e) {
logger.error("Error creating file:" + filePath);
throw new FileTransferException("Could not create file:" + filePath, e);
} finally {
if (save != null) {
try {
save.close();
} catch (Exception e) {
logger.warn("Error closing writer of file:" + filePath, e);
}
save = null;
}
if (ftpClient != null) {
try {
ftpClient.disconnect();
} catch (Exception e) {
// ignore
}
ftpClient = null;
}
}
}
/**
* Stores the file in FTP
* #param filePath
* #return
*/
public OutputStream storeFileStream(String filePath) {
try {
client.setFileType(FTP.BINARY_FILE_TYPE);
return client.storeFileStream(filePath);
} catch (Exception ioe) {
log.warn("Could not store file:" + filePath, ioe);
return null;
}
}
I read that connection is already closed
No, you read (or you should have read) that the connection is already closed by the peer.
Possibly you have exceeded an upload limit.
I am trying to retrieve files from ftp.I am getting following exception.
Sometimes concurrent access to same file can be case.My firewall is turn off.
My question is
how can get ride of this exception, because it can not download necassary files, my program does not make its work
My download code:
public boolean downloadFile(String sourceFilePath, String destinationDirPath, String newFileName)
throws IllegalStateException {
// final FTPClient client = this.getClient();
FileOutputStream output = null;
try {
client.setFileType(FTP.BINARY_FILE_TYPE);
// The remote filename to be downloaded.
String filename = new File(sourceFilePath).getName();
// Downloads the selected file to the C drive
output = new FileOutputStream(destinationDirPath + File.separator + newFileName);
this.cd(new File(sourceFilePath).getParent());
// Download file from FTP server
boolean result = client.retrieveFile(filename, output);
log.trace("File is downloaded from server:" + filename + ", to destination:"+newFileName);
return result;
}
// Indicate that an exception is occurred while downloading file
catch (Exception ioe) {
throw new FileTransferException("Could not download file", ioe);
} finally {
if (output != null) {
try {
// trying to close FileOutputStream
output.close();
} catch (IOException e) {
log.warn("Error closing writer to file:"+newFileName, e);
}
output = null;
}
}
}
And here is the stack trace:
Exception during data transfer, closing data connection socket
java.net.SocketException: Software caused connection abort: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
at org.apache.ftpserver.impl.IODataConnection.transfer(IODataConnection.java:289)
at org.apache.ftpserver.impl.IODataConnection.transferToClient(IODataConnection.java:161)
at org.apache.ftpserver.command.impl.RETR.execute(RETR.java:166)
at org.apache.ftpserver.impl.DefaultFtpHandler.messageReceived(DefaultFtpHandler.java:210)
at org.apache.ftpserver.listener.nio.FtpHandlerAdapter.messageReceived(FtpHandlerAdapter.java:61)
at