We Keep Coding

Recursive Copy Of Sub-Directories Fails Using FTPS (Possible Handshake Issue?) - java

I am trying to use FTPS to copy all sub-directories and files under a given directory from an FTP server to a local machine. The program I've written gets some of the sub-directories and files before bombing. Each time the program bombs is not always in the same place. The code and errors I am getting are below. Any ideas on how to get this to work?
import org.apache.commons.net.PrintCommandListener;
import org.apache.commons.net.ftp.FTP;
import org.apache.commons.net.ftp.FTPConnectionClosedException;
import org.apache.commons.net.ftp.FTPFile;
import org.apache.commons.net.ftp.FTPReply;
import org.apache.commons.net.ftp.FTPSClient;
public class SaigFileBuilder {
private String server = "host";
private String username = "user";
private String password = "pass";
private String remoteParentDir = "/prd/fa/out/FA003100/SAIG/";
private String filesep = System.getProperty("file.separator");
private String local = "c:" + filesep + "temp" + filesep + "saig_files";
private List<String> fileList = null;
private boolean error = false;
private String protocol = "TLS"; // SSL/TLS
private FTPSClient ftps = null;
public static void main(String[] args) {
SaigFileBuilder fb = new SaigFileBuilder();
try {
fb.getClientConnection();
fb.getFiles(false);
fb.logout();
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
}
System.exit(fb.error ? 1 : 0);
}
private void getClientConnection() throws NoSuchAlgorithmException {
this.ftps = new FTPSClient(this.protocol);
this.ftps.addProtocolCommandListener(new PrintCommandListener(new PrintWriter(System.out)));
this.ftps = getConnection(server);
this.ftps = getLogin(username, password);
}
private FTPSClient getConnection(String server) {
try
{
int reply;
// Connect to the FTP server...
this.ftps.connect(server);
System.out.println("Connected to " + server + ".");
// Get connect reply code...
reply = this.ftps.getReplyCode();
System.out.println("String reply: " + this.ftps.getReplyString());
// Verify successful connection reply code was returned...
if (!FTPReply.isPositiveCompletion(reply))
{
error = true;
System.err.println("FTP server refused connection.");
return null;
}
// FileZilla client issues following, so I will too..
this.ftps.sendCommand("OPTS UTF8 ON");
this.ftps.execPBSZ(0);
this.ftps.execPROT("P");
this.ftps.enterLocalPassiveMode();
}
catch (IOException e)
{
System.err.println("Could not connect to server.");
e.printStackTrace();
}
return this.ftps;
}
private FTPSClient getLogin(String username, String password) {
try
{
this.ftps.setBufferSize(1000);
if (!this.ftps.login(username, password))
{
System.out.println("Login failed");
this.ftps.logout();
return null;
}
System.out.println("Remote system is " + this.ftps.getSystemType());
}
catch (FTPConnectionClosedException e)
{
error = true;
System.err.println("Server closed connection.");
e.printStackTrace();
}
catch (IOException e)
{
error = true;
e.printStackTrace();
}
return this.ftps;
}
private void getFiles(boolean binaryTransfer) {
try
{
if (binaryTransfer) {
this.ftps.setFileType(FTP.BINARY_FILE_TYPE);
}
OutputStream output = null;
this.ftps.changeWorkingDirectory(this.remoteParentDir);
System.out.println("pwd: " + this.ftps.printWorkingDirectory());
this.fileList = new ArrayList<String>();
scanDirTree(this.remoteParentDir);
System.out.println("Finished scanning...");
for (String file : this.fileList) {
System.out.println(file);
}
String dirHold = " ";
String winDir = " ";
String winFile = " ";
for (String ftpFilePath : this.fileList) {
// Going to copy the remote dirs and files to local, so change the paths to
// use local Windows file separator instead of remote Unix...
winDir = ftpFilePath.substring(0, ftpFilePath.lastIndexOf("/") + 1).replace("/", filesep);
winFile = ftpFilePath.substring(ftpFilePath.lastIndexOf("/") + 1, ftpFilePath.length());
System.out.println("winDir.: " + winDir);
System.out.println("winFile: " + winFile);
// If new dir found in list, then create it locally...
if (dirHold.equals(winDir) == false) {
System.out.println("mkdir: " + local + winDir);
new File(local + winDir).mkdirs();
dirHold = winDir;
}
// And write the file locally...
System.out.println("Attempting to write: " + this.local + winDir + winFile);
output = new FileOutputStream(local + winDir + winFile);
this.ftps.retrieveFile(ftpFilePath, output);
output.close();
output = null;
}
}
catch (FTPConnectionClosedException e)
{
error = true;
System.err.println("Server closed connection.");
e.printStackTrace();
}
catch (IOException e)
{
error = true;
e.printStackTrace();
}
}
private void scanDirTree(String directory) throws IOException {
// Create and return a list of all dirs and files on remote
// server below the parent dir that was passed to this method...
FTPFile[] files = this.ftps.listFiles(directory);
if (files.length == 0) {
System.out.println("no files in " + directory);
}
for (FTPFile file : files) {
String name = file.getName();
if (!".".equals(name) && !"..".equals(name)) {
if (file.isDirectory()) {
// Recursive call, go deeper...
scanDirTree(directory + name + "/");
} else {
fileList.add(directory + file.getName());
System.out.println("Added: " + directory + file.getName());
}
}
}
}
private void logout() {
if (this.ftps.isConnected()) {
try {
System.out.println("Logging out...");
this.ftps.logout();
System.out.println("Disconnecting...");
this.ftps.disconnect();
}
catch (IOException e) {
error = true;
}
}
}
}
Here is the error:
220 (vsFTPd 2.0.5)
AUTH TLS
234 Proceed with negotiation.
Connected to xxx.xxx.xxx
String reply: 234 Proceed with negotiation.
OPTS UTF8 ON
200 Always in UTF8 mode.
PBSZ 0
200 PBSZ set to 0.
PROT P
200 PROT now Private.
USER xxx
331 Please specify the password.
PASS xxxx
230 Login successful.
SYST
215 UNIX Type: L8
Remote system is UNIX Type: L8
CWD /prd/fa/out/FA003100/SAIG/
250 Directory successfully changed.
PWD
257 "/prd/fa/out/FA003100/SAIG"
pwd: /prd/fa/out/FA003100/SAIG
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,29)
LIST /prd/fa/out/FA003100/SAIG/
150 Here comes the directory listing.
226 Directory send OK.
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,46)
LIST /prd/fa/out/FA003100/SAIG/2012/
150 Here comes the directory listing.
226 Directory send OK.
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,150)
LIST /prd/fa/out/FA003100/SAIG/2012/DirectLoan/
150 Here comes the directory listing.
226 Directory send OK.
Added: /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-G.xml
Added: /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-GCHG.xml
Added: /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-GDISB.xml
Added: /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-U.xml
Added: /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-UCHG.xml
Added: /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-UDISB.xml
Added: /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN.xml
PASV
227 Entering Passive Mode (xx,xx,xxx,x,196,232)
LIST /prd/fa/out/FA003100/SAIG/2012/Isir/
150 Here comes the directory listing.
226 Directory send OK.
no files in /prd/fa/out/FA003100/SAIG/2012/Isir/
PASV
227 Entering Passive Mode (xx,xx,xxx,x,196,254)
LIST /prd/fa/out/FA003100/SAIG/2012/Pell/
150 Here comes the directory listing.
226 Directory send OK.
Added: /prd/fa/out/FA003100/SAIG/2012/Pell/CRPG12IN.xml
Added: /prd/fa/out/FA003100/SAIG/2012/Pell/CRPG12IN_DISB.xml
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,5)
LIST /prd/fa/out/FA003100/SAIG/2012/Teach/
150 Here comes the directory listing.
226 Directory send OK.
no files in /prd/fa/out/FA003100/SAIG/2012/Teach/
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,66)
LIST /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/
150 Here comes the directory listing.
226 Directory send OK.
Added: /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.01
Added: /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.02
Added: /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.03
Added: /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.04
Added: /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.05
Added: /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.06
Added: /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.07
Added: /prd/fa/out/FA003100/SAIG/2012/TrnsMonitorTRNINFIN.01
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,3)
LIST /prd/fa/out/FA003100/SAIG/2013/
150 Here comes the directory listing.
226 Directory send OK.
PASV
227 Entering Passive Mode (xx,xx,xxx,x,196,230)
LIST /prd/fa/out/FA003100/SAIG/2013/CommRec/
150 Here comes the directory listing.
226 Directory send OK.
no files in /prd/fa/out/FA003100/SAIG/2013/CommRec/
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,135)
LIST /prd/fa/out/FA003100/SAIG/2013/DLReconciliation/
150 Here comes the directory listing.
226 Directory send OK.
no files in /prd/fa/out/FA003100/SAIG/2013/DLReconciliation/
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,124)
LIST /prd/fa/out/FA003100/SAIG/2013/DirectLoan/
150 Here comes the directory listing.
226 Directory send OK.
Added: /prd/fa/out/FA003100/SAIG/2013/DirectLoan/CRDL13IN-G.xml
Added: /prd/fa/out/FA003100/SAIG/2013/DirectLoan/CRDL13IN-GCHG.xml
Added: /prd/fa/out/FA003100/SAIG/2013/DirectLoan/CRDL13IN-GDISB.xml
Added: /prd/fa/out/FA003100/SAIG/2013/DirectLoan/CRDL13IN-U.xml
Added: /prd/fa/out/FA003100/SAIG/2013/DirectLoan/CRDL13IN-UCHG.xml
Added: /prd/fa/out/FA003100/SAIG/2013/DirectLoan/CRDL13IN-UDISB.xml
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,121)
LIST /prd/fa/out/FA003100/SAIG/2013/Isir/
150 Here comes the directory listing.
226 Directory send OK.
Added: /prd/fa/out/FA003100/SAIG/2013/Isir/CORR13IN.001
Added: /prd/fa/out/FA003100/SAIG/2013/Isir/CORR13IN.002
Added: /prd/fa/out/FA003100/SAIG/2013/Isir/CORR13IN.003
Added: /prd/fa/out/FA003100/SAIG/2013/Isir/CORR13IN.004
Added: /prd/fa/out/FA003100/SAIG/2013/Isir/CORR13IN.005
Added: /prd/fa/out/FA003100/SAIG/2013/Isir/CORR13IN.006
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,147)
LIST /prd/fa/out/FA003100/SAIG/2013/Pell/
150 Here comes the directory listing.
226 Directory send OK.
Added: /prd/fa/out/FA003100/SAIG/2013/Pell/CRPG13IN.xml
Added: /prd/fa/out/FA003100/SAIG/2013/Pell/CRPG13IN_DISB.xml
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,63)
LIST /prd/fa/out/FA003100/SAIG/2013/SysGen/
150 Here comes the directory listing.
226 Directory send OK.
no files in /prd/fa/out/FA003100/SAIG/2013/SysGen/
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,118)
LIST /prd/fa/out/FA003100/SAIG/2013/TrnsMonitor/
150 Here comes the directory listing.
226 Directory send OK.
Added: /prd/fa/out/FA003100/SAIG/2013/TrnsMonitor/TRNINFIN.01
Finished scanning...
/prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-G.xml
/prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-GCHG.xml
/prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-GDISB.xml
/prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-U.xml
/prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-UCHG.xml
/prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-UDISB.xml
/prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN.xml
/prd/fa/out/FA003100/SAIG/2012/Pell/CRPG12IN.xml
/prd/fa/out/FA003100/SAIG/2012/Pell/CRPG12IN_DISB.xml
/prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.01
/prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.02
/prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.03
/prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.04
/prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.05
/prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.06
/prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.07
/prd/fa/out/FA003100/SAIG/2012/TrnsMonitorTRNINFIN.01
/prd/fa/out/FA003100/SAIG/2013/DirectLoan/CRDL13IN-G.xml
/prd/fa/out/FA003100/SAIG/2013/DirectLoan/CRDL13IN-GCHG.xml
/prd/fa/out/FA003100/SAIG/2013/DirectLoan/CRDL13IN-GDISB.xml
/prd/fa/out/FA003100/SAIG/2013/DirectLoan/CRDL13IN-U.xml
/prd/fa/out/FA003100/SAIG/2013/DirectLoan/CRDL13IN-UCHG.xml
/prd/fa/out/FA003100/SAIG/2013/DirectLoan/CRDL13IN-UDISB.xml
/prd/fa/out/FA003100/SAIG/2013/Isir/CORR13IN.001
/prd/fa/out/FA003100/SAIG/2013/Isir/CORR13IN.002
/prd/fa/out/FA003100/SAIG/2013/Isir/CORR13IN.003
/prd/fa/out/FA003100/SAIG/2013/Isir/CORR13IN.004
/prd/fa/out/FA003100/SAIG/2013/Isir/CORR13IN.005
/prd/fa/out/FA003100/SAIG/2013/Isir/CORR13IN.006
/prd/fa/out/FA003100/SAIG/2013/Pell/CRPG13IN.xml
/prd/fa/out/FA003100/SAIG/2013/Pell/CRPG13IN_DISB.xml
/prd/fa/out/FA003100/SAIG/2013/TrnsMonitor/TRNINFIN.01
winDir.: \prd\fa\out\FA003100\SAIG\2012\DirectLoan\
winFile: CRDL12IN-G.xml
mkdir: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\DirectLoan\
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\DirectLoan\CRDL12IN-G.xml
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,16)
RETR /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-G.xml
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-G.xml (4561 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\DirectLoan\
winFile: CRDL12IN-GCHG.xml
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\DirectLoan\CRDL12IN-GCHG.xml
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,136)
RETR /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-GCHG.xml
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-GCHG.xml (9074 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\DirectLoan\
winFile: CRDL12IN-GDISB.xml
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\DirectLoan\CRDL12IN-GDISB.xml
PASV
227 Entering Passive Mode (xx,xx,xxx,x,196,239)
RETR /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-GDISB.xml
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-GDISB.xml (2047 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\DirectLoan\
winFile: CRDL12IN-U.xml
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\DirectLoan\CRDL12IN-U.xml
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,137)
RETR /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-U.xml
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-U.xml (3367 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\DirectLoan\
winFile: CRDL12IN-UCHG.xml
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\DirectLoan\CRDL12IN-UCHG.xml
PASV
227 Entering Passive Mode (xx,xx,xxx,x,196,243)
RETR /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-UCHG.xml
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-UCHG.xml (7503 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\DirectLoan\
winFile: CRDL12IN-UDISB.xml
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\DirectLoan\CRDL12IN-UDISB.xml
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,87)
RETR /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-UDISB.xml
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-UDISB.xml (2995 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\DirectLoan\
winFile: CRDL12IN.xml
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\DirectLoan\CRDL12IN.xml
PASV
227 Entering Passive Mode (xx,xx,xxx,x,196,251)
RETR /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN.xml
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN.xml (145846 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\Pell\
winFile: CRPG12IN.xml
mkdir: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\Pell\
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\Pell\CRPG12IN.xml
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,123)
RETR /prd/fa/out/FA003100/SAIG/2012/Pell/CRPG12IN.xml
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/Pell/CRPG12IN.xml (2057 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\Pell\
winFile: CRPG12IN_DISB.xml
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\Pell\CRPG12IN_DISB.xml
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,21)
RETR /prd/fa/out/FA003100/SAIG/2012/Pell/CRPG12IN_DISB.xml
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/Pell/CRPG12IN_DISB.xml (9709 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\
winFile: TRNINFIN.01
mkdir: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\TRNINFIN.01
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,119)
RETR /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.01
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.01 (453 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\
winFile: TRNINFIN.02
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\TRNINFIN.02
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,101)
RETR /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.02
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.02 (604 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\
winFile: TRNINFIN.03
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\TRNINFIN.03
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,109)
RETR /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.03
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.03 (453 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\
winFile: TRNINFIN.04
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\TRNINFIN.04
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,18)
RETR /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.04
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.04 (453 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\
winFile: TRNINFIN.05
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\TRNINFIN.05
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,84)
RETR /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.05
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.05 (604 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\
winFile: TRNINFIN.06
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\TRNINFIN.06
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,70)
RETR /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.06
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.06 (453 bytes).
226 File send OK.
winDir.: \prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\
winFile: TRNINFIN.07
Attempting to write: c:\temp\saig_files\prd\fa\out\FA003100\SAIG\2012\TrnsMonitor\TRNINFIN.07
PASV
227 Entering Passive Mode (xx,xx,xxx,x,197,119)
RETR /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.07
150 Opening BINARY mode data connection for /prd/fa/out/FA003100/SAIG/2012/TrnsMonitor/TRNINFIN.07 (3624 bytes).
org.apache.commons.net.io.CopyStreamException: IOException caught while copying.
at org.apache.commons.net.io.Util.copyStream(Util.java:135)
at org.apache.commons.net.ftp.FTPClient._retrieveFile(FTPClient.java:1695)
at org.apache.commons.net.ftp.FTPClient.retrieveFile(FTPClient.java:1669)
at edu.ohio.saig.SaigFileBuilder.getFiles(SaigFileBuilder.java:196)
at edu.ohio.saig.SaigFileBuilder.main(SaigFileBuilder.java:63)
Caused by: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
at java.io.BufferedInputStream.read1(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(Unknown Source)
at java.io.PushbackInputStream.read(Unknown Source)
Logging out...
at org.apache.commons.net.io.FromNetASCIIInputStream.read(FromNetASCIIInputStream.java:161)
at org.apache.commons.net.io.FromNetASCIIInputStream.read(FromNetASCIIInputStream.java:139)
at org.apache.commons.net.io.Util.copyStream(Util.java:101)
... 4 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
... 15 more
QUIT
Note I can use FileZilla to manually copy the parent directory to the local machine and all of the sub-directories and files are copied automatically. Here is the log from a FileZilla session:
Status: Resolving address of xxx.xxx.xxx
Status: Connecting to xx.xx.xxx.x:21...
Status: Connection established, waiting for welcome message...
Response: 220 (vsFTPd 2.0.5)
Command: AUTH TLS
Response: 234 Proceed with negotiation.
Status: Initializing TLS...
Status: Verifying certificate...
Command: USER xxxxx
Status: TLS/SSL connection established.
Response: 331 Please specify the password.
Command: PASS ************
Response: 230 Login successful.
Command: OPTS UTF8 ON
Response: 200 Always in UTF8 mode.
Command: PBSZ 0
Response: 200 PBSZ set to 0.
Command: PROT P
Response: 200 PROT now Private.
Status: Connected
Status: Sending keep-alive command
Command: TYPE A
Response: 200 Switching to ASCII mode.
Status: Sending keep-alive command
Command: PWD
Response: 257 "/prd/fa/out/FA003100/SAIG/2013/TrnsMonitor"
Status: Starting download of /prd/fa/out/FA003100/SAIG/2012/TrnsMonitorTRNINFIN.01
Command: CWD /prd/fa/out/FA003100/SAIG/2012
Status: Starting download of /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN.xml
Command: CWD /prd/fa/out/FA003100/SAIG/2012/DirectLoan
Response: 250 Directory successfully changed.
Command: PASV
Response: 250 Directory successfully changed.
Command: PASV
Response: 227 Entering Passive Mode (xx,xx,xxx,x,197,141)
Command: RETR TrnsMonitorTRNINFIN.01
Response: 227 Entering Passive Mode (xx,xx,xxx,x,197,118)
Command: RETR CRDL12IN.xml
Response: 150 Opening BINARY mode data connection for TrnsMonitorTRNINFIN.01 (3775 bytes).
Response: 150 Opening BINARY mode data connection for CRDL12IN.xml (145846 bytes).
Response: 226 File send OK.
Status: File transfer successful, transferred 3,775 bytes in 1 second
Status: Starting download of /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-UDISB.xml
Command: CWD /prd/fa/out/FA003100/SAIG/2012/DirectLoan
Response: 250 Directory successfully changed.
Command: TYPE A
Response: 200 Switching to ASCII mode.
Command: PASV
Response: 227 Entering Passive Mode (xx,xx,xxx,x,197,53)
Command: RETR CRDL12IN-UDISB.xml
Response: 150 Opening BINARY mode data connection for CRDL12IN-UDISB.xml (2995 bytes).
Response: 226 File send OK.
Status: File transfer successful, transferred 145,846 bytes in 1 second
Status: Starting download of /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-UCHG.xml
Command: PASV
Response: 226 File send OK.
Status: File transfer successful, transferred 2,995 bytes in 1 second
Status: Starting download of /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-U.xml
Command: PASV
Response: 227 Entering Passive Mode (xx,xx,xxx,x,197,123)
Command: RETR CRDL12IN-UCHG.xml
Response: 227 Entering Passive Mode (xx,xx,xxx,x,197,51)
Command: RETR CRDL12IN-U.xml
Response: 150 Opening BINARY mode data connection for CRDL12IN-UCHG.xml (7503 bytes).
Response: 150 Opening BINARY mode data connection for CRDL12IN-U.xml (3367 bytes).
Response: 226 File send OK.
Status: File transfer successful, transferred 3,367 bytes in 1 second
Status: Starting download of /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-GDISB.xml
Command: PASV
Response: 226 File send OK.
Status: File transfer successful, transferred 7,503 bytes in 1 second
Status: Starting download of /prd/fa/out/FA003100/SAIG/2012/DirectLoan/CRDL12IN-GCHG.xml
Command: PASV
Response: 227 Entering Passive Mode (xx,xx,xxx,x,197,25)
Command: RETR CRDL12IN-GDISB.xml
Response: 227 Entering Passive Mode (xx,xx,xxx,x,197,28)
Command: RETR CRDL12IN-GCHG.xml
Response: 150 Opening BINARY mode data connection for CRDL12IN-GDISB.xml (2047 bytes).
Response: 150 Opening BINARY mode data connection for CRDL12IN-GCHG.xml (9074 bytes).
Response: 226 File send OK.
.
.
.
Status: Disconnected from server

Ok, I've found the cause of the error but am unsure how to work around it. The error is caused when the method retrieveFile() is called multiple times. It appears to me the method issues a PASV command to the FTP server. The server responds with a random port number for the client to retrieve the file. However, the server sometimes reuses a port and when that happens, an exception is thrown when the RETR command is sent.
I am not sure if I need to disconnect and reconnect/login to continue. I setup a small test within the exception handling to reconnect, but the test is throwing an error javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection.
I'll keep working at it, but any comments/help would be welcome.

Related

I am trying to upload file to ftp server using Java.
What i got so far is
fun uploadData(): Boolean {
val ftpClient = FTPSClient()
ftpClient.addProtocolCommandListener(
PrintCommandListener(PrintWriter(OutputStreamWriter(System.out, "UTF-8")), true)
)
ftpClient.connect(ftpProperties.server)
try {
ftpClient.enterLocalPassiveMode()
ftpClient.login(ftpProperties.username, ftpProperties.password).takeIf { !it }?.let {
log.error("cannot login to ftp")
throw Exception("cannot login to ftp")
}
ftpClient.enterLocalPassiveMode()
ftpClient.soTimeout = 10000
ftpClient.dataTimeout = Duration.ofSeconds(10)
ftpClient.execPBSZ(0)
ftpClient.execPROT("P")
ftpClient.changeWorkingDirectory("/")
ftpClient.setFileType(FTP.BINARY_FILE_TYPE)
ftpClient.enterLocalPassiveMode()
} catch (ie: IOException) {
log.error("ftp initialization error", ie)
throw Exception("ftp initialization error")
}
val remoteFile = "top.txt.gz"
val data = FileInputStream(File("top.txt.gz"))
var done = false
try {
done = ftpClient.storeFile(remoteFile, data)
} catch (e: Exception) {
log.error(e) { "error" }
}
data.close()
if (done) {
return true
}
log.error { "${ftpClient.replyCode} ${ftpClient.replyString}" }
throw RuntimeException("File not stored $remoteFile")
}
Code does not seem to be working properly. After starting it i get error code on line with code:
done = ftpClient.storeFile(remoteFile, data)
org.apache.commons.net.io.CopyStreamException: IOException caught while copying.
Caused by: java.net.SocketException: Connection reset by peer
I could not find anything wrong in ftp log
220 Private FTP server
AUTH TLS
234 Proceed with negotiation.
USER *******
331 Please specify the password.
PASS *******
230 Login successful.
PBSZ 0
200 PBSZ set to 0.
PROT P
200 PROT now Private.
CWD /
250 Directory successfully changed.
TYPE I
200 Switching to Binary mode.
PASV
227 Entering Passive Mode (x,x,x,x,x,x).
STOR top.txt.gz
150 Ok to send data.
I am able to connect and upload files to that ftp server from the same machine that i start Java/Kotlin code, using filezilla.
Filezilla log looks like that:
Status: Connection established, waiting for welcome message...
Response: 220 Private FTP server
Command: AUTH TLS
Response: 234 Proceed with negotiation.
Status: Initializing TLS...
Status: Verifying certificate...
Status: TLS connection established.
Command: USER k8s_search
Response: 331 Please specify the password.
Command: PASS ******************
Response: 230 Login successful.
Status: Server does not support non-ASCII characters.
Command: PBSZ 0
Response: 200 PBSZ set to 0.
Command: PROT P
Response: 200 PROT now Private.
Status: Logged in
Status: Starting upload of /home/x/x/top.txt.gz
Command: CWD /
Response: 250 Directory successfully changed.
Command: TYPE I
Response: 200 Switching to Binary mode.
Command: PASV
Response: 227 Entering Passive Mode (x,x,x,x,x,x).
Command: STOR top.txt.gz
Response: 150 Ok to send data.
Response: 226 Transfer complete.
Status: File transfer successful, transferred 2.6 MB in 1 second
I connot figure out if problem is in my code or is it anything else. Filezilla working correctly indicates that problem lies in code.

I'm trying to get a JSON from a server to use it in a Python code. For test purposes, I did POST by curl:
$ curl -u trial:trial -H "Content-Type: application/json"
-X POST -d '{"BP_TSM":"22"}' http://some-host --trace-ascii -
My Java code seems to correctly handle creating JSON as a response. Please look at the result of curl command:
== Info: About to connect() to localhost port 8080 (#0)
== Info: Trying ::1...
== Info: Connected to localhost (::1) port 8080 (#0)
== Info: Server auth using Basic with user 'trial'
=> Send header, 224 bytes (0xe0)
0000: POST /get/auth HTT
0040: P/1.1
0047: Authorization: Basic dHJpYWw6dHJpYWw=
006e: User-Agent: curl/7.29.0
0087: Host: localhost:8080
009d: Accept: */*
00aa: Content-Type: application/json
00ca: Content-Length: 15
00de:
=> Send data, 15 bytes (0xf)
0000: {"BP_TSM":"22"}
== Info: upload completely sent off: 15 out of 15 bytes
<= Recv header, 23 bytes (0x17)
0000: HTTP/1.1 202 Accepted
<= Recv header, 34 bytes (0x22)
0000: Server: Payara Micro #badassfish
<= Recv header, 32 bytes (0x20)
0000: Content-Type: application/json
<= Recv header, 37 bytes (0x25)
0000: Date: Thu, 22 Mar 2018 14:30:43 GMT
<= Recv header, 21 bytes (0x15)
0000: Content-Length: 108
<= Recv header, 29 bytes (0x1d)
0000: X-Frame-Options: SAMEORIGIN
<= Recv header, 2 bytes (0x2)
0000:
<= Recv data, 108 bytes (0x6c)
0000: {"title":"Free Music Archive - Albums","message":"","total":"112
0040: 59","total_pages":2252,"page":1,"limit":"5"}
{"title":"Free Music Archive - Albums","message":"","total":"11259","total_pages
":2252,"page":1,"limit":"5"}== Info: Connection #0 to host localhost left intact
Now I would like Python script be able to receive the same message that curl did. I wrote the following Python code (note I'm not Python developer):
import pickle
import requests
import codecs
import json
from requests.auth import HTTPBasicAuth
from random import randint
req = requests.get('server/get/auth', auth=HTTPBasicAuth('trial', 'trial'))
return pickle.dumps(req)
Unfortunately, I get error message 'unicode' object has no attribute 'copy' when return pickle.dumps(req) command is executed. I also tried using return json.dumps(req) but this time I get another error:
Traceback (most recent call last):
File "/tmp/tmp8DfLJ7/usercode.py", line 16, in the_function
return json.dumps(req)
File "/usr/lib64/python2.7/json/__init__.py", line 244, in dumps
return _default_encoder.encode(obj)
File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode
return _iterencode(o, 0)
File "/usr/lib64/python2.7/json/encoder.py", line 184, in default
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: <Response [405]> is not JSON serializable
Do I have some error in Python code or is it fault of my Java server returning incorrect JSON?

There are a number of errors in your Python code.
You are using request.get to POST. Instead, use request.post.
You are not passing the BP_TSM json string into your request. Use data= in your request.post.
You are not emulating the -H switch to curl. Use headers= in your request.post.
You are using pickle for no apparent reason. Don't do that.
You are using a return statement when you are not in a function. Don't do that. If you want to print to stdout, use print() or sys.stdout.write() instead.
If you actually want to use the returned variables from the JSON (as opposed to simply printing to stdout), you shoud invoke req.json().
Here is a version of your code with problems addressed.
import requests
import json
import sys
from requests.auth import HTTPBasicAuth
data = '{"BP_TSM": "22"}' # curl -d
headers = {'content-type': 'application/json'} # curl -H
auth = HTTPBasicAuth('trial', 'trial') # curl -u
req = requests.post( # curl -X POST
'http://httpbin.org/post',
auth=auth,
data=data,
headers=headers)
sys.stdout.write(req.text) # Display JSON on stdout
returned_data = req.json()
my_ip = returned_data["origin"] # Query value from JSON
print("My public IP is", my_ip)

You're trying to dumps a Response object.
Try returning req.json() or calling json.loads(req.text)

In order to load the Json string, you'll need to use json.loads(req.text).
You must also ensure that the req string is valid json.
eg
'{"FOO":"BAR"}'

You can use requests.json() method to get json response as dict
req = requests.get('http://yourdomain.com/your/path', auth=HTTPBasicAuth('trial', 'trial'))
mydict = req.json()

I wrote a java program to implement smtp.
My input consists of five arguments:
localhost 25 localhost.com loaclhost.com "hellooo"
I am getting the following error:
D:trying to connect to server
D:connected
D:response: 220 localhost ESMTP server ready.
D:sending command: HELO SRIJA-PC
D:response: 250 localhost Hello, SRIJA-PC.
D:sending command: MAIL from: localhost.com
D:response: 250 Sender OK - send RCPTs.
D:sending command: RCPT to: localhost.com
D:response: 553 Invalid RFC821 mailbox specification.
Error while sending: Error while executing cmd RCPT to: localhost.com:553-Invalid RFC821 mailbox specification.
Can you please help me to solve the error?

The error message states, that your command RCPT to: localhost.com was faulty.
RFC821 ist the specification for the SMTP protocol.
The error code 553 represents the following:
553 Requested action not taken: mailbox name not allowed [E.g., mailbox syntax incorrect]
So you simply did not enter a correct recipient. Try to enter a full e-mail address which contains a # symbol.
This is a simple example taken from the RFC821 showing how to send an e-mail correctly:
R: 220 BBN-UNIX.ARPA Simple Mail Transfer Service Ready
S: HELO USC-ISIF.ARPA
R: 250 BBN-UNIX.ARPA
S: MAIL FROM:<Smith#USC-ISIF.ARPA>
R: 250 OK
S: RCPT TO:<Jones#BBN-UNIX.ARPA>
R: 250 OK
S: RCPT TO:<Green#BBN-UNIX.ARPA>
R: 550 No such user here
S: RCPT TO:<Brown#BBN-UNIX.ARPA>
R: 250 OK
S: DATA
R: 354 Start mail input; end with <CRLF>.<CRLF>
S: Blah blah blah...
S: ...etc. etc. etc.
S: .
R: 250 OK
S: QUIT
R: 221 BBN-UNIX.ARPA Service closing transmission channel

The RCPT to: command is supposed to contain the email address of the intended recipient, e.g. a.user#localhost.com, but you appear to have entered RCPT to: localhost.com. See Section 3.3 of RFC 5321 (https://www.rfc-editor.org/rfc/rfc5321#section-3.3):
The second step in the procedure is the RCPT command. This step of
the procedure can be repeated any number of times.
RCPT TO:<forward-path> [ SP <rcpt-parameters> ] <CRLF>
The first or only argument to this command includes a forward-path
(normally a mailbox and domain, always surrounded by "<" and ">" brackets) identifying one recipient.
"localhost.com" is not a valid email address, hence the mail server is replying with "Invalid RFC821 mailbox specification".

Dear all I have a problem with my code. My ftp client is sometimes hanging when it is retrieving a file with "ftpClient.retrieveFileStream()". Any idea what I can do? I am using apache commons FTP client v3.4.
I tried also with "ftpClient.retrieveFile()" but it did not make any difference.
setting the connection and scoket timeout did not help
setting the passive mode did not help
increasing the buffer did not help
ftpClient = new FTPClient();
ftpClient.setBufferSize(1024 * 1024);
ftpClient.setConnectTimeout(10000);
ftpClient.connect(serverAddress);
ftpClient.login(userName,password);
ftpClient.setFileType(FTP.BINARY_FILE_TYPE);
ftpClient.enterLocalPassiveMode();
ftpClient.setSoTimeout(10000);
log("retriving file " + remoteFilePath + "...");
OutputStream output = new FileOutputStream(localFilePath);
InputStream inputStream = ftpClient.retrieveFileStream(remoteFilePath); // here it is sometimes hanging.
try {
IOUtils.copy(inputStream, output);
output.flush();
}
finally {
IOUtils.closeQuietly(output);
IOUtils.closeQuietly(inputStream);
ftpClient.completePendingCommand();
}
the last message I retrieve are looking usually like this:
19:58:21.145 [main] INFO FTPDAO: retrieving file /Modell/5250/Artikel/25777/lg_keh68bgh_1.jpg...
19:58:21.145 [main] INFO Log4JFTPCommandListener - >> PASV
19:58:21.462 [main] INFO Log4JFTPCommandListener - << 227 Entering Passive Mode (195,216,65,165,168,240)
19:58:21.476 [main] INFO Log4JFTPCommandListener - >> RETR /Modell/5250/Artikel /25777/lg_keh68bgh_1.jpg
19:58:21.490 [main] INFO Log4JFTPCommandListener - << 150 Connection accepted
When I suspend the programm with the debugger, then it looks that is hanging at
java.net.SocketInputStream.socketRead0(FileDescriptor, byte[], int, int, int)
java.net.SocketInputStream.socketRead(FileDescriptor, byte[], int, int, int) line: 116
java.net.SocketInputStream.read(byte[], int, int, int) line: 170
java.net.SocketInputStream.read(byte[], int, int) line: 141
regards
Mark

I had this problem too. I changed retrieveFileStream to retrieveFile. and it doesn't hang there anymore.

I am running curl from cygwin like this
$ curl -v -i -X POST 'http://www.abc.com:10123/myapp/abc' -H "Content-Type: application/json" --data-binary "#input.json"
It starts connecting and hangs at this line STATE: CONNECT => WAITCONNECT handle 0x8001f150; line 1074 (connection #0) for 149986ms and then is able to connect and send the request (traces shown below). Now the question is why it waits for that long? How can I skip this wait time?
Thanks.
STATE: INIT => CONNECT handle 0x8001f150; line 1027 (connection #-5000)
About to connect() to www.abc.com port 10123 (#0)
Trying ::1...
Adding handle: conn: 0x80059548
Adding handle: send: 0
Adding handle: recv: 0
Curl_addHandleToPipeline: length: 1
0x8001f150 is at send pipe head!
- Conn 0 (0x80059548) send_pipe: 1, recv_pipe: 0
STATE: CONNECT => WAITCONNECT handle 0x8001f150; line 1074 (connection #0)
After 149986ms connect time, move on!
Trying 127.0.0.1...
Connected to www.abc.com (127.0.0.1) port 10