We Keep Coding

FTP using Apache commons not working - Error: 550 Dataset not found (FTPid that i am using is getting appended to filename)

I am trying to ftp the file from Linux VM to an AS400 server. I was able to login to server in passive mode but when trying to use the STOR command to upload the file getting below error:
STOR XX.YY600040.XXXZZZXXX
**550 Dataset not found, DSN=FTPID.XX.YY600040.XXXZZZXXX**
Not sure why the ftpid that i am using is getting prefixed to the filename. Is there any way to avoid it?
Below is the sample code that i am using:
private static String sendFTPFile(String fileName) throws Exception {
StringBuffer ftpMessage = new StringBuffer();
if (SHOW_DEBUG) ftpMessage.append("<ul>");
FTPClient ftp = null;
try {
String server = "****";
String username = "****";
String password = "XXXXX";
String hostDir = "";
String localFileName = fileName;
String localFilePath = "***/**/*";
boolean binaryTransfer = false, error = false;
FTPClient ftp = new FTPClient();
ftp.addProtocolCommandListener(new PrintCommandListener(
new PrintWriter(System.out)));
int reply;
ftp.connect(server)
// After connection attempt, you should check the reply code to verify
// success.
reply = ftp.getReplyCode();
if (!FTPReply.isPositiveCompletion(reply))
{
ftp.disconnect();
error = true;
}
if(!error) {
if (!ftp.login(username, password))
{
ftp.logout();
error = true;
}
if(!error) {
if (binaryTransfer)
ftp.setFileType(FTP.BINARY_FILE_TYPE);
// Use passive mode as default
ftp.enterLocalPassiveMode();
InputStream input;
input = new FileInputStream(localFilePath+localFileName);
boolean ftpSuccess = ftp.storeFile(hostDir+localFileName, input);
input.close();
if (!ftpSuccess) {
throw new Exception("File ftp error");
}else {
if (SHOW_DEBUG) ftpMessage.append("<li>isFtpSuccess()...success").append("</li>");
}
ftp.logout();
if (SHOW_DEBUG) ftpMessage.append("<li>ftp.logout()...success").append("</li>");
}
}
}
catch (Exception ex) {
ex.printStackTrace();
System.out.println("Exception occur while transfering file using ftp"+ ex.toString());
throw new Exception(ftpMessage.toString(), ex);
}
finally {
if (ftp!=null && ftp.isConnected())
{
try
{
ftp.disconnect();
}
catch (Exception e)
{
e.printStackTrace();
System.out.println("Exception occur while transfering file using ftp"+ e.toString());
throw new Exception(ftpMessage.toString(), e);
}
}
if (SHOW_DEBUG) ftpMessage.append("</ul>");
}
return ftpMessage.toString();
}

I did some more debugging on doing ftp from linux to as400. And when doing pwd after logging to ftp server its giving message as:
ftp> pwd
257 "'FTPID.'" is current prefix
And thinking how to remove that prefix so i ran the below command got output as no prefix defined:
ftp> cd ..
200 "" no prefix defined
And after that when i uploaded the file using put command it was uploaded successfully. So i did some research on how to go back one directory using Apache commons.net api's that i am using and found the CDUP method.
When i ran the FTPClient.cdup() method before uploading the file. I was able to successfully FTP the file from java code as well.
ftp.cdup();
input = new FileInputStream(localFilePath+localFileName);
boolean ftpSuccess = ftp.storeFile(hostDir+localFileName, input);

How to convert FTP from Spring MVC to Spring Boot/Thymeleaf?

So I've been working on taking the logic from an older webapp and making a new Spring Boot application out of it. I've come to a stuck spot regarding an ftp connection and call. Since I don't have a bunch of experience with this, I'm curious if there is a better/more modern way to handle most of this ftp stuff using Spring Boot/Thymeleaf and ways to go ahead and set that up. Any advice/guidance would be fantastic.
This is the older code that I'd like to modernize a bit.
String serverName = getFtpServer();
// Connect to the server
try {
ftp.connect(serverName);
ftp.enterLocalPassiveMode();
String replyText = ftp.getReplyString();
System.out.println(replyText);
} catch (Exception e) {
e.printStackTrace();
return false;
}
// Login to the server
try {
ftp.login(userName, password);
String replyText = ftp.getReplyString();
System.out.println(replyText);
} catch (Exception e) {
e.printStackTrace();
return false;
}
// Tell server that the file will have JCL records
try {
ftp.site("filetype=jes");
String replyText = ftp.getReplyString();
System.out.println(replyText);
} catch (Exception e) {
e.printStackTrace();
return false;
}
// Submit and run the JCL
try {
System.out.println("TRYING TO START MAINFRAME JCL");
submitJcl(filename, serverName);
String replyText = ftp.getReplyString();
System.out.println(replyText);
} catch (Exception e) {
String replyText = ftp.getReplyString();
System.out.println(replyText);
e.printStackTrace();
return false;
}
// Quit the server
try {
ftp.quit();
} catch (Exception e) {
e.printStackTrace();
}
Storing the file
private String submitJcl(String remoteFile, String serverName) throws IOException {
String filePath = getFilePath();
String result = "";
String fileName = filePath + remoteFile;
System.out.println("filePath = " + fileName);
FileInputStream inputStream = new FileInputStream(fileName);
ftp.storeFile(serverName, inputStream);
return result;
}

For this I figured out that there may be better ways to change it into the newer ftp format for Spring Boot, but this still completely works.
Changes I made to it regardless:
Consolidated the try/catch blocks into one.
Pushed the ftp stuff into its own function and then just called it inside the try/catch block
Changed all of the sys.out's to info.debugs.
Changed the way it gets the filePath to more of a relative path with the file stored within the system instead of user files.

Handling multiple threads in a Client-Server architecture for File synchronization

I had to implement a Client-Server application in Java that automatically updates txt files in a Server directory depending on changes in the files in the Client side, for a homework assignment (had to, because I'm past the deadline).
I have a package that handles the changes in the files correctly, but I'm stumped about how to handle the changes in multiple files. My approach was using separate threads for each file in the client directory and using corresponding threads in the server directory for the same cause. This approach works for a single file, but not for multiples.
The code below is on the client side and calls a file's thread's checkfilestate method to handle the updates.
while(true){
for (Map.Entry<String, SynchronisedFile> entry : fileList.entrySet()) {
try {
System.err.println("SyncTest: calling fromFile.CheckFileState()");
sstt.start();
entry.getValue().CheckFileState();
} catch (IOException e) {
e.printStackTrace();
System.exit(-1);
} catch (InterruptedException e) {
e.printStackTrace();
System.exit(-1);
}
try {
Thread.sleep(10000);
} catch (InterruptedException e) {
e.printStackTrace();
System.exit(-1);
}
}
And on the server side, if I start a single thread using:
Thread sstt = new Thread(new SyncThreadServer(sfileList.entrySet().iterator().next().getValue(),clientSocket));
sstt.start();
It works as expected. But if I start the serverside threads at the same time (which contains methods for decoding the Json messages from the input stream) using:
for (Map.Entry<String, SynchronisedFile> entry : sfileList.entrySet())
{
Thread sstt = new Thread(new SyncThreadServer(entry.getValue(),clientSocket));
sstt.setName(entry.getKey());
}
Threads of other files start reading JSON messages intended for other threads from the input stream. I'd like to be able to stop the serverside loop from starting the next thread, at least until the checkFile method is complete for one file/thread. But I still might run into problems after the initial stage, when all the treads are running at the same time. Any solutions on how to handle multiple threads in this case? (All threads use a single socket).
Edit: As I understand, this has to do with synchronization. Threads of other files on the server are accessing the Input stream before the first thread has even finished processing the inputs meant for it. This is the code of the server thread below. I need to somehow block the other threads from accessing the input stream before the first one has finished using it. Any help would be much appreciated. Thanks.
public class SyncThreadServer implements Runnable {
SynchronisedFile toFile; // this would be on the Server //Is an instance of the syncfile class, should be able to proc insts
Socket clientSocket;
public SyncThreadServer(SynchronisedFile tf, Socket aClientSocket){
toFile=tf;
clientSocket = aClientSocket;
}
#Override
public void run() {
Instruction inst = null;
InstructionFactory instFact=new InstructionFactory();
while(true){
{
try{
DataInputStream in = new DataInputStream(clientSocket.getInputStream());
DataOutputStream out = new DataOutputStream(clientSocket.getOutputStream());
String smsg = in.readUTF();
Instruction receivedInst = instFact.FromJSON(smsg);
System.err.println(smsg);
// The Server processes the instruction
toFile.ProcessInstruction(receivedInst);
//if(receivedInst.Type().equals("EndUpdate")){
// out.writeUTF("NEXT"); //TODO: Change to Json
// out.flush();}
//else
//{
out.writeUTF("GO"); //TODO: Change to Json
out.flush();
}
//}
catch (IOException e) {
e.printStackTrace();
System.exit(-1); // just die at the first sign of trouble
} catch (BlockUnavailableException e) {
// The server does not have the bytes referred to by the block hash.
try {
DataOutputStream out2 = new DataOutputStream(clientSocket.getOutputStream());
DataInputStream in2 = new DataInputStream(clientSocket.getInputStream());
out2.writeUTF("NGO"); //TODO: Change to Json
out2.flush();
String msg2 = in2.readUTF();
Instruction receivedInst2 = instFact.FromJSON(msg2);
toFile.ProcessInstruction(receivedInst2);
if(receivedInst2.Type().equals("EndUpdate")){
out2.writeUTF("NEXT"); //TODO: Change to Json
out2.flush();}
else
{
out2.writeUTF("GO"); //TODO: Change to Json
out2.flush();
}
} catch (IOException e1) {
e1.printStackTrace();
System.exit(-1);
} catch (BlockUnavailableException e1) {
assert(false); // a NewBlockInstruction can never throw this exception
}
}
// } //And here
}
}
}

Unable to download files from FTP server

I've created Java function that downloads files from FTP server. It works fine from my local machine. But I need to run it under linux server (means another host and port). And the function gives an error
The collection, array, map, iterator, or enumeration portion of a for statement cannot be null
Caused in a line with the code:
for(String f : ftpNames) {
ftpclient.retrieveFile(f, os); // os is OutputStream
}
So it doesn't see the files...
I added
ftpclient.enterRemotePassiveMode();
And ftpclient.getPassiveHost() returns 227 Entering Passive Mode (x,x,x,x,204,15)
Tried to list and download them via shell - it works.
How should I modify my code to solve the problem? Thanks.
UPD. I got log from FTP server I'm trying to get files from, and there is such string:
425 Cannot open data connection
Full code:
static boolean ftpFilesDownload(String ip, int port, String login, String passwd, String ftpdir, String localdir) throws IOException {
Boolean result = false;
FTPClient client = new FTPClient();
String separator = File.separator;
try {
client.connect(ip, port);
System.out.println(client.getReplyString());
client.login(login, passwd);
System.out.println(client.getReplyString());
client.setControlKeepAliveTimeout(1000*60*5);
client.setControlKeepAliveReplyTimeout(1000*60*5);
client.setFileType(FTP.BINARY_FILE_TYPE);
System.out.println("client setFileType success");
client.changeWorkingDirectory(ftpdir);
System.out.println(client.getReplyString());
client.printWorkingDirectory();
System.out.println("directory changed");
FTPFile[] ftpFiles = client.listFiles();
System.out.println(ftpFiles);
String[] ftpNames = client.listNames();
System.out.println("the files are " + Arrays.toString(ftpNames)); // so null here...
for(String f : ftpNames) {
String localfile = localdir + f;
OutputStream os = new FileOutputStream(localfile);
try {
result = client.retrieveFile(f, os);
System.out.println("DOWNLOADING STARTED);
System.out.println(client.getReplyString());
client.noop();
}
catch(Exception e) {
System.out.println(e);
result = false;
}
finally {
if(os != null)
os.close();
}
}
client.logout();
System.out.println(client.getReplyString());
}
catch(Exception e)
{
System.out.println(e);
result = false;
}
finally
{
try
{
client.disconnect();
}
catch(Exception e)
{
System.out.println(e);
result = false;
}
}
return result;
}

As the error message explains, you're trying to iterate over a null object. You should check for this (or make sure an empty Iterable is used perhaps)
If this is an execptional (error) state, I'd check for this explicitly and throw some kind of runtime exception, e.g.:
if (ftpNames == null) {
throw new IllegalArgumentException("Cannot use a null set of FTP servers");
}
for (String f : ftpNames) {
ftpclient.retrieveFile(f, os); // os is OutputStream
}
Alternatively you could try to continue with no FTP servers, but seems a bit pointless.

Try to use ftpclient.enterLocalActiveMode();

Socket closed before able to read from ObjectInputStream(BufferedInputStream(Socket.getInputStream))

I have written a small Client/Server Program which already worked once but after adding Threads and some real input Data to it, i always get a closed Socket before being able to read the Object (the String). The Program always Prints "Client has already closed Connection!" from Function handleConnection in the ProcessDataThread.
ClientCode:
synchronized private static void sendToServer(){
Socket clientSocket = null;
BufferedOutputStream socketOut = null;
ObjectOutputStream out = null;
try{
String xmlToSend = "<startTag>\n<someOtherTag id=\"5555\">\n12345\n</someOtherTag>\n</startTag>\n";
Log.d(TAG, "Trying to send the following to the Server:" + xmlToSend);
//TODO load these from file
clientSocket = new Socket( "10.0.2.2", 7777);
socketOut = new BufferedOutputStream(clientSocket.getOutputStream());
out = new ObjectOutputStream(socketOut);
out.writeObject(xmlToSend);
out.flush();
}catch(Exception ex){
Log.e(TAG, "Could not write File to Server.", ex);
}
finally{
try{
if(clientSocket != null){
clientSocket.close();
}
if(out != null){
out.close();
}
}catch(IOException ex){
Log.e(TAG, "Could not close Socket.");
}
}
}
ServerCode:
ReceiverThread:
public void run()
{
try {
ServerSocket server = new ServerSocket(port);
//Only block for 10 Seconds and try again
server.setSoTimeout(10000);
while(!server.isClosed() && !stopped){
//Run
Socket client = null;
try
{
client = server.accept();
System.out.println("Accepted ClientConnection from " + client.getRemoteSocketAddress());
new ProcessDataThread(client).start();
}
catch( SocketTimeoutException tx){
//nothing
}
catch ( IOException e ) {
e.printStackTrace();
}
finally {
if ( client != null )
try { client.close(); } catch ( IOException e ) { e.printStackTrace(); }
}
}
} catch (IOException e) {
e.printStackTrace();
}
}
ProcessDataThread:
public class ProcessDataThread extends Thread {
Socket client;
public ProcessDataThread(Socket sock) {
// xmlToProcess = xmlString;
this.client = sock;
}
private String handleConnection() {
BufferedInputStream socketIn = null;
ObjectInputStream in = null;
String xmlToProcess = null;
try {
if(!client.isClosed()){
System.out.println("Trying to read from Stream;");
socketIn = new BufferedInputStream(client.getInputStream());
in = new ObjectInputStream(socketIn);
Object xmlString = in.readObject();
System.out.println("Read some Object from Stream:" + xmlString.toString());
if (xmlString instanceof String) {
xmlToProcess = (String) xmlString;
System.out.println("Received the following XML:\n" + xmlToProcess);
}
}else{
System.out.println("Client has already closed Connection!");
}
} catch (ClassNotFoundException e) {
e.printStackTrace();
} catch (EOFException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
} finally {
try {
if (socketIn != null) {
socketIn.close();
}
if(client != null){
client.close();
}
} catch (IOException ioex) {
ioex.printStackTrace();
}
}
return xmlToProcess;
}
#Override
public void run() {
String xmlToProcess = handleConnection();
if (xmlToProcess == null || xmlToProcess.isEmpty()) {
// Es konnte kein String vom Client gelesen werden.
return;
}
System.out.println(xmlToProcess);
}
}
I made some changes with jboi's Suggestions. This is what i got now. The error stays the same. I don't even get to reading the Stream in the Server because client.getClosed()
is always true!
In the Client Code:
clientSocket = new Socket( "10.0.2.2", 7777);
clientSocket.setTcpNoDelay(true);
socketOut = new BufferedOutputStream(clientSocket.getOutputStream());
out = new ObjectOutputStream(socketOut);
out.writeObject(xmlToSend);
out.flush();
socketOut.flush();
//Close Output on Socket to signalize the Server that we finished writing!
clientSocket.shutdownOutput();
in = clientSocket.getInputStream();
byte[] receivedData = new byte[8192];
while(in.read(receivedData) != -1) {
//Wait for the Server to Close the Connection
}
In the Server Code
socketIn = new BufferedInputStream(client.getInputStream());
in = new ObjectInputStream(socketIn);
Object xmlString = in.readObject();
System.out.println("Read some Object from Stream:" + xmlString.toString());
if (xmlString instanceof String) {
xmlToProcess = (String) xmlString;
System.out.println("Received the following XML:\n" + xmlToProcess);
}
out = client.getOutputStream();
out.write(1);
//Signalize the Client that we have read everything
client.shutdownOutput();

It is very probable that your client has closed the socket in the finally block before the server was able to read the data.
In your clients finally block you should use socket.shutdownOutput, then read on the client all incoming data till EOF and then close the socket.
On your server you read till EOF and then send an object as kind of acknowledge, e.g. Number of bytes in the message. You also end the send with socket.shutdownOutput() as you've done at the client. This puts again an EOF at the end of the data. This EOF is received by the client and it will finally close the socket.

The issue seems to be the client and server are unable to identify each others state:
Client sending data to server, where server has closed the connection
Server sending/reading data to client , where client has closed the connection
Either are unable to coordinate with each other, solutions could be to establish a proper state machine. Some examples in Google if you search for (client and server state machine) gives mathematically definite state machine examples for your application: hope this comment helps.
Hence it's not useful to look into this problem in solution perspective and probably start using protocols in place like : telnet etc .

Ok now i'm feeling stupid.
I closed the Socket inside the Server Code myself.
After accepting a connection the following is executed inside the finally Block:
try {
client.close();
} catch (IOException e) {
e.printStackTrace();
}
The reason that there is this finally Block was because i didn't use Threads before so the ReceiverThread also did handle the Connection and therefore close the socket after using it.
I then moved the code to the new Thread and forgot to remove that finally block!

You can't use a buffered input stream and another kind of stream on the same socket. The buffered stream will steal data from the other one. Make up your mind. The ObjectInputStream will do everything you need. Just use that.
EDIT Re your edit, 'socket closed' means that you closed your socket and then continued to use it.