We Keep Coding

Is there a way in talend to find the files from the last 24 hours or within 1 day without reading through entire file list from ftp

The ftp folder has around 4000-5k files present there and i dont want to be passing them into a bufferinput/output component and after that sort them using tmap function .Instead i would prefer without bothering the no.of files in the ftp folder directly can i find the files from last day(24 hours timeframe) and use ftpget?
I dont want to go through the hassle of storing/reading all the files in a bufferedreader and sort/arrange by mtime_desc or by other ftpfileproperties method.Instead a direct way to fetch and retrieve the most recent files from previous day would be quite faster and effective?

I do not know Talend, but: The standard FTP can retrieve only all files in the directory. No optimization or filtering possible, no matter what language/library/framework you are using.
Some FTP servers support a non-standard LIST -t command to retrieve the listing sorted by timestamp. When you use that, you may break a download of the listing, once you get the files you need. See also Make FTP server return files listed by timestamp with Apache FTPClient.

Do I need to call sync on file descriptor after using Files move operation?

I want to move two files to a different directory in same filesystem.
Concrete example, I want to move /var/bigFile to /var/path/bigFile, and /var/smallFile to /var/path/smallFile.
Currently I use Files.move(source, target), without any options, moving the small file first and big file second. I need this order since there is another process waiting for this files to arrive, and the order is important.
Problem is that, sometimes I see the creation date for small file being greater than the creation date for the big file, like the moving order is not followed.
Initially I thought I have to do a sync, but it does not make sense.
Given the fact that the move will actually be a simple rename, there is no system buffers included, to force them to be flushed to disk.
Timestamp for the files was checked using ls -alrt command.
Does anyone have any idea what could be wrong?

Reading the time a file was added to a folder

I am writing a piece of software that is to monitor the time a file was added into a specific directory. I would need to do this in both c# and java. However, I am not so much interested in when the files was created as this could be days before they are actually moved into the directory of interest. I have been loking around, but unable to find anything. The closest I've found so far in java is:
File file = new File(yourPathHere);
long lastModified = file.lastModified();
But that does not give me the time the file was moved into the folder. Thanks for help :)

if you are using windows, have a look at this rules :
https://support.microsoft.com/en-us/kb/299648
It seems that when you move a file, it does not change its modification or creation date.
It's changed only when doing a copy.
As an alternative, you can regularly scan your folder, like every 1 minutes and when you discover a new file, you put it in a log and write it's discovery date.
As IInspectable is saying, FileSystemWatcher and FindFirstChangeNotification are probably the way to go to avoid coding a scanner

best way to check for new xml files in java

i am writing a program that parses xml files that hold tourist attractions for cities. each city has it's own xml and the nodes have info like cost, address etc... i want to have a thread on a timer to check for new xml files or more recent versions of existing ones in a specific directory. creating the thread is not the problem. i just have no idea what the best way to check for these new files or changed files is. does anyone have any suggestions as to an easy way to make do that. i was thinking of crating a csv file with names and date altered info for each file processed and then checking against this csv file when i go to check for new or altered xml, but that seems overly complicated and i would like a better solution. i have no code to offer at this point for this mechanism i am just looking for a direction to go in.
the idea is as i get xml's for different cities fitting the schema that it will update my db automatically next time the program runs or periodically if already running.

To avoid polling you should watch the directory containing the xml file. Oracle has an extensive documentation about the topic at Watching a Directory for Changes

What you are describing looks like asynchronous feeding of new info. One common pitfall on such problem is race condition : what happens if you are trying to read a file while it's being modified or if something else tries to write a file while you are reading it ? What happens if your app (or the app that edit your xml files) breaks in the middle of processing ? To avoid such problems you should move files (change name or directory) to follow their status because moves are atomical operation on normal file systems. If you want a bullet proof solution, you should have :
files being edited or transfered by an external part
files being fully edited or transfered and ready to be read by you app
files being processed
files completely processed
files containing errors (tried to process them but could not complete processing)
The 2 first are under external responsability (you just define an interface contract), the 2 latter are under yours. The cost if 4 or 5 directories (if you choose that solution), the gains are :
if there is any problem while editing-tranfering a xml file, the external app just have to restart its operation
if a file can't be processed (syntax error, oversized, ...) it is put apart for further analysis but does not prevent processing of other files
you only have to watch almost empty directories
if your app breaks in the middle of processing a file, at next start it can restart its processing.

Check if the folder and all data already exist

My Android application downloads data only the first lunch. the data is ~50 mb with ~2500 files.
1. Is it a good idea to store if the files got downloaded in SharedSettings? The problem is that if a user clears the data application (maybe by mistake), he has to redownload everything. I manually copy a prepacked database to /data/data/../databases/, is it a good idea to check if the db exists, and if no then download everything?:
if(new File(/data/data/../databases/myDB.db).exists){//dont download}
2.Is getting the folder size and checking if its the same a good way to see if the folder+data are good? or is there a better way to check if 2 folders are the same?
Thanks.

No, do not put 50MB of data into SharedSettings. That will fall over and die. A set of SharedSettings is stored in XML on disk and entirely loaded into RAM when opened. This also won't keep the user from clearing this data.
For determining whether the data has been downloaded, I would suggest just having a file you make once the download is complete indicating it is done. The user can't selectively remove files. They can clear your data, but that will also clear the sentinel file and you will know you need to re-download. (Also keep in mind you will need to deal with restarting the download if it gets interrupting in the middle.)
Also be sure you correctly handle filesystem operations as described here: http://android-developers.blogspot.com/2010/12/saving-data-safely.html

An alternate idea if you're worried about missing data files... If at any point your app looks for a file and it doesn't exist, throw an exception, pass it to a handler that shows a dialog and 'verifies' your data. You can keep a list of all needed data files, and then only download ones that don't exist. Something like a system check, if you will.
That way they don't end up downloading 50MB if they were only missing a couple files they accidentally deleted in root explorer ;-)