I need to create a java program which will create thread to search for a file in particular folder(source folder) and pick the file immediately for process work(convert it into csv file format) once it found the file in the source folder. Problem i am facing now is file which comes to source folder is big size(FTP tool is used to copy file from server to source folder), thread is picking that file immediately before it copies fully to source folder and throwing exception. How do i stop thread until the file copy into source folder completely?. It has to pick the file for processing only after the file is copied completely into source folder.
Tha safest way is to download the file to a different location and then move it to the target folder.
Another variation mentioned by Bombe is to change the file name to some other extension after downloading and look only for files with that extension.
I only read the file which is not in write mode. This is safest as this means no other process is writing in this file. You can check if file is not in write mode by using canWrite method of File class.
This solution works fine for me as I also have the exact same scenario you facing.
You could try different things:
Repeatedly check the last modification date and the size of the file until it doesn’t change anymore for a given amount of time, then process it. (As pointed out by qbeuek this is neither safe nor deterministic.)
Only process files with names that match certain criteria (e.g. *.dat). Change the FTP upload/download process to upload/download files with a different name (e.g. *.dat.temp) and rename the files once they are complete.
Download the files to a different location and move them to your processing directory once they’re complete.
As Vinegar said, if it doesn’t work the first time, try again later. :)
If you have some control on the process that does the FTP you could potentially have it create a "flag file" in the source directory immediately AFTER the ftp for the big file is finished.
Then your Java thread has to check the presence of this flag file, if it's present then there is a file ready to be processed in the source directory. Before processing the big file, the thread should remove the flag file.
Flag file can be anything (even an empty file).
Assuming you have no control over FTP process...
Let it be like this. When you get the exception, then try to process it again next time. Repeat it until the file gets processed. Its good to keep few attributes in case of exception to check it later, like; name, last-modified, size.
Check the exact exception before deciding to process it later, the exception might occur for some other reason.
If your OS is Linux, and your kernel > 2.6.13, you could use the filesystem event notification API named inotify.
There's a Java implementation here : https://bitbucket.org/nbargnesi/inotify-java.
Here's a sample code (heavily inspired from the website).
try {
Inotify i = new Inotify();
InotifyEventListener e = new InotifyEventListener() {
#Override
public void filesystemEventOccurred(InotifyEvent e) {
System.out.println("inotify event occurred!");
}
#Override
public void queueFull(EventQueueFull e) {
System.out.println("inotify event queue: " + e.getSource() +
" is full!");
}
};
i.addInotifyEventListener(e);
i.addWatch(System.getProperty("user.home"), Constants.IN_CLOSE_WRITE);
} catch (UnsatisfiedLinkError e) {
System.err.println("unsatisfied link error");
} catch (UserLimitException e) {
System.err.println("user limit exception");
} catch (SystemLimitException e) {
System.err.println("system limit exception");
} catch (InsufficientKernelMemoryException e) {
System.err.println("insufficient kernel memory exception");
}
This is in Grails and I am using FileUtils Library from the Apache commons fame. The sizeof function returns the size in bytes.
def fileModified = sourceFile.lastModified()
def fileSize = FileUtils.sizeOf(sourceFile)
Thread.sleep(3000) //sleep to calculate size difference if the file is currently getting copied
if((fileSize != FileUtils.sizeOf(sourceFile)) && (fileModified != sourceFile.lastModified())) //the file is still getting copied to return
{
if(log.infoEnabled)
log.info("File is getting copied!")
return
}
Thread.sleep(1000) //breather for picking up file just copied.
Please note that this also depends on what utility or OS you are using to transfer the files.
The safest bet is to copy the file which is been copied or has been copied to different file or directory. The copy process is robust one and it assure you that file is present after the copying process. The one I am using is from commons API.
FileUtils.copyFileToDirectory(File f, Directory D)
If you are copying a huge file which is in process of getting copied beware that this will take time and you might like to start this in parallel thread or best have a seperate application dedicated for transfer process.
Related
The thing is this, I am creating a file, that an XML resource uses, right after creation. When the program is done executing, the file should be deleted. This is what happens:
I run the program, file does not yet exist...
File should be created using FileWriter:
File file = new File("src/main/resources/org/avalin/optaplanner/solver/employeeShiftsScoreRules.drl");
try (FileWriter fileWriter = new FileWriter(file))
{
fileWriter.write("Content...");
fileWriter.flush();
fileWriter.close();
}
catch(IOException e)
{
e.printStackTrace();
}
And then I have this code:
private static synchronized Solver buildSolver()
{
SolverFactory solverFactory =
SolverFactory.createFromXmlResource(SOLVER_CONFIG_XML);
return solverFactory.buildSolver();
}
The filewriting is also wrapped in a synchronized method, but I assume that since they're not directly accessing the same variable, they have no effect what so ever. The file is being read from the SOLVER_CONFIG_XML seen above.
When the program ends, it deletes the file on the path given, so that when it runs next time it will be created accordingly to parameters given to the program.
Now this is what happens...
The first time I run the program I get an error, saying the file isn't written.
Exception in thread "main" java.lang.IllegalArgumentException: The scoreDrl (org/avalin/optaplanner/solver/employeeShiftsScoreRules.drl) does not exist as a classpath resource in the classLoader
I can make prints right after filewriting, that concludes the method HAS run through the first time around, but for some reason, the file is not "created" anyway, before the program ends executing the first time around...
The second time, program runs fine, as the file was created before it gets to the exception?
Is there a way to make sure the file is "completely written" before the next part of my program executes? The file differs in length each time, as it is dynamically created from what the user inputs, so I can't check on that. I would assume it would be completely written as it did execute the prints I made after fileWriter.close() but apparently not so.
It looks like you are writing your file to the src/main/resources folder, which is a standard location for resource sources - that is, where your build system reads files from, and not where your running program does.
Although it's possible to add your source folders to the classpath of your running problem, it is bad practice - try to find out where your build system writes its output to (probably separate folders for class files and copied resources) and write your file there.
I use ICEpdf library for PDF displaying at my desktop java application. Application adds annotations to PDF at runtime, but without changing original files — changes are displayed only during one 'session'. I recently discovered that application creates a lot of temporary files which consume quite a lot of disk space.
Method org.icepdf.core.pobjects.Document.setInputStream has the following code:
// Delete temp file on exit
tempFile.deleteOnExit();
So I suppose it has to remove temporary files after it used them, but it does not:
How can I programmatically remove all files created by application on exit or make standard file removing work?
To get temp folder path:
FileSystems.getDefault().getPath(System.getProperty("java.io.tmpdir"))
To remove files:
try (DirectoryStream<Path> paths = Files.newDirectoryStream(pathToDir, regex)){
paths.forEach(path -> path.toFile().delete());
} catch (IOException e) {
// handle io exception
}
where regex is filename pattern. In your case: "IcePdf*"
I need to write a custom batch File renamer. I've got the bulk of it done except I can't figure out how to check if a file is already open. I'm just using the java.io.File package and there is a canWrite() method but that doesn't seem to test if the file is in use by another program. Any ideas on how I can make this work?
Using the Apache Commons IO library...
boolean isFileUnlocked = false;
try {
org.apache.commons.io.FileUtils.touch(yourFile);
isFileUnlocked = true;
} catch (IOException e) {
isFileUnlocked = false;
}
if(isFileUnlocked){
// Do stuff you need to do with a file that is NOT locked.
} else {
// Do stuff you need to do with a file that IS locked
}
(The Q&A is about how to deal with Windows "open file" locks ... not how implement this kind of locking portably.)
This whole issue is fraught with portability issues and race conditions:
You could try to use FileLock, but it is not necessarily supported for your OS and/or filesystem.
It appears that on Windows you may be unable to use FileLock if another application has opened the file in a particular way.
Even if you did manage to use FileLock or something else, you've still got the problem that something may come in and open the file between you testing the file and doing the rename.
A simpler though non-portable solution is to just try the rename (or whatever it is you are trying to do) and diagnose the return value and / or any Java exceptions that arise due to opened files.
Notes:
If you use the Files API instead of the File API you will get more information in the event of a failure.
On systems (e.g. Linux) where you are allowed to rename a locked or open file, you won't get any failure result or exceptions. The operation will just succeed. However, on such systems you generally don't need to worry if a file is already open, since the OS doesn't lock files on open.
// TO CHECK WHETHER A FILE IS OPENED
// OR NOT (not for .txt files)
// the file we want to check
String fileName = "C:\\Text.xlsx";
File file = new File(fileName);
// try to rename the file with the same name
File sameFileName = new File(fileName);
if(file.renameTo(sameFileName)){
// if the file is renamed
System.out.println("file is closed");
}else{
// if the file didnt accept the renaming operation
System.out.println("file is opened");
}
On Windows I found the answer https://stackoverflow.com/a/13706972/3014879 using
fileIsLocked = !file.renameTo(file)
most useful, as it avoids false positives when processing write protected (or readonly) files.
org.apache.commons.io.FileUtils.touch(yourFile) doesn't check if your file is open or not. Instead, it changes the timestamp of the file to the current time.
I used IOException and it works just fine:
try
{
String filePath = "C:\sheet.xlsx";
FileWriter fw = new FileWriter(filePath );
}
catch (IOException e)
{
System.out.println("File is open");
}
I don't think you'll ever get a definitive solution for this, the operating system isn't necessarily going to tell you if the file is open or not.
You might get some mileage out of java.nio.channels.FileLock, although the javadoc is loaded with caveats.
Hi I really hope this helps.
I tried all the options before and none really work on Windows. The only think that helped me accomplish this was trying to move the file. Event to the same place under an ATOMIC_MOVE. If the file is being written by another program or Java thread, this definitely will produce an Exception.
try{
Files.move(Paths.get(currentFile.getPath()),
Paths.get(currentFile.getPath()), StandardCopyOption.ATOMIC_MOVE);
// DO YOUR STUFF HERE SINCE IT IS NOT BEING WRITTEN BY ANOTHER PROGRAM
} catch (Exception e){
// DO NOT WRITE THEN SINCE THE FILE IS BEING WRITTEN BY ANOTHER PROGRAM
}
If file is in use FileOutputStream fileOutputStream = new FileOutputStream(file); returns java.io.FileNotFoundException with 'The process cannot access the file because it is being used by another process' in the exception message.
Hi i have made a small program that reads a config file. This file is stored outside the actual jar file. On the same level as the jarfile actually.
When i start my program from a commandline in the actual directory (ie. D:\test\java -jar name.jar argument0 argument1) in runs perfectly.
But when i try to run the program from another location then the actual directory i get the filenotfound exception (ie. D:\java -jar D:\test\name.jar argument0 argument1).
The basic functionality does seem to work, what am i doing wrong?
As requested a part of the code:
public LoadConfig() {
Properties properties = new Properties();
try {
// load the properties file
properties.load(new FileInputStream("ibantools.config.properties"));
} catch (IOException ex) {
ex.printStackTrace();
} // end catch
// get the actual values, if the file can't be read it will use the default values.
this.environment = properties.getProperty("application.environment","tst");
this.cbc = properties.getProperty("check.bankcode","true");
this.bankcodefile = properties.getProperty("check.bankcodefile","bankcodes.txt");
} // end loadconfig
The folder looks like this:
This works:
This doesn't:
The jar doesn't contain the text file.
When reading a File using the String/path constructors of File, FileInpustream, etc.. a relative path is derived from the working directory - the directory where you started your program.
When reading a file from a Jar, the file being external to the jar, you have at least two options :
Provide an absolute path: D:/blah/foo/bar
Make the directory where your file is located part of the class path and use this.getClass().getClassLoader().getResourceAsStream("myfile")
The latter is probably more appropriate for reading configuration files stored in a path relative to the location of your application.
There could be one more possibility:
If one part of your code is writing the file and another one is reading, then it is good to consider that the reader is reading before the writer finishes writing the file.
You can cross check this case by putting your code on debug mode. If it works fine there and gives you FileNotFoundException, then surely this could be the potential reason of this exception.
Now, how to resolve:
You can use retry mechanism something similar to below code block
if(!file..exists()){
Thread.sleep(200);
}
in your code and change the sleep value according to your needs.
Hope that helps.!!
In Java, I'm working with code running under WinXP that creates a file like this:
public synchronized void store(Properties props, byte[] data) {
try {
File file = filenameBasedOnProperties(props);
if ( file.exists() ) {
return;
}
File temp = File.createTempFile("tempfile", null);
FileOutputStream out = new FileOutputStream(temp);
out.write(data);
out.flush();
out.close();
file.getParentFile().mkdirs();
temp.renameTo(file);
}
catch (IOException ex) {
// Complain and whine and stuff
}
}
Sometimes, when a file is created this way, it's just about totally inaccessible from outside the code (though the code responsible for opening and reading the file has no problem), even when the application isn't running. When accessed via Windows Explorer, I can't move, rename, delete, or even open the file. Under Cygwin, I get the following when I ls -l the directory:
ls: cannot access [big-honkin-filename]
total 0
?????????? ? ? ? ? ? [big-honkin-filename]
As implied, the filenames are big, but under the 260-character max for XP (though they are slightly over 200 characters).
To further add to the sense that my computer just wants me to feel stupid, sometimes the files created by this code are perfectly normal. The only pattern I've spotted is that once one file in the directory "locks", the rest are screwed.
Anybody ever run into something like this before, or have any insights into what's going on here?
Make sure you always close the stream in a finally block. In your case if an exception is thrown the stream might not get closed and will leak a file handle. You could use procexp from SysInternals to see which process holds the handle to the file.
Although, by definition, NTFS should handle path length up to 2^15-1, in practice the length of paths is limited to 255.
You can create files with a longer path name (filename including parent folder names), but you cannot access them afterwards. The error I get in these cases is that the file could not be found. To get rid of these files, I have to shorten the names of parent folders, until the path length is short enough.