I have been playing with a program that copies a set of files into a directory, manipulates said files, and outputs them to another directory. On 99% of the environments it has been tested / deployed on it works fine, however on one particular machine I will occasionally (roughly once every 600 files) receive FileNotFoundException: (Access denied) exceptions, which are not repeatable using the same testing conditions and inputs. After thoroughly checking all my code (and the open source lib it uses) for any unclosed resources, I ran the program alongside Process Monitor to see what other processes were accessing the files: javaw and Explorer were to be expected, but there was also a windows audio service randomly querying and locking files (wtf).
Ultimately, it would make sense to retry when catching this exception, but there is no guarantee it wont still be locked. Would it make sense to deny access rights to the SYSTEM user? Is there a file-locking mechanism I should be utilizing? Has anyone else dealt with a file access issue in the past?
Additional Info: I was able to run the program with no issue on my machine after removing SYSTEM privileges on the directories where files are stored, but I have never had an issue on this PC. I also implemented a retry after a wait whenever the exception is caught (again, never fired because this comp has never had the issue). Will be redeploying this week to see how it works: (are there still file access exceptions after changing privileges / allowing a small wait on failure)...
Final Follow-Up: On a whim I reinstated this Stackoverflow account and found this question. If I recall, the final resolution was to:
+ Add any access denied files to a list
+ Process the list at the end of processing all other files
Certainly not fool proof, but this was also running on XP, so hopefully is defunct issue now.
Related
My application needs to periodically scan filesystems in order to process files. Initially I was using java.nio.file.Files.walk to perform the scan, but soon enough I encountered an issue with some AccessDeniedException. Found out after some googling that Files.walk expects the explored directory tree to be accessible by the user, else it would crash and stop, which renders this function unusable for my application (it is self-hosted by a multitude of people on various kind of systems).
I changed the implementation to use java.nio.file.Files.walkFileTree instead, which seemed to work great and handled the AccessDeniedException in the user code.
However, someone recently reported that scanning time skyrocketed from a mere 12 seconds (using Files.walk) to 80 minutes (using Files.walkFileTree)! The user has around 10,000 folders and 120,000 files. It is running Windows, and the disk is using NTFS. Other users with similar number of folders/files but running Linux experience scan times under 10s, whatever the method used.
I am trying to understand what could cause the massive performance hit by using Files.walkFileTree on Windows with NTFS, but given I don't have access to a test system running Windows, I can't debug the code to understand where the time is spent.
Would you know if there are know issues around walking a file tree under Windows NTFS? And if there are some other methods I could use to perform that task? Keeping in mind that I need to handle AccessDeniedException in user code.
I am working on an application that polls a directory for new input files at a defined interval. The general process is:
Input files FTP'd to landing strip directory by another app
Our app wakes up
List files in the input directory
Atomic-move the files to a separate staging directory
Kick off worker threads (via a work-distributing queue) to consume the files from the staging directory
Go to back sleep
I've uncovered a problem where the app will pick up an input file while it is incomplete and still in the middle of being transferred, resulting in a worker thread error, requiring manual intervention. This is a scenario we need to avoid.
I should note the file transfer will complete successfully and the server will get a complete copy, but this will happen to occur after the app has given up due to an error.
I'd like to solve this in a clean way, and while I have some ideas for solutions, they all have problems I don't like.
Here's what I've considered:
Force the other apps (some of which are external to our company) to initially transfer the input files to a holding directory, then atomic-move them into the input directory once they're transferred. This is the most robust idea I've had, but I don't like this because I don't trust that it will always be implemented correctly.
Retry a finite number of times on error. I don't like this because it's a partial solution, it makes assumptions about transfer time and file size that could be violated. It would also blur the lines between a genuinely bad file and one that's just been incompletely transferred.
Watch the file sizes and only pick up the file if its size hasn't changed for a defined period of time. I don't like this because it's too complex in our environment: the poller is a non-concurrent clustered Quartz job, so I can't just persist this info in memory because the job can bounce between servers. I could store it in the jobdetail, but this solution just feels too complicated.
I can't be the first have encountered this problem, so I'm sure I'll get better ideas here.
I had that situation once, we got the other guys to load the files with a different extension, e.g. *.tmp, then after the file copy is completed they rename the file with the extension that my code is polling for. Not sure if that is as easily done when the files are coming in by FTP tho.
I unwittingly ran a Java program on Linux where the partition that it, it's properties and logs resided on was close to 100% full. It failed, but after clearing up the problem with the disk space, I ran it again and it failed a second time because it's properties file was 0 bytes long.
I don't have source of the file and I don't want to go as far as decompiling the class files, but I was wondering whether the corruption of the properites could be because the program failed to write to the properties file.
The mystery is that, I expected that the properties would be read-only and don't recall any items being updated by the program. Could it be that even if the properties are only read, the file is opened in read-write mode and could disappear if the partition is full?
N.b. this program has run without failure or incident at least 1000 times over several years.
I don't have source of the file and I don't want to go as far as decompiling the class files, but I was wondering whether the corruption of the properites could be because the program failed to write to the properties file.
That is the most likely explanation. There would have been an exception, but the application could have squashed it ... or maybe you didn't notice the error message. (Indeed, if the application tried to log the error to a file, that would most likely fail too.)
Could it be that even if the properties are only read, the file is opened in read-write mode and could disappear if the partition is full?
That is unlikely to be the answer in this case. Unlike many languages, in Java the code for reading a file and writing a file involves different stream classes. It is hard to imagine how / why the application's developer would open a property file for writing (with truncation) if there was never any intention to write it.
The most plausible explanation is that the application does update the properties file. (Try installing the program again, using it, stopping it, and looking at the property file's modification timestamp.)
N.b. this program has run without failure or incident at least 1000 times over several years.
And I bet this is the first time you've run it while the disk was full :-)
I want to make a java program that must run for a specific number of times only. e.g 2,5,10 etc. after that it must throw an Exception.
It is not allowed to use any FILE or Database for this. Someone gave me a hint of REGISTRY! But i don't know how to use it for this.
Please help me is this regard...
You can use java preferences (registry on windows) :
http://download.oracle.com/javase/6/docs/api/java/util/prefs/Preferences.html
You can find some sample usage here:
http://www.particle.kth.se/~lindsey/JavaCourse/Book/Part1/Java/Chapter10/Preferences.html
Whether this problem is solvable depends on what is meant by "any FILE or Database".
Depending on your point of view, the Windows Registry is a kind of file / database. Certainly, the only reason that values stay in the registry over a reboot is because registry changes are written to disc.
You can move state (such as a count of the number of times an application has been run) to some other service on the local machine. But once again, unless the service saves that state to disc (or some other stable storage medium) it will be lost is the local machine reboots.
You can move state to a service on a remote machine, but once again it may be lost if not saved to disc, etc. Moreover, you may not be able contact that remote service at the time you need the state; e.g. when starting the application.
You can copy the state to lots of remote services without discs, but a network failure (or the user unplugging from the network) will stop you accessing the state.
So to summarize, if you cannot write to disc (or nvram, tape, etc) locally, you cannot guarantee that the counter won't get reset, and that it will be available when needed. Therefore you cannot guarantee that the application won't be run more times than is allowed.
I imagine that you are trying to come up with some limited use scheme that users cannot subvert; e.g. by deleting stuff from the file / database / whatever that counter. Unfortunately, unless you physically control BOTH the hardware AND the operating system on which the application runs, you cannot prevent someone from subverting any counter stored on the machine itself. Anyone with "root" or full administrator rights, or with physical access, can ultimately change any data on the machine itself.
The best you can do is establish a secure connection to a remote server and use that to hold the usage counter. But even that is futile, because a motivated person can reverse engineer the critical part of your application and disable the code that checks the counter.
If the app. has a GUI, it can be launched using Java Web Start and use the PersistenceService. Here is a small demo. of the PersistenceService. The code is available for download.
Edit:
And the PersistenceService should work on any machine that has a JRE, as opposed to just Windows.
Even though this sounds like an attempt at copy protection, you may want to consider self-modifying code. There is an interesting discussion on this subject in Java here: Self modifying code in Java
I have a server program running a java binary code (xx.jar file). While it is running I erroneously delete the xx.jar file. The program continues to run. But I am not sure if the results will be correct, and I am not sure if the program will fail.
When I delete the xx.jar file, the program was in a method for a long time and still it is in that method call. When it calls another method call will, my program fail?
I am asking this question because If deleting the file has no harm I will be gaining about 3-4h on a server machine
There is no guarantee that the JVM will load all classes from a .jar file into memory, although it may pre-load some or all of the .jar as an optimization.
If this fails, and I imagine it would at some point, it would not happen during the middle of execution of a method. It would be at a point where a new class must be loaded from the classpath and the JVM can't access that file anymore. Then you would fail with NoClassDefFoundError or worse.
So, no, I would definitely not advise you to do this, even if it happens to work in some cases.
Depending on your operating system, this will or will not be a problem. On Linux, for example, a file isn't really deleted until all applications that have it opened close it. The file will be gone from the directory listing but it still exists and can be read (and even written!) by any application with a valid file descriptor open.
Whether or not the JVM keeps file descriptors open to all jar files of your application, I don't know. I wouldn't rely on it doing so, even if it does seem to work ok sometimes.
You will come to know when you completely deploy and redeploy an application and restart it .
Dependency functionality will fail and Expection will be thrown