I am using Ubuntu (in case it will make a difference) and I am trying use Camel to send files to processor from one folder. But the problem is that when I am saving this file in the folder (takes about 5-10 seconds) Camel picks it up straight away.
To simulate the process I am using gedit with txt file with ~500k rows so it will take some time to save.
I have tried adding options:
from("file:src/Data/new/?readLock=changed&readLockMinAge=3m")
I have tried using
.filter(header("CamelFileLastModified").isGreaterThan(new Date(System.currentTimeMillis()-120000))) to give 2 minute delay.
Nothing seems to influence its behaviour, it picks it up straight away, throws an exception because of some checks while processing file and moves it to the Error folder.
I know there is an issue with FTP file transfers which I will have to face later on, but I can not even get it working on local file system.
Any help will be appreciated!
SOLVED
from("file:src/Data/new/?readLock=changed&readLockMinAge=3m")
Parameters actually work as they should. I was using Jetty to run the project and I should have done whole project clean/install after any amendments.
I had to amend parameters a bit to:
from("file:src/Data/new/?readLock=changed&readLockTimeout=65000&readLockMinAge=1m")
because it was complaining that readLockTimeout should be more than readLockCheckInterval + readLockMinAge.
Have a look into the documentation:
Avoid reading files currently being written by another application
Beware the JDK File IO API is a bit limited in detecting whether
another application is currently writing/copying a file. And the
implementation can be different depending on OS platform as well. This
could lead to that Camel thinks the file is not locked by another
process and start consuming it. Therefore you have to do you own
investigation what suites your environment. To help with this Camel
provides different readLock options and doneFileName option that you
can use. See also the section Consuming files from folders where
others drop files directly.
So I think the doneFileName option will solve your problem.
Related
Using Java, I want to move a file on the file system despite if it's being used by another process? I've found this for deleting FileDeleteStrategy.FORCE.delete() is there an equivalent one for moving the file into another directory?
As others already pointed out, on Windows there is no such thing as move file while it's being used by other process. There is however standard way of handling such situation, but it will require OS restart.
From documentation of MoveFileEx:
MOVEFILE_DELAY_UNTIL_REBOOT 4 (0x4)
The system does not move the file until the operating system is restarted. The system moves the file immediately after AUTOCHK is executed, but before creating any paging files.
To use this function from java application, you could use JNA, but your application needs to be running with admin privileges with appropriate UAC.
From UX point of view, Jared Smith's answer make much more sense.
We want to open a file in Java and read its contents.
This file may be updated by an external application using Safe Save. That means the file will be externally read and its updated contents will be stored to a new file. Eventually the original file is deleted and the new file is renamed to match the original file's name.
Unfortunately the external process fails during rename (last part of the Safe Save) when our Java Application is reading the original file at the same time.
We played with different kind of open modes but could not get a solution that does not fail the external reader.
Is there some way to open a file that does not interfere with external processes accessing the same file? Ideally, whenever an external process moves or deletes the file we would like to get an exception in our Java application. And only there.
Do you have any ideas on how to achieve that?
EDIT:
Just some clarification regarding the use case:
This an indexer like scenario. We want to index contents of a potentially very large filesystem where 3rd party independent processes can concurrently read from or write to as well. We have no control over the 3rd party processes.
Copying the original file seems like a big overhead and we are not sure if that helps with the original problem as it will probably fail the external reader on a Safe Save as well.
Last but not least: This should work on Windows and Linux. But we are experiencing this problems on Windows.
On Windows, whether a file can be renamed or deleted while it's open is controlled by the FILE_SHARE_DELETE sharing mode flag. This flag should be passed in when the file is opened with the low level CreateFile function.
Unfortunately, Java API does not give you control over low level Windows-specific flags. There is an open bug report to have FILE_SHARE_DELETE added by default, but it's unlikely it will be done because of backwards compatibility (some applications may depend on this behavior). the A comment in the report suggests a workaround: instead of new FileInputStream(file) use the java.nio API.
InputStream in = Files.newInputStream(file.toPath());
I don't have access to Windows right now to verify that this workaround uses the right sharing mode.
Make a copy of the original file an use this within your Java program, and at the same time keep track of the original file.
Here, this might help you out:
The java.nio.file package provides a file change notification API, called the Watch Service API. This API enables you to register a directory (or directories) with the watch service. When registering, you tell the service which types of events you are interested in: file creation, file deletion, or file modification. When the service detects an event of interest, it is forwarded to the registered process. The registered process has a thread (or a pool of threads) dedicated to watching for any events it has registered for. When an event comes in, it is handled as needed. Official docs
You cannot achieve this only with files, at least not without making additional assumptions. If the processes are not synchronized you will get either (a) errors (b) corrupted data or (c) both. Furthermore, such system will be unstable, prone to race conditions and implementation-specific details. This means that even if it looks like it's working it will not work correctly always and in each case.
Depending on your circumstances you might try to use a combination of scehduling (i.e. process A runs every even minute, process B every odd minute), exclusive/shared open flags, range locks, copying files, file change notifiers, retrying on failure etc. If you can somehow ensure that your assumptions are never broken you might end up with something which is "good enough". But all in all, this is a bad engineering practice and should be avoided.
For a proper solution, you need to make both processes aware that they are talking to each other. What you have is really a textbook use case for a database. Besides using a database there are plenty of other ways to synchronize access to data - messaging, streams, locks, shared memory etc. Each way has its own benefits and downsides and without knowing more about your specific situation it is impossible to say which would be better.
I have a jar. I want the jar to be able to make a copy of itself while running. I understand windows may have problems with this. How would I do this, or am I over thinking it?
Edit: To explain a bit more....
I'm writing a repackagable firmware deployment system... http://code.google.com/p/heimdall-one-click/ The idea is that a ROM developer from XDA can make his own, then pack it up in a cross-platform deployable one-click packaging nearly as easily as it is to deploy the firmware.
My program takes alot of the work out by automating the tasks... I'm trying to automate packaging of the one-click deployable packaging system.... give the developers a form to fill out which will change the header information, then they select their firmware files to be deployed. I'm trying to keep it all in one jar.
As josh says, it would be nice if you tell why do you want to do this in order to help.
Answering only what have you post, copying the jar is just copying another file. There is the issue that it might be blocked by the OS (Windows); another issue is how do you locate it in the machine and if the user running the process has the permissions needed.
Once those two issues are solved, it is just a copy operation, the OS could not care less that if the order to copy comes from the process run from the file or from another one.
EDIT to asnwer changes in the first post.
As I told before, in the end copying a file is a OS issue. If you want to copy the current jar in Windows, then the jar must not be locked by other process so it becomes an OS question rather than a Java one.
Possible workarounds:
The faster (but dirtiest) is to launch a .bat that does a sleep of a few seconds and then does the copy. Immediately after launching it, your close your java app. If you need to continue doing things in Java, after copying the file, the .bat launches the java app again (with the appropiate parameters).
A variant of the previous is slightly sleazier... launch your java app from a .bat, and the first thing that .bat does is copying your jar to the PC temp directory. Be sure to document it well so your users do not get scared!
JNI library to unlock a file. There are several programs that (Unlocker) that try to unlock files; do not know to which point it is effective or how will it affect the JVM.
I believe you can use:
File file = new File(MyClass.class.getProtectionDomain().getCodeSource().getLocation().getPath());
to get a reference to the path to the .JAR file.
Then you just make a copy of it:
http://download.oracle.com/javase/tutorial/essential/io/copy.html
Your operating system might not allow this, but I think it should.
I have a Java app that runs on a Windows mobile device. At startup the app talks to our server to see if any files need updating and downloads them if they do. If any of the files are dlls they need to be stored in a temp directory and the device is rebooted because they might be currently in use. When the app starts it reads an xml file that lists all of the temp files and where they need to go and copies them into place.
A new requirement has come up that involves also updating the JVM files as part of this process. Since the code that does the copying is run on the JVM there is no way to do it since the files will always be in use. So we are looking at writing something in native code to do this copying process.
Before we start, I was just wondering if anyone knew of an already existing application or technique that does this (someone suggested a registry entry that tells the device to copy files on startup for example). Basically the requirement is to read some sort of configuration file that details the location of the source file and the destination then performs the copy. Any ideas before I reinvent the wheel by writing an app myself?
If your target handsets are handheld barcode scanners (Symbol, Intermec, etc.) they already have a framework in place for this. I don't have all the details, but I know from previous projects that they have a "protected" memory location that allows application to essentially re-configure / copy themselves from hard boots and similar problems. It might be worth seeing if any of that would work on your existing targets.
The scanners use either Windows CE or Windows Mobile.
In the absence of another answer, I have written a simple app to do it and put it in the startup directory. Was pretty easy, just didn't want to reinvent the wheel.
You can also rename your running executable file by the running-application itself. After this you can copy the file into the directory and simply restart your application.
I'm looking for a non-webstart/jnlp solution.
I'd like to add to my app an update feature that checks in an ftp or http server and downloads the last version (if there is a newer one) replacing the libs that has been changed.
How can i do that? I want to implement something like JDownloaders updates.
Thanks
It looks like you just described exactly how to do it. Add an update feature that checks an FTP or HTTP server and downloads the latest version.
Remember that you cannot download and overwrite a file which is in use. So you have two options for a design from where I sit:
When you start up the application, copy all of the jar and library files to a /temp folder of some sort before running them. Then, when you download the update, overwrite the files in the original place. The next time the application starts up, it will use the new files.
When you start up the application, first startup an updater. Have it connect to the server and compare all of the file versions. It will be able to overwrite any of the application files because it doesn't use them. It only uses the updater jar. Once this is done and everything has been updated, then start a new process from the updater with the actual application. You will also need to put some code in to be able to update the updater jar. Either make the main application be able to update the updater, or use the first technique and run the updater from a copy of the updater jar.
Remember when you download the files that you should be downloading them to a temporary location and then moving them to the right place when they're done. This will make sure that you never leave your application in a "half-downloaded" state.
Beyond that, getting this to work is going to be about a lot of testing. Good luck!
Have a look at http://code.google.com/p/getdown/
According to this question on stackoverflow it seem to be a viable alternative for web start (at least worth having a look at).