I implemented JNotify to determine when a new file arrives in a particular directory, and, when a file arrives, to send the filename over to another function, as follows:
public class FileDetector {
MessageProcessor mp;
class Listener implements JNotifyListener {
public void fileCreated(int wd, String rootPath, String name) {
print("created " + rootPath + " : " + name);
mp.processMessage(rootPath + "\\" + name);
}
}
}
The function mp.processMessage tries to open the file, but I keep getting an error that the file is in use by another process. However, as the file has just been created, the only other process which might be using it is JNotify.
I put a couple of print statements, and it appears that the function mp.processMessage is being called before the listener's print function. Does anyone have a suggestion for how I might resolve this, beyond putting the entire message processing inside the listener class?
#Eile What I think is As soon as one process is copying the file, you are trying to read it, 100 ms delay will complete the copy first n then you can read the file easily.
Here's what I've done so far - I added into mp.processMessage() a 100 millisecond delay before trying to open the file, and have had no issues with it. However, I am still puzzled as to why that would be necessary, and whether or not there is a better solution to this issue.
I have tried this and have found that an arbitrary delay didn't work well for me. What I did was create a DelayQueue. I added each observed new file to the queue with a 100ms delay. When the delay expired I checked if the file was readable/writable. If is was, I popped it from the queue. If not, I readded it to the queue with another 100ms delay. To check if it was readable/writable I attempt to open a FileInputStream to the file. If no exception, I close the stream and pop the file.
I am hoping that nio.2 (jsr 203) does not have this same issue. If you can use Java 7 you might want to give it a try.
Related
I'm trying to write to a file located in my $HOME directory. The code to write to that file has been packaged into a jar file. When I run the unit tests to package the jar file, everything works as expected - namely the file is populated and can be read from again.
When I try to run this code from another application where the jar file is contained the lib directory it fails. The file is created - but the file is never written to. When the app goes to read the file it fails parsing it because it is empty.
Here is the code that writes to the file:
logger.warn("TestNet wallet does not exist creating one now in the directory: " + walletPath)
testNetFileName.createNewFile()
logger.warn("Wallet file name: " + testNetFileName.getAbsolutePath)
logger.warn("Can write: "+ testNetFileName.canWrite())
logger.warn("Can read: " + testNetFileName.canRead)
val w = Wallet.fromWatchingKey(TestNet3Params.get(), testNetSeed)
w.autosaveToFile(testNetFileName, savingInterval, TimeUnit.MILLISECONDS, null)
w
}
here is the log form the above method that is relevant:
2015-12-30 15:11:46,416 - [WARN] - from class com.suredbits.core.wallet.ColdStorageWallet$ in play-akka.actor.default-dispatcher-9
TestNet wallet exists, reading in the one from disk
2015-12-30 15:11:46,416 - [WARN] - from class com.suredbits.core.wallet.ColdStorageWallet$ in play-akka.actor.default-dispatcher-9
Wallet file name: /home/chris/testnet-cold-storage.wallet
then it bombs.
Here is the definition for autoSaveToFile
public WalletFiles autosaveToFile(File f, long delayTime, TimeUnit timeUnit,
#Nullable WalletFiles.Listener eventListener) {
lock.lock();
try {
checkState(vFileManager == null, "Already auto saving this wallet.");
WalletFiles manager = new WalletFiles(this, f, delayTime, timeUnit);
if (eventListener != null)
manager.setListener(eventListener);
vFileManager = manager;
return manager;
} finally {
lock.unlock();
}
}
and the definition for WalletFiles
https://github.com/bitcoinj/bitcoinj/blob/master/core/src/main/java/org/bitcoinj/wallet/WalletFiles.java#L68
public WalletFiles(final Wallet wallet, File file, long delay, TimeUnit delayTimeUnit) {
// An executor that starts up threads when needed and shuts them down later.
this.executor = new ScheduledThreadPoolExecutor(1, new ContextPropagatingThreadFactory("Wallet autosave thread", Thread.MIN_PRIORITY));
this.executor.setKeepAliveTime(5, TimeUnit.SECONDS);
this.executor.allowCoreThreadTimeOut(true);
this.executor.setExecuteExistingDelayedTasksAfterShutdownPolicy(false);
this.wallet = checkNotNull(wallet);
// File must only be accessed from the auto-save executor from now on, to avoid simultaneous access.
this.file = checkNotNull(file);
this.savePending = new AtomicBoolean();
this.delay = delay;
this.delayTimeUnit = checkNotNull(delayTimeUnit);
this.saver = new Callable<Void>() {
#Override public Void call() throws Exception {
// Runs in an auto save thread.
if (!savePending.getAndSet(false)) {
// Some other scheduled request already beat us to it.
return null;
}
log.info("Background saving wallet, last seen block is {}/{}", wallet.getLastBlockSeenHeight(), wallet.getLastBlockSeenHash());
saveNowInternal();
return null;
}
};
}
I'm guessing it is some sort of permissions issue but I cannot seem to figure this out.
EDIT: This is all being run on the exact same Ubuntu 14.04 machine - no added complexity of different operating systems.
You cannot generally depend on the existence or writability of $HOME. There are really only two portable ways to identify (i.e. provide a path to) an external file.
Provide an explicit path using a property set on the invocation command line or provided in the environment, or
Provide the path in a configuration properties file whose location is itself provided as a property on the command line or in the environment.
The problem with using $HOME is that you cannot know what userID the application is running under. The user may or may not even have a home directory, and even if the user does, the directory may or may not be writable. In your specific case, your process may have the ability to create a file (write access on the directory itself) but write access to a file may be restricted by the umask and/or ACLs (on Windows) or selinux (on Linux).
Put another way, the installer/user of the library must explicitly provide a known writable path for your application to use.
Yet another way to think about it is that you are writing library code that may be used in completely unknown environments. You cannot assume ANYTHING about the external environment except what is in the explicit contract between you and the user. You can declare in your interface specification that $HOME must be writable, but that may be highly inconvenient for some users whose environment doesn't have $HOME writable.
A much better and portable solution is to say
specify -Dcom.xyz.workdir=[path] on the command line to indicate the work path to be used
or
The xyz library will look for its work directory in the path specified by the XYZ_WORK environment variable
Ideally, you do BOTH of these to give the user some flexibility.
savePending is always false. In the beginning of call you check that it is false, and return null. The actual save code is never executed. I am guessing you meant to check if it was true there, and also set it to true, not false. You then also need to reset it back to false in the end.
Now, why this works in your unit test is a different story. The test must be executing different code.
I have a jar that is called about every minute from another script. In this jar I have created a JavaLogger that logs what happens while the Jar runs. The log file that JavaLogger writes to is called myLog.0. I have the following code to allow it to go to .1/.2/.3/.4.
try {
FileHandler fileHandler = new FileHandler(filePath, 5242880, 5, true);
fileHandler.setFormatter(new java.util.logging.Formatter() {
#Override
public String format(LogRecord logRecord) {
return "[" + logRecord.getLevel() + " " + createDateTimeLog() + "] " + logRecord.getMessage() + "\r\n";
}
});
logger.addHandler(fileHandler);
} catch (IOException e) {}
So I expect the logs to grow. However, every once and a while the log will print to myLog.0.1. I would guess that this is because the file is locked. However, this never happens mid-run of my jar. It logs to .0.1 the entire time the jar runs. Could the file still be locked from my previous run?
If so I have even tried to close the handler before the Jar exits. There is only one exit point from the jar and I put the following code right before it:
MyLogger.logger.getHandlers()[0].close();
I have run this through the debugger and there is only ever one handler (the FileHandler that I add).
As I said, this only happens randomly. The first 3 runs of the jar could be to .0 and then the fourth to .0.1. Then the next 10 could be correct again. It's hard to say. However it does happen fairly often (I would say it writes to .0.1 about every 1/8 of the time).
Any ideas / suggestions would be great. Thanks ahead of time.
Could the file still be locked from my previous run?
Could be that two JVMs are running your jar at the same time. Add code to grab the RuntimeMXBean and then add a single log statement to record the runtime name and the start time. The runtime name usually maps to a process id and a host name.
The FileHandler does everything it can to prevent two concurrently running JVMs from writing to the same log file. If this behavior was allowed the log file would be almost impossible to read and understand.
If you really want to write everything to one log file then you have to do one of the following:
Prevent concurrent JVM processes from starting by changing how it is launched.
Have your code detect if another JVM is running your code and exit before creating a FileHandler.
Have each JVM write to a distinct log file and create code to safely merge the files into one.
Create a proxy Handler that creates and closes a FileHandler for each log record. The proxy handler would use a predefined file name (different from the log file) and a FileLock to serialize access to the log file from different JVMs.
Use a dedicated process to write to the log file and have all the JVMs send log messages to that process.
Don't know if it is clear from title, I'll explain it deeper.
First of all limitations: Java 1.5 IBM.
This is the situation:
I have spring web service that receives request with pdf document in it. I need to put this pdf into the some input directory that AFP application (not of the importance) monitors. This AFP application takes that pdf, do something with it and returns it to some output directory that I need to monitor. Monitoring of output directory would take some time, probably 30 seconds. Also, I know what is exact file name that I expect to appear in output directory. If nothing appears in 30 seconds than I would return some fault response.
Because of my poor knowledge of web services and multithreading I don't know in which possible problems I can fall into.
Also, searching the internet I realize that most of people recommend watchservice for directory monitoring, but this is introduced in Java 7.
Any suggestion, link, idea would be helpful.
So, the scenario is simple. In a main method, the following actions are done in order:
call the AFP service;
poll the directory for the output file;
deal with the output file.
We suppose here that outputFile is a File containing the absolute path to the generated file; this method returns void, adapt:
// We poll every second, so...
private static final int SAMPLES = 30;
public void dealWithAFP(whatever, arguments, are, there)
throws WhateverIsNecessary
{
callAfpService(here);
int i = 0;
try {
while (i < SAMPLES) {
TimeUnit.SECONDS.sleep(1);
if (outputFile.exists())
break;
}
throw new WhateverIsNecessary();
} catch (InterruptedException e) {
// Throw it back if the method does, otherwise the minimum is to:
Thread.currentThread().interrupt();
throw new WhateverIsNecessary();
}
dealWithOutputFile(outputFile);
}
I have multiple EBS-backed EC2 instances running and I want to be able to take a snapshot of the EBS volume behind one of them, create a new EBS volume from that snapshot, and then mount that new EBS volume onto another as an additional drive. I know how to do this via the AWS web console, but I would like to automate the process by using the AWS Java API.
If I simply call the following commands one after another:
CreateSnapshotResult snapRes
= ec2.createSnapshot(new CreateSnapshotRequest(oldVolumeID, "Test snapshot"));
Snapshot snap = snapRes.getSnapshot();
CreateVolumeResult volRes
= ec2.createVolume(new CreateVolumeRequest(snap.getSnapshotId(), aZone));
String newVolumeID = volRes.getVolume().getVolumeId();
AttachVolumeResult attachRes
= ec2.attachVolume(new AttachVolumeRequest(newVolumeID, instanceID, "xvdg"));
I get the following error:
Caught Exception: Snapshot 'snap-8e822cfd' is not 'completed'.
Reponse Status Code: 400
Error Code: IncorrectState
Request ID: 40bc6bad-43e0-49e6-a89a-0489744d24e6
To get around this, I obviously need to wait until the snapshot is completed before I create the new EBS volume from the snapshot. According to the Amazon docs, the possible values of Snapshot.getState() are "pending, completed, or error," so I decided to check in with AWS to see if the snapshot is still in the "pending" state. I wrote the following code, but it has not worked:
CreateSnapshotResult snapRes
= ec2.createSnapshot(new CreateSnapshotRequest(oldVolumeID, "Test snapshot"));
Snapshot snap = snapRes.getSnapshot();
System.out.println("Snapshot request sent.");
System.out.println("Waiting for snapshot to be created");
String snapState = snap.getState();
System.out.println("snapState is " + snapState);
// Wait for the snapshot to be created
while (snapState.equals("pending"))
{
Thread.sleep(1000);
System.out.print(".");
snapState = snapRes.getSnapshot().getState();
}
System.out.println("Done.");
When I run this, I get the following output:
Snapshot request sent.
Waiting for snapshot to be created
snapState is pending
.............................................
Where the dots continue to be printed until I kill the program. In the AWS Web Console, I can see that the snapshot has been created (it now has a green circle marking it as "completed"), but somehow my program has not gotten the message.
When I replace the while loop with a simple wait for a second (insert the line Thread.sleep(1000) after Snapshot snap = snapRes.getSnapshot(); in the first code snippet), the program will often create a new EBS volume without complaint (it then dies when I try to attach the volume to the new instance). Sometimes, however, I will get the IncorrectState error even after waiting for a second. I assume this means that there is some variance in the amount of time it takes to create a snapshot (even of the same EBS volume), and that one second is enough to account for some but not all of the possible delay times.
I could just increase the hard-coded delay to something sure to be longer than the expected time, but that approach has many faults (it waits unnecessarily for most of the times I will use it, it is still not guaranteed to be long enough, and it won't translate well into a solution for the second step, mounting the EBS volume onto the instance).
I would really like to be able to check in with AWS at regular intervals, check to see if the state of the snapshot has changed, and then proceed once it has. What am I doing wrong and how should I fix my code to allow my program to dynamically determine when the snapshot has been fully created?
EDIT: I've tried using getProgress() rather than getState() as per the suggestion. My changed code looks like this:
String snapProgress = snap.getProgress();
System.out.println("snapProgress is " + snapProgress);
// Wait for the snapshot to be created
while (!snapProgress.equals("100%"))
{
Thread.sleep(1000);
System.out.print(".");
snapProgress = snapRes.getSnapshot().getProgress();
}
System.out.println("Done.");
I get the same output as I did when using getState(). I think my problem is that the snapshot object that my code references is not being updated correctly. Is there a better way to refresh/update that object than simply calling its methods repeatedly? My suspicion is that I'm running up against some sort of issue with the way that the API handles requests.
Solved it. I think the problem was that the Snapshot.getState() call doesn't actually make a new call to AWS, but keeps returning the state of the object at the time it was created (which would always be pending).
I fixed the problem by using the describeSnapshots() method:
String snapState = snap.getState();
System.out.println("snapState is " + snapState);
System.out.print("Waiting for snapshot to be created");
// Wait for the snapshot to be created
while (snapState.equals("pending"))
{
Thread.sleep(500);
System.out.print(".");
DescribeSnapshotsResult describeSnapRes
= ec2.describeSnapshots(new DescribeSnapshotsRequest().withSnapshotIds(snap.getSnapshotId()));
snapState = describeSnapRes.getSnapshots().get(0).getState();
}
System.out.println("\nDone.");
This makes a proper call to AWS every time, and it works.
Instead of getstate() , try using getProgress() method. If you are getting it blank then your EBS snapshot is not ready. It gives output in string percentage format ( 100% when your snapshot is ready). Hopefully it should do the trick. Let me know if it works.
This question already has answers here:
Alternative to File.exists() in Java
(6 answers)
Closed 2 years ago.
I am working on a Java program that requires to check the existence of files.
Well, simple enough, the code make use calls to File.exists() for checking file existence. And the problem I have is, it reports false positive. That means the file does not actually exist but exists() method returns true. No exception was captured (at least no exception like "Stale NFS handle"). The program even managed to read the file through InputStream, getting 0 bytes as expected and yet no exception. The target directory is a Linux NFS. And I am 100% sure that the file being looked for never exists.
I know there are known bugs (kind of API limitation) exist for java.io.File.exists(). So I've then added another way round by checking file existence using Linux command ls. Instead of making call to File.exists() the Java code now runs a Linux command to ls the target file. If exit code is 0, file exists. Otherwise, file does not exist.
The number of times the issue is hit seems to be reduced with the introduction of the trick, but still pops. Again, no error was captured anywhere (stdout this time). That means the problem is so serious that even native Linux command won't fix for 100% of the time.
So there are couple of questions around:
I believe Java's well known issue on File.exists() is about reporting false negative. Where file was reported to not exist but in fact does exist. As the API does not throws IOException for File.exists(), it choose to swallow the Exception in the case calls to OS's underlying native functions failed e.g. NFS timeout. But then this does not explain the false positive case I am having, given that the file never exist. Any throw on this one?
My understanding on Linux ls exit code is, 0 means okay, equivalent to file exists. Is this understanding wrong? The man page of ls is not so clear on explaining the meaning of exit code: Exit status is 0 if OK, 1 if minor problems, 2 if serious trouble.
All right, back to subject. Any surefire way to check File existence with Java on Linux? Before we see JDK7 with NIO2 officially released.
Here is a JUnit test that shows the problem and some Java Code that actually tries to read the file.
The problem happens e.g. using Samba on OSX Mavericks. A possible reason
is explaned by the statement in:
http://appleinsider.com/articles/13/06/11/apple-shifts-from-afp-file-sharing-to-smb2-in-os-x-109-mavericks
It aggressively caches file and folder properties and uses opportunistic locking to enable better caching of data.
Please find below a checkFile that will actually attempt to read a few bytes and forcing a true file access to avoid the caching misbehaviour ...
JUnit test:
/**
* test file exists function on Network drive replace the testfile name and ssh computer
* with your actual environment
* #throws Exception
*/
#Test
public void testFileExistsOnNetworkDrive() throws Exception {
String testFileName="/Volumes/bitplan/tmp/testFileExists.txt";
File testFile=new File(testFileName);
testFile.delete();
for (int i=0;i<10;i++) {
Thread.sleep(50);
System.out.println(""+i+":"+OCRJob.checkExists(testFile));
switch (i) {
case 3:
// FileUtils.writeStringToFile(testFile, "here we go");
Runtime.getRuntime().exec("/usr/bin/ssh phobos /usr/bin/touch "+testFileName);
break;
}
}
}
checkExists source code:
/**
* check if the given file exists
* #param f
* #return true if file exists
*/
public static boolean checkExists(File f) {
try {
byte[] buffer = new byte[4];
InputStream is = new FileInputStream(f);
if (is.read(buffer) != buffer.length) {
// do something
}
is.close();
return true;
} catch (java.io.IOException fnfe) {
}
return false;
}
JDK7 was released a few months ago. There are exists and notExists methods in the Files class but they return a boolean rather than throwing an exception. If you really want an exception then use FileSystems.getDefault().provider().checkAccess(path) and it will throw an exception if the file does not exist.
If you need to be robust, try to read the file - and fail gracefully if the file is not there (or there is a permission or other problem). This applies to any other language than Java as well.
The only safe way to tell if the file exist and you can read from it is to actually read a data from the file. Regardless of a file system - local, or remote. The reason is a race condition which can occur right after you get success from checkAccess(path): check, then open file, and you find it suddenly does not exist. Some other thread (or another remote client) may have removed it, or has acquired an exclusive lock. So don't bother checking access, but rather try to read the file. Spending time in running ls just makes race condition window easier to fit.