I am trying to implement a timeout while listing files in java considering that listing files should be IO bound with some CPU need as well. Following is the code:
FileLister:
package com.timeout;
import java.io.File;
import java.util.ArrayList;
import java.util.List;
public class FileLister implements Runnable{
private String fileLocation;
private List<String> fileList = new ArrayList<String>();
public FileLister(String fileLocation){
this.fileLocation = fileLocation;
}
public void run(){
this.listFiles(this.fileLocation);
}
public void listFiles(String fileLocation){
File file = new File(fileLocation);
File testFile = null;
String[] fileList = file.list();
if(null!=fileList){
for(String fileName:fileList){
testFile = new File(fileLocation+"/"+fileName);
if(testFile.isDirectory()){
listFiles(fileLocation+"/"+fileName);
}else{
synchronized(this){
this.fileList.add(fileLocation+"/"+fileName);
}
}
}
}
}
public List<String> getFileList() {
return fileList;
}
}
Timer:
package com.timeout;
public class Timer implements Runnable{
private long timeout;
public Timer(long timeout){
this.timeout = timeout;
}
public void run(){
long expectedEndTime = System.currentTimeMillis() + this.timeout;
System.out.println("expectedEndTime---"+expectedEndTime);
while(System.currentTimeMillis()<expectedEndTime){
}
System.out.println("endTime---"+System.currentTimeMillis());
System.exit(0);
}
}
Calling class:
package com.timeout;
import java.io.IOException;
public class Timeout {
public static void main(String[] args)throws IOException{
FileLister fl = new FileLister("C:/");
Timer tm = new Timer(10000);
Thread flt = new Thread(fl);
Thread tmt = new Thread(tm);
flt.start();
try{
Thread.sleep(1000);
}catch(InterruptedException ie){
System.exit(0);
}
tmt.start();
System.out.println("Files after 11 second--");
for(String fileName:fl.getFileList()){
System.out.println(fileName);
}
}
}
I am putting the timeout as 11 seconds(10+1).
This code is giving me a concurrent modification exception. Why should that occur considering only one thread accessing the fileList variable.
Considering that listing file is IO Bound will timer work concurrently.
I am not considering TimerTask/Executor as of now.
This does not really make sense if you stick to the old File API, since when you .listFiles(), the whole directory entries are swallowed into the array that is returned. That you iterate over it afterwards doesn't make the directory entry loading "lazy".
Where is does make sense is if you use the new java.nio.file API (Java 7+) -- and you should use that and drop File --, since Files.newDirectoryStream() does lazy load directory entries. (*)
This returns a DirectoryStream which is basically a lazy Iterable over the directory entries, except that it also implements Closeable. And therefore you can interrupt this, like in:
try (
final DirectoryStream<Path> entries = Files.newDirectoryStream(...);
) {
for (final Path entry: entries) {
if (Thread.currentThread().isInterrupted())
break;
// proceed with "entry"
}
}
One more reason to use that instead of .listFiles(): if the operation to get a DirectoryStream fails, you don't get null (!!) but an appropriate exception instead: AccessDeniedException, NotDirectoryException, NoSuchFileException, FileSystemLoopException (for filesystems with symlink support), etc etc.
Again: ditch File.
(*): at least for operating systems which support it (Linux has getdents() for instance) or, more generally, for all FileSystem implementations which support it
There are 3 threads in your code, file list thread, time out thread and main thread. The while loop executing in time out thread can not block code executing after tmt.start() line. So the fileList ArrayList(not thread safe) is accessed by main thread and file list thread simultaneously, this is why concurrent modification exception caused.
Related
i know how to send all files from one directory to another in one time.
but how can i want send files (.txt) from one directory to another one by one with a delay of 60s.
import java.io.File;
public class MoveFilesJavaExample {
public static void main(String[] args) {
try {
File oldFile = new File("C:\\source\\File.txt");
if (oldFile.renameTo(new File("C:\\destination"+ oldFile.getName()))) {
System.out.println("The file was moved successfully to the new folder");
} else {
System.out.println("The File was not moved.");
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
I can use the following code but it only do the same job every 5s.
import java.util.Timer;
import java.util.TimerTask;
public class UtilTimerDemo {
public static void main(String[] argv) throws Exception {
int delay = 5000; // delay for 5 sec.
int period = 1000; // repeat every sec.
Timer timer = new Timer();
timer.scheduleAtFixedRate(new TimerTask() {
public void run() {
System.out.println("doing");
}
}, delay, period);
}
}
Thank you
Inside your TimerTask, use File#listFiles() (possibly with the FilenameFilter parameter, if that suits your needs) to get a list of files in the source directory.
Iterate through the returned Files looking for the first item that is not a directory (assuming that you want to move only normal files).
Move that file.
Optionally (depending on your requirements) if you got to the end of the array of returned Files and didn't find any non-directories, then stop the TimerTask.
Start a Timer that fires every 60 seconds.
In the constructor of the Timer, use DirectoryStream<Path> to get all the files in a directory. This is done using the static method of Files class in the nio package.
Then, use Files.move() in the Timer to move the files one by one.
Stop the Timer when you have iterated over all the files in the stream.
DirectoryStream can be globbed, thus allowing you to filter the files that you want. You can either write simple extensions or complex RegEx to decide what files are returned in your DirectoryStream.
It seems like I have a race condition when I call file.getAbsolutePath() in my Java program.
In one thread I am processing a file and when it is finished processing I am changing the filename and moving it to another directory on the UNIX file system.
In a separate thread running in parallel I am attempting to open this file that is being processed and reading its contents. In 99% of use cases this operation is fine however I have noticed sometimes that the operation fails with a FileNotFound exception.
When I catch this exception I am logging the file.getAbsolutePath() value and I see the value is the concatenation of the path of the file in the processed directory it has been moved to and also the path of the file in the directory it was present in before processing completed.
Has anyone experienced a similar problem in the past and how did you get around it?
Thanks
It seems you need to synchronize the file access from separate threads using a class that does this, let's call it FileManager.
First option in implementing the FileManager is to use an exclusive lock. For example:
class FileManager {
private Object lock = new Object();
public void processFile() {
synchronized(lock) {
...
}
}
public void readFile() {
synchronized(lock) {
...
}
}
}
If there are many more readers than writers a Read/Write Lock is more suitable as it allows multiple concurrent readers but only a single writer:
class FileManager {
private final Lock readLock;
private final Lock writeLock;
FileManager() {
ReentrantReadWriteLock readWriteLock = new ReentrantReadWriteLock(false);
readLock = readWriteLock.readLock();
writeLock = readWriteLock.writeLock();
}
public void processFile() {
writeLock.lock();
try {
...
}
finally {
writeLock.unlock();
}
}
public void readFile() {
readLock.lock();
try {
...
}
finally {
readLock.unlock();
}
}
}
I was creating an application in Java for which I want only one instance running. For this purpose I created a file and got a lock while my application is running.
I have following code which works on Windows, but failed on Linux: once I acquire a lock without unlocking it I can get another lock on it.
import java.io.File;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.channels.FileChannel;
import java.nio.channels.FileLock;
public class MyApp {
private static File f;
private static FileChannel channel;
private static FileLock lock;
public static void main(String[] args) {
try {
f = new File("RingOnRequest.lock");
// Check if the lock exist
if (f.exists()) {
// if exist try to delete it
f.delete();
}
// Try to get the lock
channel = new RandomAccessFile(f, "rw").getChannel();
lock = channel.tryLock();
if(lock == null)
{
// File is lock by other application
channel.close();
throw new RuntimeException("Only 1 instance of MyApp can run.");
}
// Add shutdown hook to release lock when application shutdown
ShutdownHook shutdownHook = new ShutdownHook();
Runtime.getRuntime().addShutdownHook(shutdownHook);
//Your application tasks here..
System.out.println("Running");
try {
Thread.sleep(10000);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
catch(IOException e)
{
throw new RuntimeException("Could not start process.", e);
}
}
public static void unlockFile() {
// release and delete file lock
try {
if(lock != null) {
lock.release();
channel.close();
f.delete();
}
} catch(IOException e) {
e.printStackTrace();
}
}
static class ShutdownHook extends Thread {
public void run() {
unlockFile();
}
}
}
You are deleting the lock file every time you run, so only one process can have a lock on it.
When you use FileLock, it is purely advisory—acquiring a lock on a file may not stop you from doing anything…reading, writing, and deleting a file may all be possible even when another process has acquired a lock. Sometimes, a lock might do more than this on a particular platform, but this behavior is unspecified, and relying on more than is guaranteed in the class documentation is a recipe for failure.
An "advisory lock" is only a signal that is visible to other processes that bother to look for it. If you depend on it for more than that, your program will break when run on some other platform.
Why would you delete the lock file anyway? The lock file is just like a boolean flag that is visible to every process on the system. Design your protocol to use it that way, and you'll have a reliable, cross platform locking mechanism.
Why don't you save the PID into a file, and instead of locking the file, verify if there's a process with that ID. If there is, and it's an instance of your application, you know it's already running.
A socket might be a good idea as well, since you can use it to communicate to the running instance something.
EDIT:
Also, from FileLock's javadoc:
Whether or not a lock actually prevents another program from accessing
the content of the locked region is system-dependent and therefore
unspecified.
Use mkdir. On unix systems this is an atomic operation – it will succeed if a new directory is successfully created, otherwise it will fail.
Example:
File lockFile = new File("/path/to/lockdir");
boolean hasLock = lockFile.mkdir();
if (!hasLock) {
throw new IOException("could not get lock");
}
// do stuff
lockFile.delete();
I used same sample as you and got same problem on Mac OS X. It seems that file lock does not prevent file deletion on POSIX systems . Your app wil still have some kind of handle to that file until you unlock it. So consider using lock file with PID in it's name( or inside file).
I tested it on both Windows and Linux. Works fine. The lock file gets deleted automatically when the application closes normally. So you don't have to worry about the lock file staying there when you restart the application. Just comment out the following lines:
if (f.exists()) {
// if exist try to delete it
f.delete();
}
However, you may want to consider what happens if your application crashes and does not close in a normal fashion.
Recently i encountered the same kind of problem, but in my case i had an advantage: my application polled some directory only after some timeout. As my application did not immediately poll for directory i wrote special class that creates lock file with his own PID inside in init method, after that before it tries to work with directory it needs to call ownedLock() - if it returns true then we can work otherwise exit(code is in Kotlin but you will get the main idea):
import java.io.File
import java.lang.management.ManagementFactory
class DirectoryLocker(private val directory: String, private val lockName: String) {
private val lockFile by lazy { File("$directory/$lockName.lock") }
// Will try to acquire lock to directory, whoever last writes its pid to file owns the directory
fun acquireLock() = with(lockFile) {
createNewFile()
writeText(currentPID())
}
fun ownedLock(): Boolean = lockFilePid() == currentPID()
fun releaseOwnedLock() {
if(lockFilePid() == currentPID()) lockFile.delete()
}
private fun currentPID(): String {
val processName = ManagementFactory.getRuntimeMXBean().name
return processName.split("#".toRegex()).first()
}
private fun lockFilePid(): String? {
return if(lockFile.exists()) lockFile.readLines().first() else null
}
}
I am using multiple threads to upload files to the server. The Java Applet is responsible for displaying the UI. Initially I start 5 threads using ThreadPoolExecutor & assign 5 files to them . After each upload, I get a notification from the server. When a thread completes execution , another new thread is assigned with a file until all the files are uploaded to the server.
Basic code structure as follows:
i> a method startUpload() is being called from the Java Applet which is responsible for handling the upload functionality.
class Upload extends Runnable{
...............................
..............................
public void startUpload() {
............................... //other initialisations done
int waitTime = 500;
Random random = new Random();
ExecutorService executor = new ThreadPoolExecutor(5, 5, 50000L, TimeUnit.MILLISECONDS, new ArrayBlockingQueue<Runnable>(300));
while (it.hasNext()) {
int time = random.nextInt(1000);
waitTime += time;
newFile = new File((String) it.next());
executor.execute(new Runnable() {
#Override
public void run() {
try{
Thread.sleep(wait);
}
catch(Exception e){
}
processFile1(newFile);
}
});
}
try {
Thread.sleep(waitTime);
executor.shutdown();
executor.awaitTermination(waitTime, TimeUnit.MILLISECONDS);
} catch (Exception e) {
}
}
}
The problem I am facing currently.
i> The UI is only updating at the end when all the files are upload. In the intermediate stage the UI is in a hanged state. It seems like the EDT is going to a blocked state.
The same code for UI rendering was working fine when I was using Thread class , notify/ sleep to implement the same functionality . I changed the code to ThreadPoolExecutor since I saw in a no of blogs/articles that its a better way of implementing multithreading from Java ver 5.0.
ii> Another thing which I noticed with the ThreadPoolExecutor , when I am uploading multiple files with size 1KB (for testing purpose) , if I remove all the wait() from the above code , the following line assigns a new file but the the same file is always being uploaded everytime by the multiple threads.
newFile = new File((String) it.next());
But on adding sleep() withing the run() , the multiple threads upload different files to the server.
Is there any implementation issue with the above code ?
Problem 1: newFile is a (static?) field instead of a local variable.
What you want is to make sure that the local capture of newFile is different each loop. As such, it should look more like:
while(it.hasNext()) {
final File newFile = new File((String) it.next());
executor.execute(new Runnable() {
#Override
public void run() {
processFile1(newFile); // Local only to this iteration of the loop.
}
}
}
Your code is all wrapped in a Runnable instance. Can you let us know what Thread this is called from? If it's on the EDT then that would explain why the UI locks up.
A small issue is the lack of generics on your iterator. In theory, you should be iterating over a collection of Strings:
Collection<String> listOfFiles = ...
Iterator<String> it = listOfFiles.iterator();
while(it.hasNext()) {
String filename = it.next(); // No cast necessary
}
The UI is hanging because you are blocking the EDT thread. This code is the culprit:
try {
Thread.sleep(waitTime);
executor.shutdown();
executor.awaitTermination(waitTime, TimeUnit.MILLISECONDS);
} catch (Exception e) {
}
The idea of an ExecutorService is that you create it one time during initialization and never shut it down until the program is ready to exit. An idiom for this might be:
ExecutorService executor = Executors.newFixedThreadPool(5);
Runtime.getRuntime().addShutdownHook(new Thread() {
public void run() {
executor.shutdown();
}
});
As #Bringer128 mentioned, the second problem is caused by the fact that you are changing the value of a static or member variable and not assigning the File reference to a new location. If the code were correct, we would expect to see the newFile declared as final File newFile because non-final local variables may not be referenced in an inner-class.
I'd like to have my thread (the main/EDT) wait until changes to a file occur and then wait. DefaultFileMonitor extends Runnable and hence runs in a thread of its own. Here is a SSCE:
import java.io.File;
import org.apache.commons.vfs.*;
import org.apache.commons.vfs.impl.DefaultFileMonitor;
public class FileChangeListener implements FileListener {
DefaultFileMonitor fm;
public final static File logFile = new File("t.txt");
public void startListening() throws FileSystemException {
final FileSystemManager fsManager = VFS.getManager();
final FileObject listendir = fsManager.toFileObject(logFile);
fm = new DefaultFileMonitor(this);
fm.addFile(listendir);
fm.start();
}
#Override
public void fileCreated(FileChangeEvent fce) throws Exception {
fileChanged(fce);
}
#Override
public void fileDeleted(FileChangeEvent fce) throws Exception {
//hmm..why deleted?
}
#Override
public void fileChanged(FileChangeEvent fce) throws Exception {
System.out.println("fileChanged executed");
}
}
The main:
import java.io.PrintWriter;
public class App {
public static void main(String[] args) {
FileChangeListener fcl = new FileChangeListener();
try {
fcl.startListening();
final PrintWriter printWriter = new PrintWriter(FileChangeListener.logFile);
printWriter.println("Hello Threads!");
printWriter.close();
//EXECUTE THE FOLLOWING ONLY AFTER fileChanged
System.out.println("Mission complete.");
} catch (Exception ex) {
}
}
}
Append the following to App.main(..) after printWriter.close():
synchronized (fcl) {
fcl.wait();
}
//EXECUTE THE FOLLOWING ONLY AFTER fileChanged
System.out.println("Mission complete.");
and append the following to FileChangeListener.fileChanged(..) after System.out.println("fileChanged executed"):
synchronized (this) {
this.notifyAll();
}
You could communicate between teh two using "Conditions" : http://download.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/locks/Condition.html
Basically, create a new "shared" Condition (say fileChanged). Now, whenever the file changes (in fileChanged() trigger this condition (fileChanged.signal()). In your main code, wait for this condition to occur (fileChanged.await()).
Hope you get the idea.
For making the condition accessible to multiple code unit, here is what I can think (decreasing order of preference) :
Assuming you are going to need as many conditions as many files you listen to, create a factory method getCondition(String file path/name/attribute) which will return the Condition object based on the file (its path or name or other attributes). Use this factory method to get the condition in all cases. The factory should internally create new Condition() instances for each new file to be listened to AND must throw away older instances as the processing of the files is complete (so probably you should add a destroy/deleteCondition(String file) method as well.)
Store the condition as a public field in the listener class (kind of hack if you have the listener instance available).
Store the condition as a public static field in the listener class (kind of hack if you have only one listener instance throughout).
Why? FileChangeListener is a callback: it is executed when the event occurs. In this specific case you've just closed the file so you alreayd know that the mission is complete on that file, so just proceed to the next step. I don't see why you need a FileChangeListener at all here.