I need create a new constructor in FolderScan that takes a list of "Checkers". And all these "Checkers" always return true (schoud write new Chekers List that just return true.)
But problem is that I don't know how do this and not decompose structure of program.
Code (FolderScan and each Cheker):
class FolderScan implements Runnable {
FolderScan(String path, BlockingQueue<File> queue, CountDownLatch latch,
File endOfWorkFile) {
this.path = path;
this.queue = queue;
this.latch = latch;
this.endOfWorkFile = endOfWorkFile;
checkers = new ArrayList<Checker>(Arrays.asList(
new ExtentionChecking(), new ProbeContentTypeCheking(),
new EncodingChecking() ));
}
#Override
public void run() {
try {
findFiles(path);
queue.put(endOfWorkFile);
latch.countDown();
} catch (InterruptedException e) {
e.printStackTrace();
}
}
private void findFiles(String path) {
try {
File root = new File(path);
File[] list = root.listFiles();
for (File currentFile : list) {
boolean checksPassed = true;
if (currentFile.isDirectory()) {
findFiles(currentFile.getAbsolutePath());
} else {
for (Checker currentChecker : checkers) {
if (!currentChecker.check(currentFile)) {
checksPassed = false;
break;
}
}
if (checksPassed) {
queue.put(currentFile);
}
}
}
} catch (InterruptedException e) {
e.printStackTrace();
}
private String path;
private BlockingQueue<File> queue;
private CountDownLatch latch;
private File endOfWorkFile;
private List<Checker> checkers;
}
class ExtentionChecking implements Checker {
#Override
public boolean check(File currentFile) {
fileName = currentFile.getName().toLowerCase();
Set<String> extensions = new HashSet<String>(Arrays.asList(".txt",
".pdf", ".doc", ".docx", ".html", ".htm", ".xml", ".djvu",
".djv", ".rar", ".rtf", ".tmp"));
if (extensions.contains(fileName.substring(fileName.lastIndexOf(".")))) {
return true;
}
return false;
}
private String fileName;
}
class EncodingChecking implements Checker {
#Override
public boolean check(File currentFile) {
return detectEncoding(currentFile);
}
public static boolean detectEncoding(File file) {
detector = new CharsetDetector();
// validate input
if (null == file) {
throw new IllegalArgumentException("input file can't be null");
}
if (file.isDirectory()) {
throw new IllegalArgumentException(
"input file refers to a directory");
}
// read input file
byte[] buffer;
try {
buffer = readUTFHeaderBytes(file);
} catch (IOException e) {
throw new IllegalArgumentException(
"Can't read input file, error = " + e.getLocalizedMessage());
}
if(detector.setText(buffer) != null){
return true;
}
return false;
}
private static byte[] readUTFHeaderBytes(File input) throws IOException {
// read data
FileInputStream fileInputStream = new FileInputStream(input);
try {
byte firstBytes[] = new byte[50];
int count = fileInputStream.read(firstBytes);
if (count < 5) {
throw new IOException("Poor file!");
}
return firstBytes;
} finally {
fileInputStream.close();
}
}
private static CharsetDetector detector;
}
class ProbeContentTypeCheking implements Checker {
#Override
public boolean check(File currentFile) {
String mimeType = null;
try {
Path path = Paths.get(currentFile.getAbsolutePath());
byte[] data = Files.readAllBytes(path);
MagicMatch match = Magic.getMagicMatch(data);
mimeType = match.getMimeType();
} catch (MagicParseException | MagicMatchNotFoundException
| MagicException | IOException e) {
e.printStackTrace();
}
if (null != mimeType) {
return true;
}
return false;
}
}
Question:
How do refactor this code - after this able to make new
AllwaysPassesBlocker() and all Checers return true?
A checker that always returns true would be
class UncriticalChecker implements Checker {
#Override
public boolean check(File currentFile) {
return true;
}
}
There's no point adding such a checker to the list of checkers, though. You might as well leave the list empty.
I don't quite see why the checkers should be constructed in the constructor of the FolderScan. It seems more natural to pass them to the constructor as an argument.
FolderScan(String path, BlockingQueue<File> queue, CountDownLatch latch,
File endOfWorkFile, List<Checker> checkers) {
this.path = path;
this.queue = queue;
this.latch = latch;
this.endOfWorkFile = endOfWorkFile;
this.checkers = checkers;
}
Then, when you initialize the FolderScan, pass it the checkers
List<Checker> checkers = new ArrayList<Checker>(Arrays.asList(
new ExtentionChecking(), new ProbeContentTypeCheking(),
new EncodingChecking() ));
FolderScan folderScan =
new FolderScan(path, queue, latch, endOfWorkFile, checkers);
Or, if you wish to create a FolderScan that returns all files, you pass it an empty list.
FolderScan folderScan =
new FolderScan(path, queue, latch, endOfWorkFile, Collections.emptyList());
EDIT:
I now understand that you wish to test the class. Then the UncriticalChecker makes sense. If you want to test the code with a checker that always says yes, pass it to the constructor:
List<Checker> checkers = Collections.singletonList(new UncriticalChecker());
FolderScan folderScan =
new FolderScan(path, queue, latch, endOfWorkFile, checkers);
Related
I want to watch a directory for file changes. And I used WatchService in java.nio. I can successfully listen for file created event. But I can't listen for file modify event. I checked official java tutorial, but still struggling.
Here is the source code.
import static java.nio.file.LinkOption.NOFOLLOW_LINKS;
import static java.nio.file.StandardWatchEventKinds.ENTRY_CREATE;
import static java.nio.file.StandardWatchEventKinds.OVERFLOW;
import static java.nio.file.StandardWatchEventKinds.ENTRY_DELETE;
import static java.nio.file.StandardWatchEventKinds.ENTRY_MODIFY;
import java.io.File;
import java.io.IOException;
import java.nio.file.FileSystem;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.WatchEvent;
import java.nio.file.WatchEvent.Kind;
import java.nio.file.WatchKey;
import java.nio.file.WatchService;
public class MainWatch {
public static void watchDirectoryPath(Path path) {
// Sanity check - Check if path is a folder
try {
Boolean isFolder = (Boolean) Files.getAttribute(path,
"basic:isDirectory", NOFOLLOW_LINKS);
if (!isFolder) {
throw new IllegalArgumentException("Path: " + path
+ " is not a folder");
}
} catch (IOException ioe) {
// Folder does not exists
ioe.printStackTrace();
}
System.out.println("Watching path: " + path);
// We obtain the file system of the Path
FileSystem fs = path.getFileSystem();
// We create the new WatchService using the new try() block
try (WatchService service = fs.newWatchService()) {
// We register the path to the service
// We watch for creation events
path.register(service, ENTRY_CREATE);
path.register(service, ENTRY_MODIFY);
path.register(service, ENTRY_DELETE);
// Start the infinite polling loop
WatchKey key = null;
while (true) {
key = service.take();
// Dequeueing events
Kind<?> kind = null;
for (WatchEvent<?> watchEvent : key.pollEvents()) {
// Get the type of the event
kind = watchEvent.kind();
if (OVERFLOW == kind) {
continue; // loop
} else if (ENTRY_CREATE == kind) {
// A new Path was created
Path newPath = ((WatchEvent<Path>) watchEvent)
.context();
// Output
System.out.println("New path created: " + newPath);
} else if (ENTRY_MODIFY == kind) {
// modified
Path newPath = ((WatchEvent<Path>) watchEvent)
.context();
// Output
System.out.println("New path modified: " + newPath);
}
}
if (!key.reset()) {
break; // loop
}
}
} catch (IOException ioe) {
ioe.printStackTrace();
} catch (InterruptedException ie) {
ie.printStackTrace();
}
}
public static void main(String[] args) throws IOException,
InterruptedException {
// Folder we are going to watch
// Path folder =
// Paths.get(System.getProperty("C:\\Users\\Isuru\\Downloads"));
File dir = new File("C:\\Users\\Isuru\\Downloads");
watchDirectoryPath(dir.toPath());
}
}
Actually you have incorrectly subscribed to events. Only last listener has been registered with ENTRY_DELETE events type.
To register for all kind of events at once you should use:
path.register(service, ENTRY_CREATE, ENTRY_MODIFY, ENTRY_DELETE);
Warning! Shameless self promotion!
I have created a wrapper around Java 1.7's WatchService that allows registering a directory and any number of glob patterns. This class will take care of the filtering and only emit events you are interested in.
DirectoryWatchService watchService = new SimpleDirectoryWatchService(); // May throw
watchService.register( // May throw
new DirectoryWatchService.OnFileChangeListener() {
#Override
public void onFileCreate(String filePath) {
// File created
}
#Override
public void onFileModify(String filePath) {
// File modified
}
#Override
public void onFileDelete(String filePath) {
// File deleted
}
},
<directory>, // Directory to watch
<file-glob-pattern-1>, // E.g. "*.log"
<file-glob-pattern-2>, // E.g. "input-?.txt"
... // As many patterns as you like
);
watchService.start();
Complete code is in this repo.
I made some classes for this.
public interface FileAvailableListener {
public void fileAvailable(File file) throws IOException;
}
and
public class FileChange {
private long lastModified;
private long size;
private long lastCheck;
public FileChange(File file) {
this.lastModified=file.lastModified();
this.size=file.length();
this.lastCheck = System.currentTimeMillis();
}
public long getLastModified() {
return lastModified;
}
public long getSize() {
return size;
}
public long getLastCheck() {
return lastCheck;
}
public boolean isStable(FileChange other,long stableTime) {
boolean b1 = (getLastModified()==other.getLastModified());
boolean b2 = (getSize()==other.getSize());
boolean b3 = ((other.getLastCheck()-getLastCheck())>stableTime);
return b1 && b2 && b3;
}
}
and
public class DirectoryWatcher {
private Timer timer;
private List<DirectoryMonitorTask> tasks = new ArrayList<DirectoryMonitorTask>();
public DirectoryWatcher() throws URISyntaxException, IOException, InterruptedException {
super();
timer = new Timer(true);
}
public void addDirectoryMonitoringTask(DirectoryMonitorTask task,long period) {
tasks.add(task);
timer.scheduleAtFixedRate(task, 5000, period);
}
public List<DirectoryMonitorTask> getTasks() {
return Collections.unmodifiableList(tasks);
}
public Timer getTimer() {
return timer;
}
}
and
class DirectoryMonitorTask extends TimerTask {
public final static String DIRECTORY_NAME_ARCHIVE="archive";
public final static String DIRECTORY_NAME_ERROR="error";
public final static String LOCK_FILE_EXTENSION=".lock";
public final static String ERROR_FILE_EXTENSION=".error";
public final static String FILE_DATE_FORMAT="yyyyMMddHHmmssSSS";
private String name;
private FileAvailableListener listener;
private Path directory;
private File directoryArchive;
private File directoryError;
private long stableTime;
private FileFilter filter;
private WatchService watchService;
private SimpleDateFormat dateFormatter = new SimpleDateFormat(FILE_DATE_FORMAT);
private Hashtable<File,FileChange> fileMonitor = new Hashtable<File,FileChange>();
public DirectoryMonitorTask(String name,FileAvailableListener listener,Path directory,long stableTime,FileFilter filter) throws IOException {
super();
this.name=name;
this.listener=listener;
this.directory=directory;
this.stableTime=stableTime;
if (stableTime<1) {
stableTime=1000;
}
this.filter=filter;
validateNotNull("Name",name);
validateNotNull("Listener",listener);
validateNotNull("Directory",directory);
validate(directory);
directoryArchive = new File(directory.toFile(),DIRECTORY_NAME_ARCHIVE);
directoryError = new File(directory.toFile(),DIRECTORY_NAME_ERROR);
directoryArchive.mkdir();
directoryError.mkdir();
//
log("Constructed for "+getDirectory().toFile().getAbsolutePath());
initialize();
//
watchService = FileSystems.getDefault().newWatchService();
directory.register(watchService,StandardWatchEventKinds.ENTRY_CREATE,StandardWatchEventKinds.ENTRY_DELETE,StandardWatchEventKinds.ENTRY_MODIFY);
log("Started");
}
private void initialize() {
File[] files = getDirectory().toFile().listFiles();
for (File file : files) {
if (isLockFile(file)) {
file.delete();
} else if (acceptFile(file)) {
fileMonitor.put(file,new FileChange(file));
log("Init file added -"+file.getName());
}
}
}
public SimpleDateFormat getDateFormatter() {
return dateFormatter;
}
public Path getDirectory() {
return directory;
}
public FileAvailableListener getListener() {
return listener;
}
public String getName() {
return name;
}
public WatchService getWatchService() {
return watchService;
}
public long getStableTime() {
return stableTime;
}
public File getDirectoryArchive() {
return directoryArchive;
}
public File getDirectoryError() {
return directoryError;
}
public FileFilter getFilter() {
return filter;
}
public Iterator<File> getMonitoredFiles() {
return fileMonitor.keySet().iterator();
}
#Override
public void run() {
WatchKey key;
try {
key = getWatchService().take();
// Poll all the events queued for the key
for (WatchEvent<?> event : key.pollEvents()) {
#SuppressWarnings("unchecked")
Path filePath = ((WatchEvent<Path>) event).context();
File file = filePath.toFile();
if ((!isLockFile(file)) && (acceptFile(file))) {
switch (event.kind().name()) {
case "ENTRY_CREATE":
//
fileMonitor.put(file,new FileChange(file));
log("File created ["+file.getName()+"]");
break;
//
case "ENTRY_MODIFY":
//
fileMonitor.put(file,new FileChange(file));
log("File modified ["+file.getName()+"]");
break;
//
case "ENTRY_DELETE":
//
log("File deleted ["+file.getName()+"]");
createLockFile(file).delete();
fileMonitor.remove(file);
break;
//
}
}
}
// reset is invoked to put the key back to ready state
key.reset();
} catch (InterruptedException e) {
e.printStackTrace();
}
Iterator<File> it = fileMonitor.keySet().iterator();
while (it.hasNext()) {
File file = it.next();
FileChange fileChange = fileMonitor.get(file);
FileChange fileChangeCurrent = new FileChange(file);
if (fileChange.isStable(fileChangeCurrent, getStableTime())) {
log("File is stable ["+file.getName()+"]");
String filename = getDateFormatter().format(new Date())+"_"+file.getName();
File lockFile = createLockFile(file);
if (!lockFile.exists()) {
log("File do not has lock file ["+file.getName()+"]");
try {
Files.createFile(lockFile.toPath());
log("Processing file ["+file.getName()+"]");
getListener().fileAvailable(file);
file.renameTo(new File(getDirectoryArchive(),filename));
log("Moved to archive file ["+file.getName()+"]");
} catch (IOException e) {
file.renameTo(new File(getDirectoryError(),filename));
createErrorFile(file,e);
log("Moved to error file ["+file.getName()+"]");
} finally {
lockFile.delete();
}
} else {
log("File do has lock file ["+file.getName()+"]");
fileMonitor.remove(file);
}
} else {
log("File is unstable ["+file.getName()+"]");
fileMonitor.put(file,fileChangeCurrent);
}
}
}
public boolean acceptFile(File file) {
if (getFilter()!=null) {
return getFilter().accept(file);
} else {
return true;
}
}
public boolean isLockFile(File file) {
int pos = file.getName().lastIndexOf('.');
String extension="";
if (pos!=-1) {
extension = file.getName().substring(pos).trim().toLowerCase();
}
return(extension.equalsIgnoreCase(LOCK_FILE_EXTENSION));
}
private File createLockFile(File file) {
return new File(file.getParentFile(),file.getName()+LOCK_FILE_EXTENSION);
}
private void createErrorFile(File file,IOException exception) {
File errorFile = new File(file.getParentFile(),file.getName()+ERROR_FILE_EXTENSION);
StringWriter sw = null;
PrintWriter pw = null;
FileWriter fileWriter = null;
try {
//
fileWriter = new FileWriter(errorFile);
if (exception!=null) {
sw = new StringWriter();
pw = new PrintWriter(sw);
exception.printStackTrace(pw);
fileWriter.write(sw.toString());
} else {
fileWriter.write("Exception is null.");
}
//
fileWriter.flush();
//
} catch (IOException e) {
} finally {
if (sw!=null) {
try {
sw.close();
} catch (IOException e1) {
}
}
if (pw!=null) {
pw.close();
}
if (fileWriter!=null) {
try {
fileWriter.close();
} catch (IOException e) {
}
}
}
}
private void validateNotNull(String name,Object obj) {
if (obj==null) {
throw new NullPointerException(name+" is null.");
}
}
private void validate(Path directory) throws IOException {
File file = directory.toFile();
if (!file.exists()) {
throw new IOException("Directory ["+file.getAbsolutePath()+"] do not exists.");
} else if (!file.isDirectory()) {
throw new IOException("Directory ["+file.getAbsolutePath()+"] is not a directory.");
} else if (!file.canRead()) {
throw new IOException("Can not read from directory ["+file.getAbsolutePath()+"].");
} else if (!file.canWrite()) {
throw new IOException("Can not write to directory ["+file.getAbsolutePath()+"] .");
}
}
private void log(String msg) {
//TODO
System.out.println("Task ["+getName()+"] "+msg);
}
}
package p1;
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import static java.nio.file.LinkOption.NOFOLLOW_LINKS;
import java.nio.file.StandardWatchEventKinds;
import java.nio.file.WatchEvent;
import java.nio.file.WatchKey;
import java.nio.file.WatchService;
import java.util.List;
public class WatchForFile {
public void WatchMyFolder(String path )
{
File dir = new File(path);
Path myDir= dir.toPath();
try
{
Boolean isFolder = (Boolean) Files.getAttribute(myDir,"basic:isDirectory", NOFOLLOW_LINKS);
if (!isFolder)
{
throw new IllegalArgumentException("Path: " + myDir + " is not a folder");
}
}
catch (IOException ioe)
{
ioe.printStackTrace();
}
System.out.println("Watching path: " + myDir);
try {
WatchService watcher = myDir.getFileSystem().newWatchService();
myDir.register(watcher, StandardWatchEventKinds.ENTRY_CREATE,StandardWatchEventKinds.ENTRY_DELETE, StandardWatchEventKinds.ENTRY_MODIFY);
WatchKey watckKey = watcher.take();
List<WatchEvent<?>> events = watckKey.pollEvents();
for (WatchEvent event : events) {
if (event.kind() == StandardWatchEventKinds.ENTRY_CREATE) {
System.out.println("Created: " + event.kind().toString());
}
if (event.kind() == StandardWatchEventKinds.ENTRY_DELETE) {
System.out.println("Delete: " + event.context().toString());
}
if (event.kind() == StandardWatchEventKinds.ENTRY_MODIFY) {
System.out.println("Modify: " + event.context().toString());
}
}
}
catch (Exception e)
{
System.out.println("Error: " + e.toString());
}
}
}
Check this Code...
https://github.com/omkar9999/FileWatcherHandler
This project allows watching files for different file events like create, modify & delete and then act on these events in a generic way.
How to Use?
Create a Path object representing the directory to monitor for file events.
Path path = Paths.get("/home/omkar/test");
Implement the FileHandler interface to perform an action detected by file event registered.
public class FileHandlerTest implements FileHandler {
private static final Logger LOGGER = Logger.getLogger(FileHandlerTest.class.getName());
/*
* This implemented method will delete the file
*
* #see com.io.util.FileHandler#handle(java.io.File,
* java.nio.file.WatchEvent.Kind)
*/
public void handle(File file, Kind<?> fileEvent) {
LOGGER.log(Level.INFO,"Handler is triggered for file {0}",file.getPath());
if(fileEvent == StandardWatchEventKinds.ENTRY_CREATE) {
try {
boolean deleted = Files.deleteIfExists(Paths.get(file.getPath()));
assertTrue(deleted);
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
Create an instance of an Implemented FileHandler
FileHandlerTest fileHandlerTest = new FileHandlerTest();
Create an instance of a FileWatcher by passing path, an instance of an Implemented FileHandler, and types of file events that you want to monitor separated by commas.
FileWatcher fileWatcher = new FileWatcher(path, fileHandlerTest, StandardWatchEventKinds.ENTRY_CREATE);
Now Create and start a new Thread.
Thread watcherThread = new Thread(fileWatcher);
watcherThread.start();
This thread will start polling for your registered file events and will invoke your custom handle method once any of the registered events are detected.
public class FileWatcher implements Runnable {
private static final Logger LOGGER =Logger.getLogger(FileWatcher.class.getName());
private WatchService watcher;
private FileHandler fileHandler;
private List<Kind<?>> watchedEvents;
private Path directoryWatched;
/**
* #param directory
* #Path directory to watch files into
* #param fileHandler
* #FileHandler implemented instance to handle the file event
* #param watchRecursive
* if directory is to be watched recursively
* #param watchedEvents
* Set of file events watched
*
* #throws IOException
*/
public FileWatcher(Path directory, FileHandler fileHandler, boolean watchRecursive,
WatchEvent.Kind<?>... watchedEvents) throws IOException {
super();
this.watcher = FileSystems.getDefault().newWatchService();
this.fileHandler = fileHandler;
this.directoryWatched = directory;
this.watchedEvents = Arrays.asList(watchedEvents);
if (watchRecursive) {
// register all subfolders
Files.walkFileTree(directory, new SimpleFileVisitor<Path>() {
#Override
public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) throws IOException {
LOGGER.log(Level.INFO, "Registering {0} ", dir);
dir.register(watcher, StandardWatchEventKinds.ENTRY_CREATE, StandardWatchEventKinds.ENTRY_DELETE,
StandardWatchEventKinds.ENTRY_MODIFY);
return FileVisitResult.CONTINUE;
}
});
} else {
directory.register(watcher, watchedEvents);
}
}
#SuppressWarnings({ "unchecked" })
public void run() {
LOGGER.log(Level.INFO, "Starting FileWatcher for {0}", directoryWatched.toAbsolutePath());
WatchKey key = null;
while (true) {
try {
key = watcher.take();
if (key != null) {
for (WatchEvent<?> event : key.pollEvents()) {
WatchEvent.Kind<?> kind = event.kind();
WatchEvent<Path> ev = (WatchEvent<Path>) event;
//directory in which file event is detected
Path directory = (Path) key.watchable();
Path fileName = ev.context();
if (watchedEvents.contains(kind)) {
LOGGER.log(Level.INFO, "Invoking handle on {0}", fileName.toAbsolutePath());
fileHandler.handle(directory.resolve(fileName).toFile(), kind);
}
}
key.reset();
}
} catch (InterruptedException ex) {
LOGGER.log(Level.SEVERE, "Polling Thread was interrupted ", ex);
Thread.currentThread().interrupt();
}
}
}
}
I have some places in a excel file, each of the point have a lng and lat coordinate.
Now I try to create a static Map for each point using the google map static map api.
And I have Two component, a parser and a loader.
The Parser is used to read the excel file while the loaded is used to load tiles.
And I make the loader run in a seprate Thread.
public class Parser {
private static Parser instance;
private StaticMapLoader loader;
private Parser(StaticMapLoader loader) {
this.loader = loader;
}
public synchronized static Parser getInstance(StaticMapLoader loader) {
if (instance == null) {
instance = new Parser(loader);
}
return instance;
}
public void parse(String path) {
List<Branch> result = new ArrayList<Branch>();
InputStream inp;
try {
inp = new FileInputStream(path);
Workbook wb = WorkbookFactory.create(inp);
Sheet sheet = wb.getSheetAt(0);
int rows = sheet.getLastRowNum();
for(Row r : sheet.getRows){
loader.addTask(r.type,r.name,r.x,r.y);
}
} catch (InvalidFormatException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
// Branch bc = new Branch("网点1", null, null);
return result;
}
}
Loader:
public class StaticMapLoader extends Thread {
private final static Logger log = Logger.getLogger(StaticMapLoader.class);
private List<Task> tasks = new ArrayList<Task>();
private String tilePath;
private boolean running = false;
public StaticMapLoader(String saveDir) {
this.tilePath = saveDir;
}
#Override
public void run() {
while (running) {
log.debug("run " + tasks.size());
if (tasks.size() > 0) {
Task t = tasks.get(0);
if (t != null && t.status == Status.waiting) {
tasks.remove(0);
t.status = Status.running;
downLoad(t);
}
}
}
}
private void downLoad(Task t) {
log.debug(String.format("load data for " + t.toString()));
//down tiles and save
t.status=Status.success;
}
public void addTask(String type, String name, double x, double y) {
log.debug(String.format("add task of :%s,%s", type, name));
tasks.add(new Task(type,name,x,y));
}
public void startRunning() {
running = true;
this.start();
}
public void stopRunning() {
running = false;
this.interrupt();
}
class Task {
Status status = Status.waiting;
String type, name;
double x,y;
Task(String type, String name, double x,double y) {
this.type = type;
this.name = name;
this.xian = xian;
this.x = x;
this.y = y;
}
}
enum Status {
waiting, running, fail, success
}
}
The process is rather simple, the StaticMapLoader have a field of ArrayList. While the Parser parse a record(place), it will be thrown to the list.
And the loader will iterator the list and download the data.
However I meet a strange problem here:
#Override
public void run() {
while (running) {
log.debug("run " + tasks.size());
if (tasks.size() > 0) {
Task t = tasks.get(0);
if (t != null && t.status == Status.waiting) {
tasks.remove(0);
t.status = Status.running;
downLoad(t);
}
}
}
}
The above codes runs, and I will get the logs like this:
run 1
add task of ..
run 2
add task of ...
However , if I comment the log line, the downLoad will be never called, I will get:
run 1
run 2
......
It seems that this may be caused by the Thread , do I miss anything?
BTW, the above codes ran inside the HttpServlet context, and I start them like this:
#Override
public void init() throws ServletException {
super.init();
try {
URL fileUrl = getServletContext().getResource(getInitParameter("xlsxFile"));
URL tilePath = getServletContext().getResource(getInitParameter("tilePath"));
StaticMapLoader loader = new StaticMapLoader(tilePath.getPath());
loader.startRunning();
Parser.getInstance(loader).parse(fileUrl.getPath());
} catch (MalformedURLException e) {
}
}
I'm begginer and keep yourself in hends.
I need to do organize multithreadings find in files:
User input where find(path) and what find(word);
First thread finds .txt files in folder and add result to queue;
When queue has some file => Second thread start find in this file
what need to find(word).
If was finded success would show path this
file + how offen times this word meets in file.
Qestions:
Can we use ArrayList (or exist any alternatives) for queue which works with few threads?
How to do if queue is empty, Second thread don't start but waits when First finded need file?
Need we use synchronized for this task and inherited MultiThreadingSearch(or better to use composition)?
Code:
import java.util.*;
import java.io.*;
class ArrayListOfFiles {
private Node first, last;
private class Node {
String item;
Node next;
}
public boolean isEmpty() {
return first == null;
}
public synchronized void enqueue(String item) {
Node oldlast = last;
last = new Node();
last.item = item;
last.next = null;
if (isEmpty())
first = last;
else
oldlast.next = last;
}
public synchronized String dequeue() {
String item = first.item;
first = first.next;
if (isEmpty())
last = null;
return item;
}
}
class FolderScan extends MultiThreadingSearch implements Runnable {
FolderScan(String path, String whatFind) {
super(path, whatFind);
}
#Override
public void run() {
findFiles(path);
}
ArrayListOfFiles findFiles(String path) {
File root = new File(path);
File[] list = root.listFiles();
for (File titleName : list) {
if (titleName.isDirectory()) {
findFiles(titleName.getAbsolutePath());
} else {
if (titleName.getName().toLowerCase().endsWith((".txt"))) {
textFiles.enqueue(titleName.getName());
}
}
}
return textFiles;
}
}
class FileScan extends MultiThreadingSearch implements Runnable {
Scanner scanner = new Scanner((Readable) textFiles);
Set<String> words = new HashSet<String>();
int matches = 0;
FileScan(String file, String whatFind) {
super(file, whatFind);
Thread wordFind = new Thread();
wordFind.start();
}
#Override
public void run() {
while (scanner.hasNext()) {
String word = scanner.next();
words.add(word);
}
if (words.contains(this.whatFind)) {
System.out.println("File:" + this.path);
matches++;
}
System.out.println(matches);
}
}
public class MultiThreadingSearch {
String path;
String whatFind;
ArrayListOfFiles textFiles;
MultiThreadingSearch(String path, String whatFind) {
this.path = path;
this.whatFind = whatFind;
this.textFiles = new ArrayListOfFiles();
Thread pathFind = new Thread(new FolderScan(path, whatFind));
// pathFind.start();
if (!textFiles.isEmpty()) {
#SuppressWarnings("unused")
FileScan fileScan = new FileScan(textFiles.dequeue(), whatFind);
}
}
// ask user about input
public static void askUserPathAndWord() {
BufferedReader bufferedReader = new BufferedReader(
new InputStreamReader(System.in));
String path;
String whatFind;
try {
System.out.println("Please, enter a Path and Word"
+ "(which you want to find):");
System.out.println("Please enter a Path:");
path = bufferedReader.readLine();
System.out.println("Please enter a Word:");
whatFind = bufferedReader.readLine();
if (path != null && whatFind != null) {
new MultiThreadingSearch(path, whatFind);
System.out.println("Thank you!");
} else {
System.out.println("You did not enter anything");
}
} catch (IOException | RuntimeException e) {
System.out.println("Wrong input!");
e.printStackTrace();
}
}
public static void main(String[] args) {
askUserPathAndWord();
}
}
I got Exception in thread "main" java.lang.StackOverflowError from this code.
How able to solve this task?
Thanks,
Nazar.
Check BlockingQueue it does exactly what you need. Thread can block until some other thread add new item to queue.
As to how decompose you system. I'd do following:
Create class for searching txt files in path. It implements Runnable. You pass path and queue to it. And it searches path for txt files and adds them to the queu.
Create class for searching file content. It implements Runnable. You pass whatFind and queue to it and it takes new file from queue and checks it's content.
Something like:
BlockingQueue<File> queue = new LinkedBlockingQueue<File>();
String path = ...;
String whatFind = ...;
FolderScan folderScan = new FolderScan(path, queue);
FileScan fileScan = new FileScan(whatFind, queue);
Executor executor = Executors.newCachecThreadPool();
executor.execute(folderScan);
executor.execute(fileScan);
If you want FileScan to wait until FolderScan adds something to the queue you can use take method:
BlockingQueue<File> queue;
File toProcess = queue.take(); // this line blocks current thread (FileScan) until someone adds new item to the queue.
After changes:
package task;
import java.util.concurrent.*;
import java.util.*;
import java.io.*;
class FolderScan implements Runnable {
private String path;
private BlockingQueue<File> queue;
private CountDownLatch latch;
private File endOfWorkFile;
FolderScan(String path, BlockingQueue<File> queue, CountDownLatch latch,
File endOfWorkFile) {
this.path = path;
this.queue = queue;
this.latch = latch;
this.endOfWorkFile = endOfWorkFile;
}
public FolderScan() { }
#Override
public void run() {
findFiles(path);
queue.add(endOfWorkFile);
latch.countDown();
}
private void findFiles(String path) {
try {
File root = new File(path);
File[] list = root.listFiles();
for (File currentFile : list) {
if (currentFile.isDirectory()) {
findFiles(currentFile.getAbsolutePath());
} else {
if (currentFile.getName().toLowerCase().endsWith((".txt"))) {
queue.put(currentFile);
}
}
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
public class FileScan implements Runnable {
private String whatFind;
private BlockingQueue<File> queue;
private CountDownLatch latch;
private File endOfWorkFile;
public FileScan(String whatFind, BlockingQueue<File> queue,
CountDownLatch latch, File endOfWorkFile) {
this.whatFind = whatFind;
this.queue = queue;
this.latch = latch;
this.endOfWorkFile = endOfWorkFile;
}
public FileScan() { }
Set<String> words = new HashSet<String>();
int matches = 0;
#Override
public void run() {
while (true) {
try {
File file;
file = queue.take();
if (file == endOfWorkFile) {
break;
}
scan(file);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
latch.countDown();
}
private void scan(File file) {
Scanner scanner = null;
try {
scanner = new Scanner(file);
} catch (FileNotFoundException e) {
System.out.println("FileNotFoundException.");
e.printStackTrace();
}
while (scanner.hasNext()) {
String word = scanner.next();
words.add(word);
}
if (words.contains(this.whatFind)) {
matches++;
}
String myStr = String.format("File: %s and the number of matches "
+ "is = %d", file.getAbsolutePath(), matches);
System.out.println(myStr);
matches = 0;
}
// ask user about input
public void askUserPathAndWord() {
BufferedReader bufferedReader = new BufferedReader(
new InputStreamReader(System.in));
String path;
String whatFind;
BlockingQueue<File> queue = new LinkedBlockingQueue<File>();
try {
System.out.println("Please, enter a Path and Word"
+ "(which you want to find):");
System.out.println("Please enter a Path:");
path = bufferedReader.readLine();
System.out.println("Please enter a Word:");
whatFind = bufferedReader.readLine();
if (path != null && whatFind != null) {
File endOfWorkFile = new File("GameOver.tmp");
CountDownLatch latch = new CountDownLatch(2);
FolderScan folderScan = new FolderScan(path, queue, latch,
endOfWorkFile);
FileScan fileScan = new FileScan(whatFind, queue, latch,
endOfWorkFile);
Executor executor = Executors.newCachedThreadPool();
executor.execute(folderScan);
executor.execute(fileScan);
latch.await();
System.out.println("Thank you!");
} else {
System.out.println("You did not enter anything");
}
} catch (IOException | RuntimeException e) {
System.out.println("Wrong input!");
e.printStackTrace();
} catch (InterruptedException e) {
System.out.println("Interrupted.");
e.printStackTrace();
}
}
/**
* #param args
*/
public static void main(String[] args) {
new FileScan().askUserPathAndWord();
}
}
This may not sound too constructive, but you can either fix that code or read something like this first and then throw away your code.
Stackoverflow usually results from a recursion running deeper than expected. Make sure there is some condition in you recursive method that stops recursion.
I have a program that performs lots of calculations and reports them to a file frequently. I know that frequent write operations can slow a program down a lot, so to avoid it I'd like to have a second thread dedicated to the writing operations.
Right now I'm doing it with this class I wrote (the impatient can skip to the end of the question):
public class ParallelWriter implements Runnable {
private File file;
private BlockingQueue<Item> q;
private int indentation;
public ParallelWriter( File f ){
file = f;
q = new LinkedBlockingQueue<Item>();
indentation = 0;
}
public ParallelWriter append( CharSequence str ){
try {
CharSeqItem item = new CharSeqItem();
item.content = str;
item.type = ItemType.CHARSEQ;
q.put(item);
return this;
} catch (InterruptedException ex) {
throw new RuntimeException( ex );
}
}
public ParallelWriter newLine(){
try {
Item item = new Item();
item.type = ItemType.NEWLINE;
q.put(item);
return this;
} catch (InterruptedException ex) {
throw new RuntimeException( ex );
}
}
public void setIndent(int indentation) {
try{
IndentCommand item = new IndentCommand();
item.type = ItemType.INDENT;
item.indent = indentation;
q.put(item);
} catch (InterruptedException ex) {
throw new RuntimeException( ex );
}
}
public void end(){
try {
Item item = new Item();
item.type = ItemType.POISON;
q.put(item);
} catch (InterruptedException ex) {
throw new RuntimeException( ex );
}
}
public void run() {
BufferedWriter out = null;
Item item = null;
try{
out = new BufferedWriter( new FileWriter( file ) );
while( (item = q.take()).type != ItemType.POISON ){
switch( item.type ){
case NEWLINE:
out.newLine();
for( int i = 0; i < indentation; i++ )
out.append(" ");
break;
case INDENT:
indentation = ((IndentCommand)item).indent;
break;
case CHARSEQ:
out.append( ((CharSeqItem)item).content );
}
}
} catch (InterruptedException ex){
throw new RuntimeException( ex );
} catch (IOException ex) {
throw new RuntimeException( ex );
} finally {
if( out != null ) try {
out.close();
} catch (IOException ex) {
throw new RuntimeException( ex );
}
}
}
private enum ItemType {
CHARSEQ, NEWLINE, INDENT, POISON;
}
private static class Item {
ItemType type;
}
private static class CharSeqItem extends Item {
CharSequence content;
}
private static class IndentCommand extends Item {
int indent;
}
}
And then I use it by doing:
ParallelWriter w = new ParallelWriter( myFile );
new Thread(w).start();
/// Lots of
w.append(" things ").newLine();
w.setIndent(2);
w.newLine().append(" more things ");
/// and finally
w.end();
While this works perfectly well, I'm wondering:
Is there a better way to accomplish this?
Your basic approach looks fine. I would structure the code as follows:
import java.io.BufferedWriter;
import java.io.File;
import java.io.IOException;
import java.io.Writer;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.TimeUnit;
public interface FileWriter {
FileWriter append(CharSequence seq);
FileWriter indent(int indent);
void close();
}
class AsyncFileWriter implements FileWriter, Runnable {
private final File file;
private final Writer out;
private final BlockingQueue<Item> queue = new LinkedBlockingQueue<Item>();
private volatile boolean started = false;
private volatile boolean stopped = false;
public AsyncFileWriter(File file) throws IOException {
this.file = file;
this.out = new BufferedWriter(new java.io.FileWriter(file));
}
public FileWriter append(CharSequence seq) {
if (!started) {
throw new IllegalStateException("open() call expected before append()");
}
try {
queue.put(new CharSeqItem(seq));
} catch (InterruptedException ignored) {
}
return this;
}
public FileWriter indent(int indent) {
if (!started) {
throw new IllegalStateException("open() call expected before append()");
}
try {
queue.put(new IndentItem(indent));
} catch (InterruptedException ignored) {
}
return this;
}
public void open() {
this.started = true;
new Thread(this).start();
}
public void run() {
while (!stopped) {
try {
Item item = queue.poll(100, TimeUnit.MICROSECONDS);
if (item != null) {
try {
item.write(out);
} catch (IOException logme) {
}
}
} catch (InterruptedException e) {
}
}
try {
out.close();
} catch (IOException ignore) {
}
}
public void close() {
this.stopped = true;
}
private static interface Item {
void write(Writer out) throws IOException;
}
private static class CharSeqItem implements Item {
private final CharSequence sequence;
public CharSeqItem(CharSequence sequence) {
this.sequence = sequence;
}
public void write(Writer out) throws IOException {
out.append(sequence);
}
}
private static class IndentItem implements Item {
private final int indent;
public IndentItem(int indent) {
this.indent = indent;
}
public void write(Writer out) throws IOException {
for (int i = 0; i < indent; i++) {
out.append(" ");
}
}
}
}
If you do not want to write in a separate thread (maybe in a test?), you can have an implementation of FileWriter which calls append on the Writer in the caller thread.
One good way to exchange data with a single consumer thread is to use an Exchanger.
You could use a StringBuilder or ByteBuffer as the buffer to exchange with the background thread. The latency incurred can be around 1 micro-second, doesn't involve creating any objects and which is lower using a BlockingQueue.
From the example which I think is worth repeating here.
class FillAndEmpty {
Exchanger<DataBuffer> exchanger = new Exchanger<DataBuffer>();
DataBuffer initialEmptyBuffer = ... a made-up type
DataBuffer initialFullBuffer = ...
class FillingLoop implements Runnable {
public void run() {
DataBuffer currentBuffer = initialEmptyBuffer;
try {
while (currentBuffer != null) {
addToBuffer(currentBuffer);
if (currentBuffer.isFull())
currentBuffer = exchanger.exchange(currentBuffer);
}
} catch (InterruptedException ex) { ... handle ... }
}
}
class EmptyingLoop implements Runnable {
public void run() {
DataBuffer currentBuffer = initialFullBuffer;
try {
while (currentBuffer != null) {
takeFromBuffer(currentBuffer);
if (currentBuffer.isEmpty())
currentBuffer = exchanger.exchange(currentBuffer);
}
} catch (InterruptedException ex) { ... handle ...}
}
}
void start() {
new Thread(new FillingLoop()).start();
new Thread(new EmptyingLoop()).start();
}
}
Using a LinkedBlockingQueue is a pretty good idea. Not sure I like some of the style of the code... but the principle seems sound.
I would maybe add a capacity to the LinkedBlockingQueue equal to a certain % of your total memory.. say 10,000 items.. this way if your writing is going too slow, your worker threads won't keep adding more work until the heap is blown.
I know that frequent write operations
can slow a program down a lot
Probably not as much as you think, provided you use buffering.
I would like to know how to access the Service and Domains properly in this sample class placed in src/java folder
public class NewsIngestion implements Runnable {
private String str;
private int num;
private Logger log = Logger.getLogger("grails.app");
private static boolean isRunning;
private Thread t;
private WorkerJobService jobService;
private NewsService newsService;
public NewsIngestion(String s, int n)
{
jobService = new WorkerJobService();
newsService = new NewsService();
str = s;
num = n;
isRunning = false;
t = new Thread (this, "NewsIngestion");
}
public void run ()
{
while(isRunning){
try{
if(jobService.isJobEnabled("ConsumeFeedsJob") && jobService.lockJob("ConsumeFeedsJob")){
log.info("${this.class.name}: ConsumeFeedsJob started");
try{
// get all sources
List sources = (List) InvokerHelper.invokeMethod(RSSFeed.class, "list", null);
for(int i = 0; i < sources.size(); i++) {
RSSFeed s = (RSSFeed) sources.get(i);
// check if it's time to read the source
int diff = DateTimeUtil.getSecondsDateDiff(s.getLastChecked(), new Date());
if(s.getLastChecked() == null || diff >= s.getCheckInterval()){
List keyword_list = (List) InvokerHelper.invokeMethod(Keyword.class, "list", null);
for(int j = 0; j < keyword_list.size(); j++) {
String keyword = (String) keyword_list.get(j);
try{
newsService.ingestNewsFromSources(keyword, s);
}catch(Exception e){
log.error("${this.class.name}: ${e}");
}
log.debug("Completed reading feeds for ${keyword}.");
log.info("${this.class.name}: Reading feeds for '${keyword}' (${s.feedName}) took ${Float.toString(st2.getDuration())} second(s).");
}
s.setLastChecked(new Date());
InvokerHelper.invokeMethod(RSSFeed.class, "save", null);
}
log.info("${this.class.name}: Reading feeds for '${s.feedName}' for all keywords took ${Float.toString(st.getDuration())} second(s).");
}
}catch(Exception e){
log.error("${this.class.name}: Exception: ${e}");
}
log.info("${this.class.name}: ConsumeFeedsJob ended.");
// unlock job
jobService.unlockJob("ConsumeFeedsJob");
}
log.info("alfred: success");
}
catch (Exception e){
log.info("alfred exception: " + e.getMessage());
}
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
log.info(e.getMessage());
}
}
}
public void start() {
if(t == null){
t = new Thread (this, "NewsIngestion");
}
if(!isRunning){
isRunning = true;
t.start();
}
}
public void stop() {
isRunning = false;
}
public boolean isRunning() {
return isRunning;
}
}
I'm encountering this error message:
No Hibernate Session bound to thread,
and configuration does not allow
creation of non-transactional one here
Thanks.
You shouldn't instantiate the service class by yourself, but instead take the class instance from the main context
import org.codehaus.groovy.grails.commons.ApplicationHolder
def ctx = ApplicationHolder.application.mainContext
def newsService = ctx.newsService
If you're using Java
import org.codehaus.groovy.grails.commons.ApplicationHolder
public class SomeClass {
SomeService someService;
public SomeClass() {
someService = (SomeService) ApplicationHolder.getApplication().getMainContext().getBean("someService");
}
}
Consider using Spring and #Transactional annotation or AOP.