What is the best way to share a file between two "writer" services in the same application?
Edit:
Sorry I should have given more details I guess.
I have a Service that saves entries into a buffer. When the buffer gets full it writes all the entries to the file (and so on). Another Service running will come at some point and read the file (essentially copy/compress it) and then empty it.
Here is a general idea of what you can do:
public class FileManager
{
private final FileWriter writer = new FileWriter("SomeFile.txt");
private final object sync = new object();
public void writeBuffer(string buffer)
{
synchronized(sync)
{
writer.write(buffer.getBytes());
}
}
public void copyAndCompress()
{
synchronized(sync)
{
// copy and/or compress
}
}
}
You will have to do some extra work to get it all to work safe, but this is just a basic example to give you an idea of how it looks.
A common method for locking is to create a second file in the same location as the main file. The second file may contain locking data or be blank. The benefit to having locking data (such as a process ID) is that you can easily detect a stale lockfile, which is an inevitability you must plan for. Although PID might not be the best locking data in your case.
example:
Service1:
creates myfile.lock
creates/opens myfile
Service2:
Notices that myfile.lock is present and pauses/blocks/waits
When myfile.lock goes away, it creates it and then opens myfile.
It would also be advantageous for you to double-check that the file contains your locking information (identification specific to your service) right after creating it - just in case two or more services are waiting and create a lock at the exact same time. The last one succeeds and so all other services should notice that their locking data is no longer in the file. Also - pause a few milliseconds before checking its contents.
Related
I write a thread safe class to get input from multiple threads and upload the result to S3 once it runs up to a fixed size.
S3Exporter class
// this class is thread safe.
public class S3Exporter {
private static final int BUFFER_PADDING = 1000;
private final int targetSize;
private final ByteArrayOutputStream buf;
private volatile boolean started;
public S3Exporter(final int targetSize) {
buf = new ByteArrayOutputStream(targetSize + BUFFER_PADDING);
this.targetSize = targetSize;
started = false;
}
public synchronized void start() {
started = true;
}
public synchronized void end() {
started = false;
flush();
}
public synchronized void export(byte[] data) throws IOException {
Preconditions.checkState(started, "Not started!");
buf.write(b, buf.size(), b.length);
flushIfNeeded();
}
private void flushIfNeeded() {
if (buf.size() >= targetSize) {
flush();
}
}
public synchronized void flush() {
if (buf.size() > 0) {
// upload buf to s3, it's a time-consuming operation
buf.reset();
}
}
}
The client calls export method to pass data and if exception is thrown the client will pass that data later.
To avoid losing data when restarting the application, I add a shutdown hook when creating S3Exporter object:
S3Exporter exporter = new S3Exporter(10000);
Runtime.getRuntime().addShutdownHook(new Thread(() -> exporter.end()));
My concern is the class is not scalable, I mean it could become bottleneck of the system when data are getting more. I could figure out 2 ways to improve the situation:
do the time-consuming upload operation asynchronously: use an executor to upload and call ThreadPoolExecutor.awaitTermination() in the shutdown hook.
just put data to a LinkedBlockingQueue in export method and use multiple threads to handle it.( This way is more scalable than the first per my understanding)
Then I need to do more work in the shutdown hook thread to make sure not losing the accepted data and it's not a good idea as I know. I'll take the risk of losing data when restarting the application, which is the last thing I wanna see.
My question
Is my concern about the scalability a really problem?( To make the question less stupid, let's say the data size is a few bytes and TPS to call export method is 500)
If the answer to the 1st question is yes, what about my improvements, are they right? How to do the cleanup work to avoid losing data?
Scalability depends on requirements, constraints, desired service level, personal preferences, expected users growth rate, and especially money: given infinite resources, every piece of software can be scaled. You didn't mention any, so I guess you don't have any actual figure. In this phase, as a programmer, your job is to make a correct program that uses a predictable amount of resources.
Your program seems correct, and most of your assumptions are correct, too. However I suggest to immediately store chunks to some local persistent database (or the raw filesystem) and have a periodic job, run in a separate thread, that upload group of chunks to S3, and remove any shutdown hooks (you can use Camel for the boring parts). This is because such hooks are unreliable and should only be used as last resources for quick and optional cleanup (optional in the sense that you must be prepared that the cleanup could not have been run properly until the end).
Using a file instead of memory, your data can survive fatal errors and the working memory required by your application is almost independent on the load: there's an irrelevant amount of extra CPU and some disk I/O that is way cheaper then memory.
Note: this question isn't necessary just for Android developers. It should be the same for Java developers that performed root operations on Linux too.
Background
My app allows the user the perform various root operations (an operation that can only be performed on rooted devices).
Until now, each time the app wishes to perform a root operation, I had opened a new process using the Runtime.getRuntime().exec("su") command.
I wish to improve the way the app works, by calling this command only once, and then writing to the outputStream and reading from the inputStream while doing so.
A similar app that does this is "Terminal emulator". Once you type there "su" (and grant it root permission), it will allow you go perform any root operation and read any path of the device.
The problem
It seems that only the "echo" command works, but any other command doesn't. Also, I couldn't think of a good way to separate between the root operations output.
What I've tried
First, in order to initialize the root batching operations, I did this:
process=Runtime.getRuntime().exec("su");
_inputStreamReader=new BufferedReader(new InputStreamReader(process.getInputStream()));
_out=process.getOutputStream();
final String testLine=Long.toString(System.currentTimeMillis());
_out.write(("echo "+testLine+"\n").getBytes());
_out.flush();
final String resultLine=_inputStreamReader.readLine();
if(resultLine==null)
{
_out.write("exit \n".getBytes());
_out.flush();
_out.close();
_out=null;
_inputStreamReader.close();
_inputStreamReader=null;
process.waitFor();
return false;
}
while(!resultLine.equals(testLine))
resultLine=_inputStreamReader.readLine();
return true;
the code above (is supposed to) ensures I have root and that I can continue working on the "_out" and "inputStreamReader" . the call to "echo" has 2 purposes : to have some fake operation being done, and to also become a starting point and a separator between commands.
Here the second purpose is fake, but for each operation that I make further, this is how I thought of handling the separation of various root operations.
Now, if I want to make another root operation (assuming the above works fine), this is an example of an operation I wish to perform :
_out.write(("find "+someFolderPath+"/* -type f\n").getBytes());
_out.flush();
final String line=_inputStreamReader.readLine();
Thing is, it gets stuck on the "readLine" part, even though it should return a list of all of the file/folders within the specified path.
This command works perfectly using the previous way I performed root operations, and it should work here too.
using "echo" before those lines and then using "readLine()" shows that it's still available, as I will still get the result line. However, trying to call "readLine()" again (after the above command) will still make it stuck.
I've also tried adding "su" (or "su -" or "su -c") before the "find" command, but it didn't help.
This is the old code, which works fine:
final Process p=Runtime.getRuntime().exec("su");
final InputStream inputStream=p.getInputStream();
final DataOutputStream os=new DataOutputStream(p.getOutputStream());
os.writeBytes("find "+someFolderPath+"/* -type f\n");
os.writeBytes("exit\n");
os.flush();
//read from the inputstream (using BufferedReader) as much as I wish, till it ends.
The question
How should I avoid re-creating new processes for root operations? Is this a good way to do it?
How can I fix the above code? What is wrong with it?
Also, Is there a good way to run multiple root operations using a single "su" command, yet without any kind of separator like the one I've used (so that I could differentiate when a new command output starts and the previous one ends).
I am sure, that your find command is just too slow. You should run it in the different process.
There is great library to do this. And a manual.
OK, even though it's not quite what I wanted, and even though I wanted to learn how to do it myself, I've found that "libsuperuser" library (by "ChainFire") does it much easier.
What I've asked for is called "interactive mode" in this library.
Here's a sample class I've made that helps you do it using this library:
public class Root
{
private static final Root _instance =new Root();
private Boolean _hasRoot =null;
private Shell.Interactive _rootSession;
public interface IGotRootListener
{
public void onGotRootResult(boolean hasRoot);
}
public static Root getInstance()
{
return _instance;
}
public boolean hasRoot()
{
return _hasRoot!=null&&_hasRoot;
}
//should be called on the UI thread
public void getRoot(final IGotRootListener listener)
{
if(_hasRoot!=null&&_hasRoot)
{
listener.onGotRootResult(true);
return;
}
final AtomicReference<Interactive> rootSessionRef=new AtomicReference<>();
rootSessionRef.set(new Shell.Builder().useSU().setWantSTDERR(true).setWatchdogTimeout(5).setMinimalLogging(true).open(new Shell.OnCommandResultListener()
{
#Override
public void onCommandResult(final int commandCode,final int exitCode,final List<String> output)
{
final boolean success=exitCode==Shell.OnCommandResultListener.SHELL_RUNNING;
if(success)
_rootSession=rootSessionRef.get();
_hasRoot=success;
listener.onGotRootResult(success);
}
}));
}
... // TODO add functions to call commands via _rootSession.addCommand , only if got root
Still, if anyone has a solution to my original question, I would love to know about it.
I have a situation where I have a large number of classes that need to do file (read only) access. This is part of a web app running on top of OSGI, so there will be a lot of concurrent needs to access.
So I'm building an OSGI service to access the file system for all the other pieces that will need it and provide a centralized access as this also simplifies configuration of file locations, etc.
It occurs to me that a multi-threaded approach makes the most sense along with a thread pool.
So the question is this:
If I do this and I have a service with an interface like:
FileService.getFileAsClass(class);
and the method getFileAsClass(class) looks kinda like this: (this is a sketch it may not be perfect java code)
public < T> T getFileAsClass(Class< T> clazz) {
Future<InputStream> classFuture = threadpool.submit(new Callable< InputStream>() {
/* initialization block */
{
//any setup from configs.
}
/* implement Callable */
public InputStream call() {
InputStream stream = //new inputstream from file location;
boolean giveUp = false;
while(null == stream && !giveUp) {
//Code that tries to read in the file 4
// times with a Thread.sleep() then gives up
// this is here t make sure we aren't busy updating file.
}
return stream;
}
});
//once we have the file, convert it and return it.
return InputStreamToClassConverter< T>.convert(classFuture.get());
}
Will that correctly wait until the relevant operation is done to call InputStreamtoClassConverter.convert?
This is my first time writing multithreaded java code so I'm not sure what I can expect for some of the behavior. I don't care about order of which threads complete, only that the file handling is handled async and once that file pull is done, then and only then is the Converter used.
General purpose of program
To read in a bash-pattern and specified location from command line, and find all files matching that pattern in the location but I have to make the program multi-threaded.
General structure of the program
Driver/Main Class which parses arguments and initiates other classes.
ProcessDirectories Class which adds all directory addresses found from the specified root directory to a string array for processing later
DirectoryData Class which holds the addresses found in the above class
ProcessMatches Class which examines each directory found, and adds any files inside that match the pattern to a string array for printing results later
Main/Driver once again takes over and prints the results :)
The Problem
I need to be processing matches even whilst the ProcessDirectories class is still working (for efficiency so I don't unnecessarily wait for the list to populate before doing work). To do this I try to: a) make ProcessMatches threads wait() if DirectoryData is empty b) make ProcessDirectories notifyAll() if added a new entry.
The Question :)
Every tutorial I look at is focused on the producer and consumer being in the same object, or dealing with just one data structure. How can I do this when I am using more than one data structure and more than one class for producing and consuming?
How about something like:
class Driver(String args)
{
ProcessDirectories pd = ...
BlockingQueue<DirectoryData> dirQueue = new LinkedBlockingQueue<DirectoryData>();
new Thread(new Runnable(){public void run(){pd.addDirs(dirQueue);}}).start();
ProcessMatches pm = ...
BlockingQueue<File> fileQueue = new LinkedBlockingQueue<File>();
new Thread(new Runnable()
{
public void run()
{
for (DirectoryData dir = dirQueue.take(); dir != DIR_POISON; dir = dirQueue.take())
{
for (File file : dir.getFiles())
{
if (pm.matches(data))
fileQueue.add(file)
}
}
fileQueue.add(FILE_POISON);
}
}).start();
for (File file = fileQueue.take(); file != FILE_POISON; file = fileQueue.take())
{
output(file);
}
}
This is just a rough idea of course. ProcessDirectories.addDirs() would just add DirectoryData objects to the queue. In production you'd want to name the threads. Perhaps use an executor to provide manage threads. Perhaps use some other mechanism to indicate end of processing than a poison message. Also, you might want to reduce the limit on the queue size.
Have one data structure that's associated with the data the two threads communicate with each other. This can be a queue that has "get data from queue, waiting if empty" and "put data on queue, waiting if full" functions. Those functions should internally call notify and wait on the queue itself and they should be synchronized to that queue.
I have an object responsible for persisting JTable state to disk. It saves/loads visible columns, their size, position etc. A few interesting bits from its class definition are below.
class TableSaver {
Timer timer = new Timer(true);
TableSaver() {
timer.schedule(new TableSaverTimerTask(), 15000, SAVE_STATE_PERIOD);
}
synchronized TableColumns load(PersistentTable table) {
String xml = loadFile(table.getTableKey());
// parse XML, return
}
synchronized void save(String key, TableColumns value) {
try {
// Some preparations
writeFile(app.getTableConfigFileName(key), xml);
} catch (Exception e) {
// ... handle
}
}
private class TableSaverTimerTask extends TimerTask {
#Override
public void run() {
synchronized (TableSaver.this) {
Iterator<PersistentTable> iterator = queue.iterator();
while (iterator.hasNext()) {
PersistentTable table = iterator.next();
if (table.getTableKey() != null) {
save(table.getTableKey(), dumpState(table));
}
iterator.remove();
}
}
}
}
}
There only exists one instance of TableSaver, ever.
load() can be called from many threads. Timer clearly is another thread.
loadFile() and writeFile() do not leave open file streams - they use a robust, well tested and broadly used library which always closes the streams with try ... finally.
Sometimes this fails with an exception like:
java.lang.RuntimeException: java.io.FileNotFoundException: C:\path\to\table-MyTable.xml (The requested operation cannot be performed on a file with a user-mapped section open)
at package.FileUtil.writeFile(FileUtil.java:33)
at package.TableSaver.save(TableSaver.java:175)
at package.TableSaver.access$600(TableSaver.java:34)
at package.TableSaver$TableSaverTimerTask.run(TableSaver.java:246)
at java.util.TimerThread.mainLoop(Unknown Source)
at java.util.TimerThread.run(Unknown Source)
Caused by: java.io.FileNotFoundException: C:\path\to\table-MyTable.xml (The requested operation cannot be performed on a file with a user-mapped section open)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(Unknown Source)
at java.io.FileOutputStream.<init>(Unknown Source)
at package.FileUtilWorker.writeFile(FileUtilWorker.java:57)
... 6 more
So I have two questions:
How can this kind of synchronization fail? Note that I am sure there only is one instance of TableSaver.
What is this thing in the stacktrace: package.TableSaver.access$600(TableSaver.java:34)? Line 34 is the line with class TableSaver {. Can this be the reason why the synchronization is not working?
Google learns me that this seems to be Windows specific. Here's an extract of Bug 6354433:
This is Windows platform issue with memory-mapped file, i.e. MappedByteBuffer. The Java 5.0 doc for FileChannel state that "the buffer and the mapping that it represents will remain valid until the buffer itself is garbage-collected". The error occurs when we tried to re-open the filestore and the mapped byte buffer has not been GC. Since there is no unmap() method for mapped byte buffer (see bug 4724038), we're at the mercy of the underlying operating system on when the buffer get free up. Calling System.gc() might free up the buffer but it is not guarantee. The problem doesn't occurs on Solaris; may be due to the way shared memory is implemented on Solaris. So the work-around for Windows is not to use memory-mapped file for the transaction information tables.
What Java/Windows version are you using? Does it have the latest updates?
Here are two other related bugs with some useful insights:
Bug 4715154 - Memory mapped file cannot be deleted.
Bug 4469299 - Memory mapped files are not GC'ed.
As to your second question, that's just the autogenerated classname of an inner or anonymous class.
Assuming there are no issues with the code I have seen this occur when a virus scanner is running in the background which is cheerfully opening files to scan them behind the scenes. If you have a memory resident virus scanner that checks files in the background try disabling it, or at least disabling it for the directory you reading from/writing to.
Your code looks fine. Are you sure it's not related to file permission? Does the application has write privilege to this folder? To this file?
[EDIT] This seems to be Windows related, not Java The requested operation cannot be performed on a file with a user-mapped section open.
I had this issue with some tightly threaded Java code. I took a look at the referenced .NET conversation and the penny dropped. It is simply that I have contention for the same file, among different threads. Looking more closely the contention is (also) for some internals too. So my best course is to synchronize-d around the shared object when the update it.
This works and the error dissolves in the mist.
private static ShortLog tasksLog = new ShortLog( "filename" );
private static Boolean tasksLogLock = false;
...
synchronized( tasksLogLock ){
tasksLog.saveLastDatum( this.toString() );
}
see also:
Synchronization of non-final field
Your synchronization only protects against access from your own process. If you want to protect against accesses from any process, you have to use file locking:
http://download.oracle.com/javase/1.4.2/docs/api/java/nio/channels/FileLock.html