I have an SQLite3 database on a shared folder. I want to overwrite the db file from a Java application. Though there is low read and write traffic on this file, I want to ensure that a) the overwrite doesn't corrupt the db file, and b) anyone who might be looking to access the db file will essentially see it locked until the overwrite is complete. My current plan looks something like this...
String query = "BEGIN EXCLUSIVE TRANSACTION";
/* Execute this query*/
File sourceFile = new File(LocalPath);
File destFile = new File(DbPath);
InputStream inStream = new FileInputStream(sourceFile);
OutputStream outStream = new FileOutputStream(destFile);
byte[] buffer = new byte[1024];
int length;
while((length = inStream.read(buffer)) > 0) {
outStream.write(buffer, 0, length);
}
inStream.close();
outStream.close();
/* Now release lock */
query = "ROLLBACK TRANSACTION";
/* Execute query */
So reading guidance from SQLite here http://www.sqlite.org/howtocorrupt.html, it would seem this lock would exist in the journal and be updated after copy when I run the rollback transaction. In the meantime, if a client tries to access the Db while I'm copying, I imagine they would just not find the file and my SQLite driver assume the db doesn't exist. Right?
My question is... is it a moot point to place a lock on the db? Is there a better strategy or a way to make the db appear locked rather than missing? Also, am I running a huge risk in DB corruption that makes this unfeasible? One other thought I've had would be to lock the db file, write the new file to a different name, then rename after writing is complete, then release the lock... any thoughts on that?
Not sure if overwriting the DB file is a brilliant idea, but the only feasible thing I can think of for the resources I have at my disposal (running huge transactions on a db in a shared folder over the network is unacceptably slow. I am aware I run higher risk of corruption working with an SQLite db on a shared folder). I am writing updates locally and then letting the user elect to "commit" changes and initiate the db file copy.
In addition to answering my questions, any general advice on this case is welcome...
The Backup API allows you to overwrite a database.
(I don't know if your Java wrapper exposes this API.)
Related
I have a temporary file which I want to send the client from the controller in the Play Framework. Can I delete the file after opening a connection using FileInputStream? For example can I do something like this -
File file = getFile();
InputStream is = new FileInputStream(file);
file.delete();
renderBinary(is, "name.txt");
What if file is a large file? If I delete the file, will subsequent reads() on InputStream give an error? I have tried with files of around 1MB I don't get an error.
Sorry if this is a very naive question, but I could not find anything related to this and I am pretty new to Java
I just encountered this exact same scenario in some code I was asked to work on. The programmer was creating a temp file, getting an input stream on it, deleting the temp file and then calling renderBinary. It seems to work fine even for very large files, even into the gigabytes.
I was surprised by this and am still looking for some documentation that indicates why this works.
UPDATE: We did finally encounter a file that caused this thing to bomb. I think it was over 3 Gb. At that point, it became necessary to NOT delete the file while the rendering was in process. I actually ended up using the Amazon Queue service to queue up messages for these files. The messages are then retrieved by a scheduled deletion job. Works out nicely, even with clustered servers on a load balancer.
It seems counter-intuitive that the FileInputStream can still read after the file is removed.
DiskLruCache, a popular library in the Android world originating from the libcore of the Android platform, even relies on this "feature", as follows:
// Open all streams eagerly to guarantee that we see a single published
// snapshot. If we opened streams lazily then the streams could come
// from different edits.
InputStream[] ins = new InputStream[valueCount];
try {
for (int i = 0; i < valueCount; i++) {
ins[i] = new FileInputStream(entry.getCleanFile(i));
}
} catch (FileNotFoundException e) {
....
As #EJP pointed out in his comment on a similar question, "That's how Unix and Linux behave. Deleting a file is really deleting its name from the directory: the inode and the data persist while any processes have it open."
But I don't think it is a good idea to rely on it.
I output several temporary files in my application to tmp directories but was wondering if it is best practise that I delete them on close or should I expect the host OS to handle this for me?
I am pretty new to Java, I can handle the delete but want to keep the application as multi-OS and Linux friendly as possible. I have tried to minimise file deletion if I don't need to do it.
This is the method I am using to output the tmp file:
try {
java.io.InputStream iss = getClass().getResourceAsStream("/nullpdf.pdf");
byte[] data = IOUtils.toByteArray(iss);
iss.read(data);
iss.close();
String tempFile = "file";
File temp = File.createTempFile(tempFile, ".pdf");
FileOutputStream fos = new FileOutputStream(temp);
fos.write(data);
fos.flush();
fos.close();
nopathbrain = temp.getAbsolutePath();
System.out.println(tempFile);
System.out.println(nopathbrain);
} catch (IOException ex) {
ex.printStackTrace();
System.out.println("TEMP FILE NOT CREATED - ERROR ");
}
createTempFile() only creates a new file with a unique name, but does not mark it for deletion. Use deleteOnExit() on the created file to achieve that. Then, if the JVM shuts down properly, the temporary files should be deleted.
edit:
Sample for creating a 'true' temporary file in java:
File temp = File.createTempFile("temporary-", ".pdf");
temp.deleteOnExit();
This will create a file in the default temporary folder with a unique random name (temporary-{randomness}.pdf) and delete it when the JVM exits.
This should be sufficient for programs with a short to medium run time (e.g. scripts, simple GUI applications) that do sth. and then exit. If the program runs longer or indefinitely (server application, a monitoring client, ...) and the JVM won't exit, this method may clog the temporary folder with files. In such a situation the temporary files should be deleted by the application, as soon as they are not needed anymore (see delete() or Files helper class in JDK7).
As Java already abstracts away OS specific file system details, both approaches are as portable as Java. To ensure interoperability have a look at the new Path abstraction for file names in Java7.
I'm trying to make an applet that reads a file on the local file system (the users computer) at a very frequent interval (several times a second), then makes the contents of the file available to the web page via javascript.
The file the applet needs to read is updated at a high frequency by a program on the user's computer. What I'm concerned about is what might happen if the applet reads data from the file when the file is in the middle of being updated.
I don't know how likely this is, but if it is a valid concern is there a way to make sure the file is not currently being written to before reading it?
I'm not positive about this, but you could try java.io.FileInputStream, or some other option from that package.
Also, this question may be a duplicate. This might answer your question:
How do I use Java to read from a file that is actively being written?
reading a file while it's being written
Read a file while it's being written
Reading data from a File while it is being written to
its very monster to make such a disk access, any way try Sockets if you can or if again you sits back try to lock file in both ends if the one of the locking fails then make sure that other is locking ,make up this to your use
File file = new File(fileName);
FileChannel channel = new RandomAccessFile(file, "rw").getChannel();
// Get an exclusive lock on the whole file
FileLock lock = channel.lock();
try {
lock = channel.tryLock();
// Ok. You get the lock
} catch (OverlappingFileLockException e) {
// File is open by other end
} finally {
lock.release();
}
I'm writing an application (for educational purposes), which needs to use database management system (I wrote my own extremely primitive DBMS, it is part of the task). And I want to ensure that at any time my application is running contents of all tables are correct. For that purposes I wrote method, which looks through each file and make necessary checks. The problem is that I want to call this method only once, when application starts and deny access to files to ensure that nobody changed their contents while my program is working.
I use the following approach. When application starts, I initialize InputStreamReader and OutputStreamWriter, store them and close them only when my application is terminated.
Part of initialization method:
FileInputStream fis = new FileInputStream(file);
FileOutputStream fos = new FileOutputStream(file, true);
InputStreamReader isr = new InputStreamReader(fis, "UTF-8");
OutputStreamWriter osw = new OutputStreamWriter(fos, "UTF-8");
this.tables.get(table).put("fis", fis);
this.tables.get(table).put("fos", fos);
this.tables.get(table).put("isr", isr);
this.tables.get(table).put("osw", osw);
Close method:
try {
for(Map<String, Object> table_map: tables.values()) {
OutputStreamWriter osw = (OutputStreamWriter)table_map.get("osw");
InputStreamReader isr = (InputStreamReader)table_map.get("isr");
if (osw != null)
osw.close();
if (isr != null)
isr.close();
}
}
catch (IOException e) {
throw new DBException("Closing error");
}
Partly, this approach works, because when I try to modify any of these files using MS Notepad, I get the following error
"The process cannot access the file because it is being used by
another process"
That's what I want to see. But if I use Notepad++, I can make any modifications when my application is running, that's not what I expect to see. So what can I do to ensure that no other process can modify my files?
I tried to use FileLock, but it denies access only for my process, if I'm not mistaken.
Sorry for my poor English, hope you will understand my question anyway.
I'm not sure this is a problem worth solving. Whatever approach you take, someone with the correct privileges can probably undo your file protection and could make changes anyway.
It is best to focus on gracefully handling invalid data and otherwise trusting what is in the file. Adding some kind of integrity check (per row or table) will make it harder for someone to accidentally or maliciously change your data in a way that leaves it looking "valid".
If you read the section "Platform dependencies" in the java.nio.channels.FileLock docsyou see that:
FileLocks are not (only) for locking inside one JVM but for all processes on the computer.
File locks (note the different spelling) are greatly platform and configuration specific.
So you basicyll have to ask yourself: What protection do I really need?
If you only want to guard against running your programm multiple times on the same data you can assume that your programm "behaves well" and
use FileLocks or
use a marker lock file or
use a "dirty/locked" marker inside the file
If you want to protect against every other program then you are lost as you have seen in the Notepad++ scenario: Considering all platforms and all possible ways to circumvent locks and using Java- you have no chance.
In my application I need to save some file (a pdf) to the filesystem. My current method involves creating a directory for storing the files:
FileConnection fc = (FileConnection)Connector.open("file:///SDCard/BlackBerry/pdfs/");
if (!fc.exists())
fc.mkdir();
fc.close();
I then write to the directory with my file:
fc = (FileConnection)Connector.open("file:///SDCard/BlackBerry/pdfs/" + filename, Connector.READ_WRITE);
if (!fc.exists())
fc.create();
OutputStream outStream = fc.openOutputStream();
outStream.write(pdf);
outStream.close();
fc.close();
This all works fine, and my pdf arrives in my created directory. My question is: will I run into trouble with the fact that I have hard coded a file path as my save destination. With the BlackBerry API is it possible to retrieve a writeable folder which exists on all models/configurations?
You can query the system for the available roots using FileSystemRegistry.listRoots(). Note that it is not guaranteed that there will be an sdcard, or that it will be visible even if there is one (when in mass storage mode, for instance). I think that the only root guaranteed to be on all devices is internal storage ("file:///Store").
There's (a little) more information here.