I have a java project that uses java.io.RandomAccessFile to manage data loading. It seeks through the file creating a map of key points which can then be loaded as needed later. This works great.
I want to make it run as an applet but it requires security permissions to create a temp file that a downloaded file could be stored in, and that's a huge barrier for it's intended usage.
I think I can spare the memory (a few MB) to store the contents in a memory buffer of some sort and then random access it in the same way I treat local files...
Is there a way to create a temp file without requiring security permissions (I assume not)?
What is the best buffering option? How would I get the contents of a URL based input stream into the buffer, read bytes from it, and be able to record and change the current seek position?
Check this discussion first: RandomAccessFile-like API for in-memory byte array?
Related
So far I managed to do something with Byte Stream : read the original file, and write in a new file while omitting the desired bytes (and then finish by deleting/renaming the files so that there's only one left).
I'd like to know if there's a way to directly modify the bytes without requiring to manipulate more than one file. The reason is because this has to be performed when there is low memory and the file is too big, so cloning the file before trimming it may not be the best option.
I'd like to know if there's a way to directly modify the bytes without requiring to manipulate more than one file.
There isn't a SAFE way to do it.
The unsafe way to do it involves (for example) mapping the file using a MappedByteBuffer, and shuffling the bytes around.
But the problem is that if something goes wrong while you are doing this, you are liable to end up with a corrupted file.
Therefore, if the user asks to perform this operation when the device's memory is too full to hold a second copy of the file, the best thing is to tell the user to "delete some files first".
The reason is because this has to be performed when there is low memory and the file is too big, so cloning the file before trimming it may not be the best option.
If you are primarily worried about "memory" on the storage device, see above.
If you are worried about RAM, then #RealSkeptic's observation is correct. You shouldn't need to hold the entire file in RAM at the same time. You can read, modify, write it a buffer at a time.
You can't remove bytes in the middle of the file without placing the rest of the file in memory. But you can replace bytes if it can help you.
If I have a property of an object which is a large String (say the contents of a file ~ 50KB to 1 MB, maybe larger), what is the practice around declaring such a property in a POJO? All I need to do is to be able to set a value from one layer of my application and transfer it to another without making the object itself "heavy".
I was considering if it makes sense to associate an InputStream or OutputStream to get / set the value, rather than reference the String itself - which means when I attempt to read the value of the contents, I read it as a stream of bytes, rather than a whole huge string loaded into memory... thoughts?
What you're describing depends largely on your anticipated use of the data. If you're delivering the contents in raw form, then there may be more efficient ways to manage it.
For example, if your app has a web interface, your app may just provide a URL for a web server to stream the contents to the requester. If it's a CLI-based app, you may be able to get away with a simple file copy. If your app is processing the file, however, then perhaps your POJO could retain only the results of that processing rather than the raw data itself.
If you wish to provide a general pattern along the lines of using POJO's with references to external streams, I would suggest storing in your POJO something akin to a URI that tells where to find the stream (like a row ID in a database or a filename or a URI) rather than storing an instance of the stream itself. In doing so, you'll reduce the number of open file handles, prevent potential concurrency issues, and will be able to serialize those objects locally if needed without having to duplicate the raw data persisted elsewhere.
You could have an object that supplies a stream or an iterator every time you access it. Note that the content has to live on some storage, like a file. I.e your object will store a pointer (e.g. a file path) to the storage and every time someone access it, you open a stream or create an iterator and let that party read. Note also that in order to save on memory, whoever consumes it has to make sure not to store the whole content in memory.
However, 50KB or 1MB is really tiny. Unless you have like gigabytes (or maybe hundred megabytes), I wouldn't try to do something like that.
Also, even if you have large data, it's often simpler to just use files or whatever storage you'll use.
tl;dr: Just use String.
I saw this nifty guide on how to do streaming file uploads via Apache Commons. This got me thinking where is the data stored? And is it necessary to "close" or "clean" that location?
Thanks!
where is the data stored?
I don't think it is stored.
The Streaming API doesn't use DiskFileItemFactory. But it does use a buffer for copying data as BalusC has posted.
Once you have the stream of the upload, you can use
long bytesCopied = Streams.copy(yourInputStream, yourOutputStream, true);
Look at the API
Here is the javadoc for DiskFileItemFactory.
The default FileItemFactory implementation. This implementation
creates FileItem instances which keep their content either in memory,
for smaller items, or in a temporary file on disk, for larger items.
The size threshold, above which content will be stored on disk, is
configurable, as is the directory in which temporary files will be
created.
If not otherwise configured, the default configuration values are as
follows:
Size threshold is 10KB.
Repository is the system default temp directory, as returned by System.getProperty("java.io.tmpdir").
Temporary files, which are created for file items, should be deleted
later on. The best way to do this is using a FileCleaningTracker,
which you can set on the DiskFileItemFactory. However, if you do use
such a tracker, then you must consider the following: Temporary files
are automatically deleted as soon as they are no longer needed. (More
precisely, when the corresponding instance of File is garbage
collected.) This is done by the so-called reaper thread, which is
started automatically when the class FileCleaner is loaded. It might
make sense to terminate that thread, for example, if your web
application ends. See the section on "Resource cleanup" in the users
guide of commons-fileupload.
So, yes close and cleanup are necessary, as FileItem may denote a real file on disk.
It's stored as a byte[] in the Java memory.
I am developing an application that is writing data to a file. Lets assume while it is writing the data we plug off the battery. What will happen to the file? will it be half-writen (corrupted), empty or same as before we wrote to it? My guess is that it will be corrupted. How to check if it is corrupted when we restart the phone when the file was storing an arraylist of objects? will java throw a corrupted file exception or say that the read arraylist is null or that it is an unknown object?
PS. maybe create another file that will keep the MD5 checksum of the data file? And whenever I write to the file data first I produce its checksum and then when I read from the data file produce a checksum and compare it with the previous. That will indicate whether my data are intact but it wont allow me to roll back to a previous state (pre-corrupted one). I would like a method that would be as lightweight as possible, I am already using the CPU too much by reading/writing changes to my storage on every attribute change of a set of thousands. Probably a database would have been a better idea.
I can't say how Java will read in a corrupted serialized array, but for safety let's assume that there's no error detection.
In that case, you have two easy options:
Store a checksum of your data inside your data structure, before you serialize it.
Compute the checksum of the final serialized file.
Either case will work the same way, though the first option might be a bit faster since you compute the checksum before you've written anything to disk (and therefor avoid an extra round of file I/O).
As you mentioned, MD5 would be fine for this. (Even a basic CRC would probably be fine -- you don't need a cryptograhpic hash for this.)
If you want to allow rolling back to a previous version -- I'd just store each version as a separate file and then have a pointer to the most recent one. (If you update the pointer as the last step of your write operation, this will also provide an extra level of protection against corrupt data being input to your app -- though you'll have to prepare for this pointer to be corrupt as well. Since this is essentially a commit step, you could interpret a corrupt pointer as "use the last version".)
And yes, at this point you might want to just use the SQLite functionality built into Android. :)
I need some way to store a configuration/status file that needs to be changed rapidly. The status of each key value pair (key-value) is stored in that file. The status needs to be changed rather too rapidly as per the status of a communication (Digital multimedia broadcasting) hardware.
What is the best way to go about creating such a file? ini? XML? Any off the shelf filewriter in Java? I can't use databases.
It sounds like you need random access to update parts of the file frequently without re-writing the entire file. Design binary file format and use RandomAccessFile API to read/write it. You are going to want to use fixed number of bytes for key and for value, such that you can index into the middle of the file and update the value without having to re-write all of the following records. Basically, you would be re-implementing how a database stores a table.
Another alternative is to only store a single key-value pair per file such that the cost of re-writing the file is minor. Maybe you can think of a way to use file name as the key and only store value in the file content.
I'd be inclined to try the second option unless you are dealing with more than a few thousand records.
The obvious solution would be to put the "configuration" information into a Properties object, and then use Properties.store(...) or Properties.storeToXML(...) to save to a file output stream or writer.
You also need to do something to ensure that whatever is reading the file will see a consistent snapshot. For instance, you could write to a new file each time and do a delete / rename dance to replace the the old with the new.
But if the update rate for the file is too high, you are going to create a lot of disc traffic, and you are bound slow down your application. This is going to apply (eventually) no matter what file format / API you use. So, you may want to consider not writing to a file at all.
At some point, configuration that changes too rapidly becomes "program state" and not configuration. If it is changing so rapidly, why do you have confidence that you can meaningfully write it to, and then read it from, a filesystem?
Say more about what the status is an who the consumer of the data is...