We Keep Coding

Java framework to manage BLOB data outside of database

I want to store my blobs outside of the database in files, however they are just random blobs of data and aren't directly linked to a file.
So for example I have a table called Data with the following columns:
id
name
comments
...
I can't just include a column called fileLink or something like that because the blob is just raw data. I do however want to store it outside of the database. I would love to create a file called 3.dat where 3 is the id number for that row entry. The only thing with this setup is that the main folder will quickly start to have a large number of files as the id is a flat folder structure and there will be OS file issues. And no the data is not grouped or structured, it's one massive list.
Is there a Java framework or library that will allow me to store and manage the blobs so that I can just do something like MyBlobAPI.saveBlob(id, data); and then do MyBlobAPI.getBlob(id) and so on? In other words something where all the File IO is handled for me?

Simply use an appropriate database which implements blobs as you described, and use JDBC. You really are not looking for another API but a specific implementation. It's up to the DB to take care of effective storing of blobs.

I think a home rolled solution will include something like a fileLink column in your table and your api will create files on the first save and then write that file on update.
I don't know of any code base that will do this for you. There are a bunch that provide an in memory file system for java. But it's only a few lines of code to write something that writes and reads java objects to a file.
You'll have to handle any file system limitations yourself. Though I doubt you'll ever burn through the limitations of modern file systems like btrfs or zfs. FAT32 is limited to 65K files per directory. But even last generation file systems support something on the order of 4 billion files per directory.
So by all means, write a class with two functions. One to serialize an object to a file; given it a unique key as a name. And another to deserialize the object by that key. If you are using a modern file system, you'll never run out of resources.

As far as I can tell there is no framework for this. The closest I could find was Hadoop's HDFS.
That being said the advice of just putting the BLOB's into the database as per the answers below is not always advisable. Sometimes it's good and sometimes it's not, it really depends on your situation. Here are a few links to such discussions:
Storing Images in DB - Yea or Nay?
https://softwareengineering.stackexchange.com/questions/150669/is-it-a-bad-practice-to-store-large-files-10-mb-in-a-database
I did find some addition really good links but I can't remember them offhand. There was one in particular on StackOverFlow but I can't find it. If you believe you know the link please add it in the comments so that I can confirm it's the right one.

How would I create my own zip like file?

I want to be able to create a file that'll act like a zip but at the same time it isn't an actual zip.
Let's say I have a program that'll take a bunch of files and directories and store them into a single file with a name and extention of data.rds and you would need the same program to extract them out of it. I've seen in lots of different games that they use file formats such as .arc, .nsa, .mxdl etc which all store many files inside of them, .rar is probably the most commonly known format. The four extentions can't be opened as a normal zip and require a specific program in order to extract the files from them, I want to learn as to how you would encrypt and decrypt many files into a single one without making it readable like it would be in a normal zip file.
Pretty much how would one go about doing this? I know it would be a long process and won't be answered with a few simple lines of code but if someone could point me in a direction towards learning as to how to do such a thing, that would help helpful.

No matter what format you invent, someone will figure it out. Anyone can decompile your code and see your algorithm.
I would just use the Zip format and give the file a different extension (which it sounds like you're already doing). An easy way to keep casual observers from opening your file is to put a couple junk bytes at the front of it:
private static final byte[] secretSignature = { 10, 20 };
void writeData(Path file)
throws IOException {
try (OutputStream out = new BufferedOutputStream(
Files.newOutputStream(file))) {
out.write(secretSignature);
ZipOutputStream zip = new ZipOutputStream(out);
// Write zip entries
zip.finish();
}
}
void readData(Path file)
throws IOException {
try (InputStream in = new BufferedInputStream(
Files.newInputStream(file))) {
in.skip(secretSignature.length);
ZipInputStream zip = new ZipInputStream(in);
ZipEntry entry;
while ((entry = zip.getNextEntry()) != null) {
// Read entry
}
}
}

You could approach it like this:
1) start with an application that does "simply" store the contents of directories, list of files, ... in a single file. Meaning: learn how to collect all these files; and how to push them into a single uncompressed archive (and of course: ensure that you can extract things afterwards again)
2) when that step is working (and properly and extensively tested); then add a "compression" resp. "decompression" step.
Your favorite search engine will give you many results when searching for "compression algorithms".

It depends on your goal.
I'm going to assume you wish to write your own algorithm for fun.
If you just want to pack things together and encrypt them, well, just take the files you need and write their binary content in a sequential manner, prepending at the start of the file something like an index table, that tells you where in the big-file each file starts. Then encrypt everything using your algorithm of choice.
If you want to also compress them, the simplest algorithm I feel suggesting you to implement is Huffman encoding of your binary content. Note that, while simple enough in theory, it can still be quite an ordeal to implement, so think carefully if it's worth it or if you can rely on something off-the-shelf.
Bottom line: if you are doing it to teach yourself something, go for it. If you need it in a bigger project where the end goal isn't learning these things, just take something that already exists.

I sense that you are more concerned about authenticity, that is, that the archive is not modified. I will further assume that you don't really want to implement your own compression algorithms.
That being said, what you could is the following:
Create a zip with different extension.
Compute the SHA1 hash of the file
Use the SHA1 hash to check if that archive hasn't been changed.

Reading and editing .wav (or flac) details in Java

I have tons of ripped .wav files (I'm ready to convert them into flacs if it's easier) which details I want to insert in a MySQL database. When I right click the .wav files in Windows Explorer (not the browser) and select Properties -> Details I can see some details about the song. For example the artist, genre and duration. How can I read and edit these details in Java?

To get durration information, see this link: Java - reading, manipulating and writing WAV files
Essentially, a WAV file is broken up into chunks, which either contain audio data, or describe the audio data in some way, or provide information about it. If the reader doesn't understand one of those chunks it is able to skip it, which allows placing a lot of different kinds of information in the file. One of those chunks contains information like the samplerate, number of channels and total number of sample frames, from which you can calculate the length.
For artist, genre and so on... well there's no standard chunk for that, so if that's really in the file, and not in the windows db somewhere, it's probably stored in ID3 tags embedded in the WAV. I don't know for sure what the chunkID is for ID3, but it's probably "id3 ", or "ID3 " (including the space). You coud probably figure this out by searching for strings of length 4 or more in the file -- usually data chunks are in the beginning and audio is at the end. (on unix/macos I would use the "strings" command, maybe with "head") ID3 tags are standard for MP3, and you can figure out how to parse them by googling. To get to them, you'll need to understand WAV files first, at least enough to know what chunks are, chunkIds, how to skip chunks you don't care about, and so on.
I don't know of a library that will read ID3 tags in WAV files in Java, so you'll either have to write one, or wrap one written in another language. I suspect libsndfile will work, but it doesn't have an MP3 reader, so maybe not. You could also try SOX. You can also check out http://javamusictag.sourceforge.net/ which I've never used, but it came up in a search.
good luck!

I ended up converting them into flac files and using JAudiotagger. Thanks for the responses, this time I ended up this way.
http://www.jthink.net/jaudiotagger/

Creating an uneditable data file in Java

I am currently writing a program which takes user input and creates rows of a comma delimited .csv file. I am in need of a way to save this data in a way in which users are not able to easily edit this data. It does not need to be super secure, just enough so that it couldn't accidentally be edited. I also need another file (or the same file?) created to then be easily accessible (in the file system) by the user so that they may then email this file to a system admin who can then open the .csv file. I could provide this second person with a conversion program if necessary.
The file I save data in and the file to be sent can be two different files if there are any advantages to this. I was currently considering just using a file with a weird file extension, but saving it as a text file so that the user will only be able to open it if they know to try that. The other option being some sort of encryption, but I'm not sure if this is necessary and even if it was where I would start.
Thanks for the help :)
Edit: This file is meant to store the actual data being entered. Currently the data is being gathered on paper forms which are then sent to the admin to manually enter all of the data. This little app is meant to have someone else enter the data from the paper form and then tell them if they've entered it all correctly. After they've entered it all they then need to send the data to the admin. It would be preferable if the sending was handled automatically, but this app needs to be very simple and low budget and I don't want an internet connection to be a requirement.

You could store your data in a serializable object and save that. It would resist casual editing and be very simple to read and write from your app. This page should get you started: http://java.sun.com/developer/technicalArticles/Programming/serialization/

From your question, I am guessing that the uneditable file's purpose is to store some kind of system config and you don't want it to get messed up easily. From your own suggestions, it seems that even knowing that the file has been edited would help you, since you can then avoid using it. If that is the case, then you can use simple checks, such as save the total number of characters in the line as the first or last comma delimited value. Then, before you use the file, you just run a small validation code on it to verify that the file is indeed unaltered.

Another approach may just be to use a ZIP (file) of a "plain text format" (CSV, XML, other serialization method, etc) and, optionally, utilize a well-known (to you) password.
This approach could be used with other stream/package types: the idea behind using a ZIP (as opposed to an object serializer directly) is so that one can open/inspect/modify said data/file(s) easily without special program support. This may or may not be a benefit and using a password may or may not even be required, see below.
Some advantages of using a ZIP (or CAB):
The ability for multiple resources (aids in extensibility)
The ability to save the actual data in a "text format" (XML, perhaps)
Maintain competitive file-sizes for "common data"
Re-use existing tooling support (also get checksum validation for free!)
Additionally, using a non-ZIP file extension will prevent most users from casually associating the file (a similar approach to what is presented in the original post, but subtly different because the ZIP format itself is not "plain text") with the ZIP format and being able to open it. A number of modern Microsoft formats utilize the fact that the file-extension plays an important role and use CAB (and sometimes ZIP) formats as the container format for the document. That is, an ".XSN" or ".WSP" or ".gadget" file can be opened with a tool like 7-zip, but are generally only done so by developers who are "in the know". Also, just consider ".WAR" and ".JAR" files as other examples of this approach, since this is Java we're in.
Traditional ZIP passwords are not secure, and more-so is using a static password embedded in the program. However, if this is just a deterrent (e.g. not for "security") then those issues are not important. Coupled with an "un-associated" file-type/extension, I believe this offers the protection asked for in the question while remaining flexible. It may be possible to entirely drop the password usage and still prevent "accidental modifications" just by using a ZIP (or other) container format, depending upon requirement/desires.
Happy coding.

Can you set file permissions to make it read-only?

Other than doing a binary output file, the file system that Windows runs (I know for sure it works from XP through x64 Windows 7) has a little trick that you can use to hide data from anyone simply perusing through your files:
Append your output and input files with a colon and then an arbitrary value, eg if your filename is "data.csv", make it instead "data.csv:42". Any existing or non-existing file can be appended to to access a whole hidden area (and every file for every value after the colon is distinct, so "data.csv:42" != "data.csv:carrots" != "second.csv:carrots").
If this file doesn't exist, it will be created and initialized to have 0 bytes of data with it. If you open up the file in Notepad you will indeed see that it holds exactly the data it held before writing to the :42 file, no more, no less, but in reality subsequent data read from this "data.csv:42" file will persist. This makes it a perfect place to hide data from any annoying user!
Caveats: If you delete "data.csv", all associated hidden data will be deleted too. Also, there are indeed programs that will find these files, but if your user goes through all that trouble to manually edit some csv file, I say let them.
I also have no idea if this will work on other platforms, I've never thought to try it.

Java: Where can I find advanced file manipulation source/libraries?

I'm writing arbitrary byte arrays (mock virus signatures of 32 bytes) into arbitrary files, and I need code to overwrite a specific file given an offset into the file. My specific question is: is there source code/libraries that I can use to perform this particular task?
I've had this problem with Python file manipulation as well. I'm looking for a set of functions that can kill a line, cut/copy/paste, etc. My assumptions are that these are extremely common tasks, and I couldn't find it in the Java API nor my google searches.
Sorry for not RTFM well; I haven't come across any information, and I've been looking for a while now.

Maybe you are looking for something like the RandomAccessFile class in the standard Java JDK. It supports reads and writes at some offset, as well as byte arrays.

Java's RandomAccessFile is exactly what you want.
It includes methods like seek(long) that allow you to move wherever you need in the file. It also allows for reading and writing at the same time.

As far as I know, Java has primarily lower level functions for manipulating files directly. Here is the best I've come up with
The actions you describe are standard in the Swing world, and for text comes down to manipulating a Document object. These act on data in memory. The class java.nio.channels.FileChannel has similar methods that act directly on a file. Neither fine the end of lines automatically, but other classes in java.io and java.nio do.
Apache Commons has a sandbox library called Flatfile which looks like it does what you want. The problem is that no code has been released yet. You may, however, want to talk to people working on it to get some more ideas. I didn't do a general check on libraries.

Have you looked into File/FileReader/FileWriter/BufferedReader? You can get the contents of the files and manipulate it as you like, you can search the data in the files, you can overwrite files, create new, append to an existing....
I am not sure this is exactly what you are asking for but I use these APIs all the time for logging, RTF editors, text file creation for email, and many other things.
As far as cut/copy/past goes, I have not come across the ability to do that directly, however, you can output the contents of the file and "copy" what part of it you want and "paste" it into a new file, or append it to an existing.

While writing a byte array to a file is a common task, writing to a give file 32-bytes byte array just once is just not something you are going to find in java.io :)
To get started, would the below method and comments look reasonable to you? I bet someone here, maybe even myself, could whip it out quick like.
public static void writeFauxVirusSignature(File file, byte[] bytes, long offset) {
//open file
//move to offset
//write bytes
//close file
}
Questions:
How big could the potential target files be?
Do you need performance?
I ask because clean, easy to read code would use Apache Commons lib's, but large file writes in a performance sensitive environment will necessitate using java.nio libraries