I want to be able to create a file that'll act like a zip but at the same time it isn't an actual zip.
Let's say I have a program that'll take a bunch of files and directories and store them into a single file with a name and extention of data.rds and you would need the same program to extract them out of it. I've seen in lots of different games that they use file formats such as .arc, .nsa, .mxdl etc which all store many files inside of them, .rar is probably the most commonly known format. The four extentions can't be opened as a normal zip and require a specific program in order to extract the files from them, I want to learn as to how you would encrypt and decrypt many files into a single one without making it readable like it would be in a normal zip file.
Pretty much how would one go about doing this? I know it would be a long process and won't be answered with a few simple lines of code but if someone could point me in a direction towards learning as to how to do such a thing, that would help helpful.
No matter what format you invent, someone will figure it out. Anyone can decompile your code and see your algorithm.
I would just use the Zip format and give the file a different extension (which it sounds like you're already doing). An easy way to keep casual observers from opening your file is to put a couple junk bytes at the front of it:
private static final byte[] secretSignature = { 10, 20 };
void writeData(Path file)
throws IOException {
try (OutputStream out = new BufferedOutputStream(
Files.newOutputStream(file))) {
out.write(secretSignature);
ZipOutputStream zip = new ZipOutputStream(out);
// Write zip entries
zip.finish();
}
}
void readData(Path file)
throws IOException {
try (InputStream in = new BufferedInputStream(
Files.newInputStream(file))) {
in.skip(secretSignature.length);
ZipInputStream zip = new ZipInputStream(in);
ZipEntry entry;
while ((entry = zip.getNextEntry()) != null) {
// Read entry
}
}
}
You could approach it like this:
1) start with an application that does "simply" store the contents of directories, list of files, ... in a single file. Meaning: learn how to collect all these files; and how to push them into a single uncompressed archive (and of course: ensure that you can extract things afterwards again)
2) when that step is working (and properly and extensively tested); then add a "compression" resp. "decompression" step.
Your favorite search engine will give you many results when searching for "compression algorithms".
It depends on your goal.
I'm going to assume you wish to write your own algorithm for fun.
If you just want to pack things together and encrypt them, well, just take the files you need and write their binary content in a sequential manner, prepending at the start of the file something like an index table, that tells you where in the big-file each file starts. Then encrypt everything using your algorithm of choice.
If you want to also compress them, the simplest algorithm I feel suggesting you to implement is Huffman encoding of your binary content. Note that, while simple enough in theory, it can still be quite an ordeal to implement, so think carefully if it's worth it or if you can rely on something off-the-shelf.
Bottom line: if you are doing it to teach yourself something, go for it. If you need it in a bigger project where the end goal isn't learning these things, just take something that already exists.
I sense that you are more concerned about authenticity, that is, that the archive is not modified. I will further assume that you don't really want to implement your own compression algorithms.
That being said, what you could is the following:
Create a zip with different extension.
Compute the SHA1 hash of the file
Use the SHA1 hash to check if that archive hasn't been changed.
Related
I implement a file manipulation functionality, and I paid attention that Java provides multiple techniques to copy and move files. Below you can find code snippets, briefly describing these approaches:
Approach #1:
File from = new File(src.getPath());
File to = new File(dst.getPath());
from.renameTo(to);
Approach #2:
FileChannel inChannel = new FileInputStream(src).getChannel();
FileChannel outChannel = new FileOutputStream(dst).getChannel();
inChannel.transferTo(0, inChannel.size(), outChannel);
Approach #3:
InputStream in = getContentResolver().openInputStream(selectedImageUri);
OutputStream out = new FileOutputStream("/sdcard/wallpapers/" + wall);
byte[] buffer = new byte[1024];
int read;
while ((read = in.read(buffer)) != -1) {
out.write(buffer, 0, read);
}
Approach #4:
import static java.nio.file.StandardCopyOption.*;
Files.copy(source, target, REPLACE_EXISTING);
All these approaches work, but I can't grasp when should I use each of them? What are the pros and cons of each of these methods, especially from the performance and the reliability points of view? Is there any specific scenario when I have to prefer one technique over another?
It is already discussed enough here and the following is from here
Your first approach is File rename that has nothing to do with File copy
java.io.File class doesn’t have any shortcut method to copy file from source to destination.
1. Using Stream: This is the conventional way of file copy in java, here we create two Files, source and destination. Then we create InputStream from source and write it to destination file using OutputStream.
2. Using java.nio.channels.FileChannel: Java NIO classes were introduced in Java 1.4 and FileChannel can be used to copy file in java. According to transferFrom() method javadoc, this way of copy file is supposed to be faster than using Streams to copy files.
3. Using Apache Commons IO: Apache Commons IO FileUtils.copyFile(File srcFile, File destFile) can be used to copy file in java. If you are already using Apache Commons IO in your project, it makes sense to use this for code simplicity. Internally it uses Java NIO FileChannel, so you can avoid this wrapper method if you are not already using it for other functions.
4. Java 7 Files class: If you are working on Java 7, you can use Files class copy() method to copy file in java. It uses File System providers to copy the files.
Now to see which one of these methods is more efficient we will copy a large file[1 GB] using each one of them in a simple program. To avoid any performance speedups from caching we are going to use four different source files and four different destination files.{Refer code in link}
Time taken by FileStreams Copy = 127572360
Time taken by FileChannels Copy = 10449963
Time taken by Java7 Files Copy = 10808333
Time taken by Apache Commons IO Copy = 17971677
From the output it’s clear that Stream Copy is the best way to copy File in Java. FileChannels is the best way to copy large files. If you work with even larger files you will notice a much bigger speed difference
We can divide your four approaches into two types:
Use a built-in standard library method (such as File.renameTo() and Files.move()).
Do the work ourselves - by copying bytes from source to target.
First, note that File doesn't have a copy method, so you only have one option for built-in, standard library method when you're talking about copy.
Also note that "do the work ourselves" when renaming is going to be very bad - you're going to copy the entire file, then delete the old file. Not a good or efficient approach. In most cases, renaming/moving within the same filesystem requires just changing file metadata without actually touching the content, so it's really a lot better to use a standard library.
So you have two cases:
Renaming
The options are really using either File.renameTo() or Files.move(). No point in using streams and copying data.
File is an outdated class. It shouldn't really be used anymore. There is an excellent explanation why, which sums up to the fact that File doesn't give you any information when any of its standard methods fail, whereas Files provides you with very accurate exceptions when that happens.
Copying
You have two choices - either use Files.copy() or one of the "do it yourself" approaches.
By far, if what you are copying are actual files, your choice should be Files.copy(). There is no need to re-invent the wheel. It does exactly what you want, is well documented, you're not likely to introduce bugs accidentally. And yes, it's very efficient.
Files.copy() relies on underlying "providers" for its operation. What it means is that there are specialized vendor (or operating system) specific classes that do the operation that is the most efficient for that filesystem. Whether it's a Linux filesystem or a Windows one, the copy will be optimized for it. There are even providers for specialized cases, such as zip files, so you can copy files inside a zip, jar or war file using Files.copy() - which is a lot more complicated if you try the "do it yourself" approach.
Besides, Files.copy() checks lots of things that you might forget when you write "your own" copy. For example, did you remember to check that the file that you are reading from and the file you are writing to are not the same file? It could cause serious trouble. Files.copy() does it. It checks permissions, it checks if the target of the copy is a directory, and so on. So it's very reliable.
So why do you have the option to do "your own"? Because well, Java is a general-purpose language. You have the option to read from a file, the option to write to a file, so you can write your own "copy" method. That doesn't mean you should.
Note that in your "approach #3", the "source" file is not actually a file! It's produced from an Image URI, which means it could be a network source. When your source is not a file, but a stream or channel based on a socket, database BLOB, web server request etc., you can't really use Files.copy(). This is where you'd need to write your own.
Actually, Files also has options for copying from a file to an OutputStream or from an InputStream to a file, so if one side of the copy is a stream and the other a file, you can use that. It will be readable, safe, and throw meaningful exceptions.
So write your own copy:
when you need to move data from sources to targets which are not files,
when you need to filter or process the data somehow rather than copy as-is from source to target,
when you are using old versions of Java, prior to 1.7. In this case, channels would probably be better than streams.
I have a lot of files in a directory but I only want to read the ones with a certain extension (say .txt). I want these files added to the same BufferedInputStream so that I can read them in one go. When I call read() at the end of a file, the next one should begin.
It really feels like there should be an obvious answer to this but I had no luck finding it.
You might want to take a look at SequenceInputStream:
A SequenceInputStream represents the logical concatenation of other
input streams. It starts out with an ordered collection of input
streams and reads from the first one until end of file is reached,
whereupon it reads from the second one, and so on, until end of file
is reached on the last of the contained input streams.
To me the "obvious answer" is:
Just iterate through all the files in the directory using a proper filter. For each file create a FileInputStream, read it and close it.
I don't think there is an obvious answer to this question.
Probably you need to create a Wrapper InputStream with a list of files you want to read from. Internally you will open/close streams as needed, namely when a file is completely read.
It is not obvious but should not be difficult. This way you can work 'only' with one InputStream for all files.
Ok so I know the value of the line, I dont have the line number, how would I edit only 1 line?
Its a config file, i.e
x=y
I want a command to edit x=y to x=y,z.
or even x=z.
In Java you can use `Properties class:
app.config file:
x=y
java:
public void writeConfig() throws Exception {
Properties tempProp = new Properties();
tempProp.load(new FileInputStream("app.config"));
tempProp.setProperty("x", "y,z");
tempProp.store(new FileOutputStream("app.config"), null);
}
If you are using that configuration format, you might want to use
java.util.Properties
component to read/write on that file.
But if you just want to edit it by hand, you can just read the file line by line and match the variable you want to change.
One way to do it is to:
Read the file into memory; e.g. as an array of Strings representing the lines of the file.
Locate the String/line you want to change.
Use a regex (or whatever) to modify the String/line
Write a new version of the file from the in memory version.
There are many variations on this. You also need to take care when you write the new version of the file to guard against losing everything if something goes wrong during the write. (Typically you write the new version to a temporary file, rename the old version out of the way (e.g. as a backup) and rename the new version in place of the old one.)
Unfortunately, there is no way to add or remove characters in the middle of a regular text file without rewriting a large part of the file. This "problem" is not specific to Java. It is fundamental to the way that text files are modelled / represented on most mainstream operating systems.
Unless the new line has the exact same length as the old one, your best bet is to
Open a temporary output file
Read the config file, line by line
Search for your key
If you can't find it, just write the line you just read to the output file
If you can find it, write the new value to the temporary file instead
Until you hit EOF
Delete old file
Rename new file to the old file
IF your config file is small, you can also do the whole parsing/modification step in memory and then write the final result back to the config file, that way you skip the temporary file (although a temporary file is a good way to prevent corruption if something breaks while you write the file).
If this is not what you're looking for, you should edit your question to be a lot more clear. I'm just guessing what you're asking for.
If your data is all key and value pairs, for example ...
key1=value1
key2=value2
... then load them into a Properties object. Off the top of my head, you'll need a FileInputStream to load the properties, modify with myProperties.put(key, value) and then save the properties with the use of a FileOutputStream.
Hope this helps!
rh
In my program i want the user to be able to take some images from a directory, and save them under a single file, that can be transferred to another computer possibly, and actually read and displayed(using the same program).
How would i go about doing this, especially if i want to save other data along with it, perhaps objects and such. I know you can use the ObjectOutputStream class, but im not sure how to integrate it with images.
So overall, i want the program to be able to read/write data, objects, and images to/from a single file.
Thanks in Advance.
[EDIT - From Responses + Comment regarding Zip Files]
A zip might be able to get the job done.
But i want it to be read only be the program. ( You think making it a zip, changing the file extension would work, then when reading it just chaing it back and reading as a zip?? ) I dont want users to be able to see the contents directly.
Ill elaborate a bit more saying its a game, and users can create their own content using xml files, images and such. But when a user creates something i dont want other users to be able to see exactly how they created it, or what they used, only the end result.
You can programatically create a zip file, and read a zip file from Java, no need to expose it as a regular .zip file.
See: java.io.zip pacakge for more information, and these others for code samples on how to read/write zip using java.
Now if you want to prevent the users from unzipping this file, but you don't want to complicate your life by encrypting the content, or creating a complex format, you can emulate a simple internet message format, similar to the one used for e-mails to attach files.
You can read more about the internet message format here
This would be a custom file format only used by your application so you can do it as simple as you want. You just have to define your format.
It could be:
Header with the names ( and number ) of files in that bundle.
Followed by a list of separators ( for instance limit.a.txt=yadayada some identifier to know you have finished with that content )
Actual content
So, you create the bundle with something like the following:
public void createBundle() {
ZipOutputStream out = ....
writeHeader( out );
writeLimits( out yourFiles );
for( File f : youFiles ) {
writeFileTo( f, out );
}
out.close();
}
Sort of...
And the result would be a zipped file with something like:
filenames =a.jpg, b.xml, c.ser, d.properties, e.txt
limits.a.jpg =poiurqpoiurqpoeiruqeoiruqproi
limits.b.xml =faklsdjfñaljsdfñalksjdfa
limit.s.ser =sdf09asdf0as9dfasd09fasdfasdflkajsdfñlk
limit.d.properties =adfa0sd98fasdf90asdfaposdifasdfklasdfkñm
limit.e.txt =asdf9asdfaoisdfapsdfñlj
attachments=
<include binary data from a.jpg here>
--poiurqpoiurqpoeiruqeoiruqproi
<include binary data from b.xml here>
--faklsdjfñaljsdfñalksjdfa
etc
Since is your file format you can keep it as simple as possible or complicate your life at infinitum.
If you manage to include a MIME library in your app, that could save you a lot of time.
Finally if you want to add extra security, you have to encrypt the file, which is not that hard after all, the problems is, if you ship the encrypting code too, your users could get curious about it and decompile them to find out. But a good encrypting mechanism would prevent this.
So, depending on your needs you can go from a simple zip, a zip with a custom format, a zip with a complicated customformat or a zip with a custom complicated encrypted format.
Since that's to broad you may ask about specific parts here: https://stackoverflow.com/questions/ask
In your case I would use a ZIP library to package all the images in a ZIP file. For the metadata you want to save along with these, use XML files. XML and ZIP are quite a de-facto standard today, simple to handle and though flexible if you want to add new files or metadata. There are also serializing tools to serialize your objects into XML. (I don't know them exactly in Java, but I'm sure there are.)
Yep, just pack/unpack them with java.util.zip.* which is pretty straightforward to go. Every Windows Version since XP has built in zip support, so your good to go. There are many good (and faster) free zip libraries for java/c#, too.
I know you can use the ObjectOutputStream class, but im not sure how to integrate it with images.
Images are binary data, so reading it into a byte[] and writing the byte[] to ObjectOutputStream should work. It's however only memory hogging since every byte eats at least one byte of JVM's memory. You'll need to take this into account.
I'm writing arbitrary byte arrays (mock virus signatures of 32 bytes) into arbitrary files, and I need code to overwrite a specific file given an offset into the file. My specific question is: is there source code/libraries that I can use to perform this particular task?
I've had this problem with Python file manipulation as well. I'm looking for a set of functions that can kill a line, cut/copy/paste, etc. My assumptions are that these are extremely common tasks, and I couldn't find it in the Java API nor my google searches.
Sorry for not RTFM well; I haven't come across any information, and I've been looking for a while now.
Maybe you are looking for something like the RandomAccessFile class in the standard Java JDK. It supports reads and writes at some offset, as well as byte arrays.
Java's RandomAccessFile is exactly what you want.
It includes methods like seek(long) that allow you to move wherever you need in the file. It also allows for reading and writing at the same time.
As far as I know, Java has primarily lower level functions for manipulating files directly. Here is the best I've come up with
The actions you describe are standard in the Swing world, and for text comes down to manipulating a Document object. These act on data in memory. The class java.nio.channels.FileChannel has similar methods that act directly on a file. Neither fine the end of lines automatically, but other classes in java.io and java.nio do.
Apache Commons has a sandbox library called Flatfile which looks like it does what you want. The problem is that no code has been released yet. You may, however, want to talk to people working on it to get some more ideas. I didn't do a general check on libraries.
Have you looked into File/FileReader/FileWriter/BufferedReader? You can get the contents of the files and manipulate it as you like, you can search the data in the files, you can overwrite files, create new, append to an existing....
I am not sure this is exactly what you are asking for but I use these APIs all the time for logging, RTF editors, text file creation for email, and many other things.
As far as cut/copy/past goes, I have not come across the ability to do that directly, however, you can output the contents of the file and "copy" what part of it you want and "paste" it into a new file, or append it to an existing.
While writing a byte array to a file is a common task, writing to a give file 32-bytes byte array just once is just not something you are going to find in java.io :)
To get started, would the below method and comments look reasonable to you? I bet someone here, maybe even myself, could whip it out quick like.
public static void writeFauxVirusSignature(File file, byte[] bytes, long offset) {
//open file
//move to offset
//write bytes
//close file
}
Questions:
How big could the potential target files be?
Do you need performance?
I ask because clean, easy to read code would use Apache Commons lib's, but large file writes in a performance sensitive environment will necessitate using java.nio libraries