I have a binary file which contains keys and after every key there is an image associated with it. I want to jump off different keys but could not find any method which changes the index positioning in input stream. I have seen the mark() method but it does not jump on different places.
Does anybody have any idea how to do that?
There's a long skip(long n) method that you may be able to use:
Skips over and discards n bytes of data from this input stream. The skip method may, for a variety of reasons, end up skipping over some smaller number of bytes, possibly 0. This may result from any of a number of conditions; reaching end of file before n bytes have been skipped is only one possibility. The actual number of bytes skipped is returned. If n is negative, no bytes are skipped.
As documented, you're not guaranteed that n bytes will be skipped, so doublecheck the returned value always. Note that this does not allow you to "skip backward", but if it's markSupported(), you can reset() first and then skip forward to an earlier position if you must.
Other options
You may also use java.io.RandomAccessFile, which as the name implies, permits random access with its seek(long pos) method.
You mentioned images, so if you are using Java Advanced Imaging, another possible option is com.sun.media.jai.codec.FileSeekableStream, which is a SeekableStream that takes its input from a File or RandomAccessFile. Note that this class is not a committed part of the JAI API. It may be removed or changed in future releases of JAI.
Related
Can read file from start index to end index.
Files.lines(Paths.get("file.csv")).skip(1000000).limit(1000).forEach(s-> {});
But it isn't performance. It is possible to read csv file performance from middle of file?
Would java RandomAccessFile class methods help? Something like seek, skipBytes, etc. You can find tutorials.
It depends on how predictable it is, and on what exactly you mean by "middle", and what you consider to be "read from middle".
If the "middle" has to be an exact line, then you will have to read all the bytes before that middle, because otherwise you can miss on bytes that end lines (and the only way, with a CSV file, of knowing where line N is, is to have read exactly N-1 end-of-line characters until arriving at that position). Having to read all bytes up to a point is linear in time, and is certainly not as fast as actually jumping there in 1 go - but it may count as "reading from middle" for you.
If the file is highly predictable (all lines have approximately the same length), and you do not care much about getting exactly at the middle, then you can always take the length of the file, L, and jump to the last position before position L/2 which contains a newline character. The next position is, with high probability (since your file is predictable), the "middle line".
Introduction
We store tuples (string,int) in a binary file. The string represents a word (no spaces nor numbers). In order to find a word, we apply binary search algorithm, since we know that all the tuples are sorted with respect to the word.
In order to store this, we use writeUTF for the string and writeInt for the integer. Other than that, let's assume for now there are no ways to distinguish between the start and the end of the tuple unless we know them in advance.
Problem
When we apply binary search, we get a position (i.e. (a+b)/2) in the file, which we can read using methods in Random Access File, i.e. we can read the byte at that place. However, since we can be in the middle of the word, we cannot know where this words starts or finishes.
Solution
Here're two possible solutions we came up with, however, we're trying to decide which one will be more space efficient/faster.
Method 1: Instead of storing the integer as a number, we thought to store it as a string (using eg. writeChars or writeUTF), because in that case, we can insert a null character in the end of the tuple. That is, we can be sure that none of the methods used to serialize the data will use the null character, since the information we store (numbers and digits) have higher ASCII value representations.
Method 2: We keep the same structure, but instead we separate each tuple with 6-8 (or less) bytes of random noise (same across the file). In this case, we assume that words have a low entropy, so it's very unlikely they will have any signs of randomness. Even if the integer may get 4 bytes that are exactly the same as those in the random noise, the additional two bytes that follow will not (with high probability).
Which of these methods would you recommend? Is there a better way to store this kind of information. Note, we cannot serialize the entire file and later de-serialize it into memory, since it's very big (and we are not allowed to).
I assume you're trying to optimize for speed & space (in that order).
I'd use a different layout, built from 2 files:
Interger + Index file
Each "record" is exactly 8 bytes long, the lower 4 are the integer value for the record, and the upper 4 bytes are an integer representing the offset for the record in the other file (the characters file).
Characters file
Contiguous file of characters (UTF-8 encoding or anything you choose). "Records" are not separated, not terminated in any way, simple 1 by 1 characters. For example, the records Good, Hello, Morning will look like GoodHelloMorning.
To iterate the dataset, you iterate the integer/index file with direct access (recordNum * 8 is the byte offset of the record), read the integer and the characters offset, plus the character offset of the next record (which is the 4 byte integer at recordNum * 8 + 12), then read the string from the characters file between the offsets you read from the index file. Done!
it's less than 200MB. Max 20 chars for a word.
So why bother? Unless you work on some severely restricted system, load everything into a Map<String, Integer> and get a few orders of magnitude speed up.
But let's say, I'm overlooking something and let's continue.
Method 1: Instead of storing the integer as a number, we thought to store it as a string (using eg. writeChars or writeUTF), because in that case, we can insert a null character
You don't have to as you said that your word contains no numbers. So you can always parse things like 0124some456word789 uniquely.
The efficiency depends on the distribution. You may win a factor of 4 (single digit numbers) or lose a factor of 2.5 (10-digit numbers). You could save something by using a higher base. But there's the storage for the string and it may dominate.
Method 2: We keep the same structure, but instead we separate each tuple with 6-8 (or less) bytes of random noise (same across the file).
This is too wasteful. Using four zeros between the data byte would do:
Find a sequence of at least four zeros.
Find the last zero.
That's the last separator byte.
Method 3: Using some hacks, you could ensure that the number contains no zero byte (either assuming that it doesn't use the whole range or representing it with five bytes). Then a single zero byte would do.
Method 4: As disk is organized in blocks, you should probably split your data into 4 KiB blocks. Then you can add some time header allowing you quick access to the data (start indexes for the 8th, 16th, etc. piece of data). The range between e.g., the 8th and 16th block should be scanned sequentially as it's both simpler and faster than binary search.
I am reading from an Inputstream with a bounded buffer of 200 bytes and I want to find a substring in it. I used the string.indexOf(substring).
But it does not return the right answer if substring crosses the boundary. e.g starts from 199th byte.
Any suggestions?
There are two approaches that I can think of:
Normalize the circular buffer (*) before executing indexOf(). By "normalize" I mean copy the bytes within the buffer so that the beginning of the buffer is at index 0, and therefore the contents of the buffer are not circular anymore. This will greatly improve the performance of searching through the buffer, but it will incur a performance penalty on the first search that follows a modification of the buffer, because at that moment you will have to first normalize. Since you are only dealing with a 200 byte buffer, the penalty will be negligible, and if you are planning to do multiple searches per buffer modification, the savings might end up being huge.
Write your own indexOf( MyCircularBuffer, String ) method which searches inside your circular buffer for the first character of the string and when found, performs the comparison of the remainder of the string by generating indexes based on the same logic that your circular buffer uses for generating indexes.
* We are writing software for computers with finite memory, so every single buffer is by definition a bounded buffer, so the term "bounded buffer" does not convey any useful information either with respect to how you are supposed to use it, or with respect to how it is internally structured. What you are referring to as "Bounded Buffer" is in fact a "Circular Buffer". The term "circular" still gives no hint about its use, but at least it gives a hint about its internal structure.
I have a large text file with N number of lines. Now I have to read these lines in i iterations. Which means that I have to read n = Math.floor(N/i) lines in a single iteration. Now in each iteration I have to fill a string array of n length. So the basic question is that how should I read n lines in optimum time? The simplest way to do this is to use a BufferedReader and read one line at time with BufferedReader.readLine() but it will significantly decrease performance if n is too large. Is there a way to read exactly n lines at a time?
To read n lines from a text file, from a system point of view there is no other way than reading as many characters as necessary until you have seen n end-of-line delimiters (unless the file has been preprocessed to detect these, but I doubt this is allowed here).
As far as I know, no file I/O system in the world does support a function to read "until the nth occurrence of some character", nor "the n following lines" (but I am probably wrong).
If you really want to minimize the number of I/O function calls, your last resort is block I/O with which you could read a "page" at a time (say of length n times the expected or maximum line length), and detect the end-of-lines yourself.
I agree with Yves Daoust's answer, except for the paragraph recommending
If you really want to minimize the number of I/O function calls, your last resort is block I/O with which you could read a "page" at a time (say of length n times the expected or maximum line length), and detect the end-of-lines yourself.
There's no need to "detect the end-of-lines yourself". Something like
new BufferedReader(new InputStreamReader(is, charset), 8192);
creates a reader with a buffer of 8192 chars. The question is how useful this is for reading data in blocks. For this a byte[] is needed and there's a sun.nio.cs.StreamDecoder in between which I haven't looked into.
To be sure use
new BufferedReader(new InputStreamReader(new BufferedInputStream(is, 8192), charset));
so you get a byte[] buffer.
Note that 8192 is the default size for both BufferedReader and InputStreamReader, so leaving it out would change nothing in my above examples. Note that using much larger buffers makes no sense and can even be detrimental for performance.
Update
So far you get all the buffering needed and this should suffice. In case it doesn't, you can try:
to avoid the decoder overhead. When your lines are terminated by \n, you can look for (byte) \n in the file content without decoding it (unless you're using some exotic Charset).
to prefetch the data yourself. Normally, the OS should take care of it, so when your buffer becomes empty, Java calls into the OS, and it has the data already in memory.
to use a memory mapped file, so that no OS calls are needed for fetching more data (as all data "are" there when the mapping gets created).
I don't know where to seek clarifications and confirmations on Java API documentation and Java code, so I'm doing it here.
In the API documentation for FileChannel, I'm finding off-by-one errors w.r.t. to file position and file size in more places than one.
Here's just one example. The API documenation for transferFrom(...) states:
"If the given position is greater than the file's current size then no bytes are transferred."
I confirmed that the OpenJDK code too contains this code...
public long transferFrom(ReadableByteChannel src, long position, long count)
throws IOException
{
// ...
if (position > size())
return 0;
// ...
}
... in file FileChannelImpl.java consistent with the documentation.
Now, while the above code snippet and the API documentation appear mutually consistent, I 'feel' that the above should be 'greater than or equal to' and not merely 'greater than' because position being a 0-based index into the file's data, reading at position == size() will have no data to return to the caller! (At position == size() - 1, at least 1 byte -- the last byte of the file -- could be returned to the caller.)
Here are some other similar instances in the same API documentation page:
position(...): "Setting the position to a value that is greater than the file's current size is legal but does not change the size of the file." (Should have been 'greater than or equal to'.)
transferTo(...): " If the given position is greater than the file's current size then no bytes are transferred." (Should have been 'greater than or equal to'.)
read(...): "If the given position is greater than the file's current size then no bytes are read." (Should have been 'greater than or equal to'.)
Lastly, the documentation section for the return value of read(...) fails to remain even self-consistent with the rest of the documentation. Here's what it states:
read(...)
Returns:
The number of bytes read, possibly zero, or -1 if the given position is greater than or equal to the file's current size
So, in this lone instance, I do see them mention the right thing.
Overall, I don't know what to make of all of this. If I write my code today matching this documentation, then a future bug fix in Java (code or documentation) will render my code buggy, requiring a fix from my side as well. If I do the right thing myself today with things as they stand today, then my code becomes buggy to start with!
This could be a bit clearer in the javadoc and is now tracked here:
https://bugs.openjdk.java.net/browse/JDK-8029370
Note that clarifying the javadoc shouldn't change anything for implementations of FileChannel (for example, the transfer methods return the number of bytes transferred so this is 0 when the position is at size or beyond size).
It is not an off-by-one bug in that there is no behavior problem right? At best a doc problem. The docs aren't wrong here maybe just not complete.
However I am not sure they are missing something. position == size() is not an exceptional situation. It is a situation where 0 bytes can be read kind of by definition. position > size() is exceptional: less than 0 bytes can be read. It needs a note. Does an exception throw? Or nothing. read() is different since it has to return a byte so having 0 to read is the exceptional condition too.
I personally think the fact that you're asking means the docs could be more explicit. It is also not clear why that method doesnt short circuit rather than try to transfer 0 bytes. But there looks like a possible logic behind it.
(Or do you mean it does something not documented?)