I have the following function:
byte[] test1 = {1,2,3,4};
byte[] test2 = {5,6,7,8};
ByteBuffer bbtest1 = ByteBuffer.wrap(test1).order(ByteOrder.LITTLE_ENDIAN);
ByteBuffer bbtest2= ByteBuffer.wrap(test2).order(ByteOrder.LITTLE_ENDIAN);
ByteBuffer contents = ByteBuffer.allocate(bbtest1.limit() + bbtest2.limit());
contents.put(bbtest1);
contents.put(bbtest2);
CRC32 checksum = new CRC32();
checksum.update(contents);
System.out.println(checksum.getValue());
No matter what values I assign my byte arrays, getValue() always returns 0 when I have concatenated by byteBuffers. According to this thread, this is a valid way of concatenating byteBuffers. If I only call put() on a single byteBuffer (for example if I comment out the line byte[] test2 = {5,6,7,8}; then getValue() actually returns a valid value.
Is this an issue with the way the ByteBuffers are concatenated, update(ByteBuffer) performs on concatenated ByteBuffers, or maybe something else altogether?
Your calls to put have advanced the position of the ByteBuffer to the point that there is no byte left to read from it for CRC32.update().
If you only put one of the two byte buffers, then there will still be 4 bytes to read for the CRC checksum (all 4 have the value 0).
You need to reset the position of your bytebuffer to zero before calling checksum.update(contents). You can use rewind or flip for that:
contents.rewind();
or
contents.flip();
Flip does the same as rewind(), but additionally it sets the limit of the ByteBuffer to the position it had before flipping, so if you first constructed the content of the ByteBuffer and then want to read from it, flip() is more correct, as you don't run the risk of reading from parts of the ByteBuffer that you didn't yet write into.
EJP's answer is insightful, as he points out that you don't need to concatenate byte buffers at all.
You can alternatively do one of:
update the CRC32 with the individual ByteBuffers
use ByteBuffer.put(byte[]) to directly put the test1 and test2 byte arrays in the contents ByteBuffer
skip ByteBuffers altogether and call CRC32.update(byte[]) with test1 and test2 in sequence.
You need to prepare the buffer for getting (or writing to a channel) by calling flip() after the puts (or reads from a channel), and compact() it afterwards.
But I don't know why you're concatenating ByteBuffers at all. Just do
checksum.update(bbtest1);
checksum.update(bbtest2);
There's no advantage to creating yet another copy of the data, and NIO contains scatter/gather methods to operate on multiple ByteBuffers at once.
Related
Why does ByteBuffer's flip() method called "flip"? What is "flipped" here? According to apidoc, two successive flips won't restore original state, and multiple flips will probably tend limit() to become zero.
Can I "unflip" somehow to reuse bytes went out of a limit?
Can I concatenate tail to be flipped with some other data?
One fairly common use case for the ByteBuffer is to construct some data structure piece-by-piece and then write that whole structure to disk. flip is used to flip the ByteBuffer from "reading from I/O" (putting) to "writing to I/O" (getting): after a sequence of puts is used to fill the ByteBuffer, flip will set the limit of the buffer to the current position and reset the position to zero. This has the effect of making a future get or write from the buffer write all of what was put into the buffer and no more.
After finishing the put, you might want to reuse the ByteBuffer to construct another data structure. To "unflip" it, call clear. This resets the limit to the capacity (making all of the buffer usable), and the position to 0.
So, a typical usage scenario:
ByteBuffer b = new ByteBuffer(1024);
for(int i=0; i<N; i++) {
b.clear();
b.put(header[i]);
b.put(data[i]);
b.flip();
out.write(b);
}
Flip assigns the current position value to the limit property and sets the position property to 0. Flip is useful to drain only active elements from a buffer.
For example, below program prints "hello" instead of empty elements of the buffer. Method calls limit and position can be replaced with flip.
CharBuffer cbuff = CharBuffer.allocate(40);
cbuff.put("hello");
// These two lines below are what flip does
cbuff.limit(cbuff.position());
cbuff.position(0);
while(cbuff.hasRemaining()) {
System.out.println(cbuff.get());
}
See http://www.zoftino.com/java-nio-tutorial for more information on buffers and channels.
flip() method makes a buffer ready for a new sequence of channel-write or relative get operations: It sets the limit to the current position and then sets the position to zero.
Buffer keeps track of the data written into it. Post writing, flip() method is called to switch from writing to reading mode.
A buffer has a fixed capacity; it maintains 2 pointers: start and end. get() returns the byte at the start position and increments start. put() puts the byte at the end position and increments end. No flip()!
I am mapping a file to memory and reading it back with java's ByteBuffer. This proves to be a really fast way of reading large files. However, I can only read the values sequentially. Meaning that once I read them buffer.getInt()the buffer pointer moves to the next bytes. So If I want to use a value more than once I have to store it to another variable:
int a = buffer.getInt()
I am noticing that this approach of copying a piece of memory to another is taking a long time (especially with a very large file) compared to just reading bytes. Is there a way I can re-read those bytes instead of copying them?
Just use position(int) to seek in ByteBuffer. Then you can read from anywhere.
ByteBuffer buffer=ByteBuffer.allocate(1000);
byte[] data=new byte[10];
buffer.position(100);
//read 10 from postion 100
buffer.get(data);
I would like to read the bytes into the direct ByteBuffer and then decode them without rewrapping the original buffer into the byte[] array to minimize memory allocations.
Hence I'd like to avoid using StandardCharsets.UTF_8.decode() as it allocates the new array on the heap.
I'm stuck on how to decode the bytes. Consider the following code that writes a string into the buffer and then reads id again.
ByteBuffer byteBuffer = ByteBuffer.allocateDirect(2 << 16);
byteBuffer.put("Hello Dávid".getBytes(StandardCharsets.UTF_8));
byteBuffer.flip();
CharBuffer charBuffer = byteBuffer.asCharBuffer();
for (int i = charBuffer.position(); i < charBuffer.length(); i++) {
System.out.println(charBuffer.get());
}
The code output:
䡥汬漠
How can I decode the buffer?
I would like to read the bytes into the direct ByteBuffer and then decode them without rewrapping the original buffer into the byte[] array to minimize memory allocations.
ByteBuffer.asCharBuffer() fits your need, indeed, since both wrappers share the same underlying buffer.
This method's javadoc says:
The new buffer's position will be zero, its capacity and its limit will be the number of bytes remaining in this buffer divided by two
Although it's not explicitly said, it's a hint that CharBuffer uses UTF-16 character encoding over the given buffer. Since we don't have control over what encoding the charbuffer uses, you don't have much choice but to necessarily write the character bytes in that encoding.
byteBuffer.put("Hello Dávid".getBytes(StandardCharsets.UTF_16));
One thing about your printing for loop. Be careful that CharBuffer.length() is actually the number of remaining chars between the buffer's position and limit, so it decreases as you call CharBuffer.get(). So you should use get(int) or change the for termination condition to limit().
You can't specify the encoding of a CharBuffer. See here: What Charset does ByteBuffer.asCharBuffer() use?
Also, since buffers are mutable, I don't see how you could ever possibly create a String from it which are always immutable without doing a memory re-allocation...
While studying the ByteBuffer class I got to thinking about an array wrapped ByteBuffer that might be constructed as follows:
byte data[] = new byte[10];
// Populate data array
ByteBuffer myBuffer = ByteBuffer.wrap(data);
int i = myBuffer.getInt();
Which, I thought, might retrieve the first 4 bytes of my byte array as an int value. However, as I studied further, I seemed to find that the ByteBuffer has two modes which are read and write and we can flip between them using the flip() method. However, since flip is basically a toggle, it pre-supposes than one knows the initial value to meaningfully flip between the read and write states.
What is the definition of the initial state of a ByteBuffer?
write?
read?
A function of how it was created (eg. allocate vs wrap)?
Strictly speaking the ByteBuffer itself doesn't track if it is "read" or "write"; that's merely a function of how it is used. A ByteBuffer can read and write at any time. The reason why we say flip switches the "mode" is because it is part of the common task of writing to the buffer, flipping it, then reading from the buffer.
Indeed, both allocate and wrap set the limit and capacity to be equal to the array size, and the position to zero. This means that read operations can read up to the whole array, and write operations can fill the whole array. You can therefore do either reading or writing with a newly allocated or wrapped ByteBuffer.
Consider we have a socket connection between two device (A and B). Now if I write only 16 bytes (size doesn't matter here) to the output stream (not BufferedOutputStream) of the socket in side A 3 times or in general more than once like this:
OutputStream outputStream = socket.getOutputStream();
byte[] buffer = new byte[16];
OutputStream.write(buffer);
OutputStream.write(buffer);
OutputStream.write(buffer);
I read the data in side B using the socket input stream (not BufferedInputStream) with a buffer larger than sending buffer for example 1024:
InputStream inputStream = socket.getInputStream();
byte[] buffer = new byte[1024];
int read = inputStream.read(buffer);
Now I wonder how the data is received on side B? May it get accumulated or it exactly read the data as A sends it? In another word may the read variable get more than 16?
InputStream makes very few guarantees about how much data will be read by any invocation of the multi-byte read() methods. There is a whole class of common programming errors that revolve around misunderstandings and wrong assumptions about that. For example,
if InputStream.read(byte[]) reads fewer bytes than the provided array can hold, that does not imply that the end of the stream has been reached, or even that another read will necessarily block.
the number of bytes read by any one invocation of InputStream.read(byte[]) does not necessarily correlate to any characteristic of the byte source on which the stream draws, except that it will always read at least one byte when not at the end of the stream, and that it will not read more bytes than are actually available by the time it returns.
the number of available bytes indicated by the available() method does not reliably indicate how many bytes a subsequent read should or will read. A nonzero return value reliably indicates only that the next read will not block; a zero return value tells you nothing at all.
Subclasses may make guarantees about some of those behaviors, but most do not, and anyway you often do not know which subclass you actually have.
In order to use InputStreams properly, you generally must be prepared to perform repeated reads until you get sufficient data to process. That can mean reading until you have accumulated a specific number of bytes, or reading until a delimiter is encountered. In some cases you can handle any number of bytes from any given read; generally these are cases where you are looping anyway, and feeding everything you read to a consumer that can accept variable length chunks (many compression and encryption interfaces are like that, for example).
Per the docs:
public int read(byte[] b) throws IOException
Reads some number of bytes from the input stream and stores them into the buffer array b. The number of bytes
actually read is returned as an integer. This method blocks until
input data is available, end of file is detected, or an exception is
thrown. If the length of b is zero, then no bytes are read and 0 is
returned; otherwise, there is an attempt to read at least one byte. If
no byte is available because the stream is at the end of the file, the
value -1 is returned; otherwise, at least one byte is read and stored
into b.
The first byte read is stored into element b[0], the next one into
b[1], and so on. The number of bytes read is, at most, equal to the
length of b. Let k be the number of bytes actually read; these bytes
will be stored in elements b[0] through b[k-1], leaving elements b[k]
through b[b.length-1] unaffected.
Read(...) tells you how many bytes it put into the array and yes, you can read further; you'll get whatever was already there.