While studying the ByteBuffer class I got to thinking about an array wrapped ByteBuffer that might be constructed as follows:
byte data[] = new byte[10];
// Populate data array
ByteBuffer myBuffer = ByteBuffer.wrap(data);
int i = myBuffer.getInt();
Which, I thought, might retrieve the first 4 bytes of my byte array as an int value. However, as I studied further, I seemed to find that the ByteBuffer has two modes which are read and write and we can flip between them using the flip() method. However, since flip is basically a toggle, it pre-supposes than one knows the initial value to meaningfully flip between the read and write states.
What is the definition of the initial state of a ByteBuffer?
write?
read?
A function of how it was created (eg. allocate vs wrap)?
Strictly speaking the ByteBuffer itself doesn't track if it is "read" or "write"; that's merely a function of how it is used. A ByteBuffer can read and write at any time. The reason why we say flip switches the "mode" is because it is part of the common task of writing to the buffer, flipping it, then reading from the buffer.
Indeed, both allocate and wrap set the limit and capacity to be equal to the array size, and the position to zero. This means that read operations can read up to the whole array, and write operations can fill the whole array. You can therefore do either reading or writing with a newly allocated or wrapped ByteBuffer.
Related
Why does ByteBuffer's flip() method called "flip"? What is "flipped" here? According to apidoc, two successive flips won't restore original state, and multiple flips will probably tend limit() to become zero.
Can I "unflip" somehow to reuse bytes went out of a limit?
Can I concatenate tail to be flipped with some other data?
One fairly common use case for the ByteBuffer is to construct some data structure piece-by-piece and then write that whole structure to disk. flip is used to flip the ByteBuffer from "reading from I/O" (putting) to "writing to I/O" (getting): after a sequence of puts is used to fill the ByteBuffer, flip will set the limit of the buffer to the current position and reset the position to zero. This has the effect of making a future get or write from the buffer write all of what was put into the buffer and no more.
After finishing the put, you might want to reuse the ByteBuffer to construct another data structure. To "unflip" it, call clear. This resets the limit to the capacity (making all of the buffer usable), and the position to 0.
So, a typical usage scenario:
ByteBuffer b = new ByteBuffer(1024);
for(int i=0; i<N; i++) {
b.clear();
b.put(header[i]);
b.put(data[i]);
b.flip();
out.write(b);
}
Flip assigns the current position value to the limit property and sets the position property to 0. Flip is useful to drain only active elements from a buffer.
For example, below program prints "hello" instead of empty elements of the buffer. Method calls limit and position can be replaced with flip.
CharBuffer cbuff = CharBuffer.allocate(40);
cbuff.put("hello");
// These two lines below are what flip does
cbuff.limit(cbuff.position());
cbuff.position(0);
while(cbuff.hasRemaining()) {
System.out.println(cbuff.get());
}
See http://www.zoftino.com/java-nio-tutorial for more information on buffers and channels.
flip() method makes a buffer ready for a new sequence of channel-write or relative get operations: It sets the limit to the current position and then sets the position to zero.
Buffer keeps track of the data written into it. Post writing, flip() method is called to switch from writing to reading mode.
A buffer has a fixed capacity; it maintains 2 pointers: start and end. get() returns the byte at the start position and increments start. put() puts the byte at the end position and increments end. No flip()!
I am mapping a file to memory and reading it back with java's ByteBuffer. This proves to be a really fast way of reading large files. However, I can only read the values sequentially. Meaning that once I read them buffer.getInt()the buffer pointer moves to the next bytes. So If I want to use a value more than once I have to store it to another variable:
int a = buffer.getInt()
I am noticing that this approach of copying a piece of memory to another is taking a long time (especially with a very large file) compared to just reading bytes. Is there a way I can re-read those bytes instead of copying them?
Just use position(int) to seek in ByteBuffer. Then you can read from anywhere.
ByteBuffer buffer=ByteBuffer.allocate(1000);
byte[] data=new byte[10];
buffer.position(100);
//read 10 from postion 100
buffer.get(data);
I have the following function:
byte[] test1 = {1,2,3,4};
byte[] test2 = {5,6,7,8};
ByteBuffer bbtest1 = ByteBuffer.wrap(test1).order(ByteOrder.LITTLE_ENDIAN);
ByteBuffer bbtest2= ByteBuffer.wrap(test2).order(ByteOrder.LITTLE_ENDIAN);
ByteBuffer contents = ByteBuffer.allocate(bbtest1.limit() + bbtest2.limit());
contents.put(bbtest1);
contents.put(bbtest2);
CRC32 checksum = new CRC32();
checksum.update(contents);
System.out.println(checksum.getValue());
No matter what values I assign my byte arrays, getValue() always returns 0 when I have concatenated by byteBuffers. According to this thread, this is a valid way of concatenating byteBuffers. If I only call put() on a single byteBuffer (for example if I comment out the line byte[] test2 = {5,6,7,8}; then getValue() actually returns a valid value.
Is this an issue with the way the ByteBuffers are concatenated, update(ByteBuffer) performs on concatenated ByteBuffers, or maybe something else altogether?
Your calls to put have advanced the position of the ByteBuffer to the point that there is no byte left to read from it for CRC32.update().
If you only put one of the two byte buffers, then there will still be 4 bytes to read for the CRC checksum (all 4 have the value 0).
You need to reset the position of your bytebuffer to zero before calling checksum.update(contents). You can use rewind or flip for that:
contents.rewind();
or
contents.flip();
Flip does the same as rewind(), but additionally it sets the limit of the ByteBuffer to the position it had before flipping, so if you first constructed the content of the ByteBuffer and then want to read from it, flip() is more correct, as you don't run the risk of reading from parts of the ByteBuffer that you didn't yet write into.
EJP's answer is insightful, as he points out that you don't need to concatenate byte buffers at all.
You can alternatively do one of:
update the CRC32 with the individual ByteBuffers
use ByteBuffer.put(byte[]) to directly put the test1 and test2 byte arrays in the contents ByteBuffer
skip ByteBuffers altogether and call CRC32.update(byte[]) with test1 and test2 in sequence.
You need to prepare the buffer for getting (or writing to a channel) by calling flip() after the puts (or reads from a channel), and compact() it afterwards.
But I don't know why you're concatenating ByteBuffers at all. Just do
checksum.update(bbtest1);
checksum.update(bbtest2);
There's no advantage to creating yet another copy of the data, and NIO contains scatter/gather methods to operate on multiple ByteBuffers at once.
Why does ByteBuffer's flip() method called "flip"? What is "flipped" here? According to apidoc, two successive flips won't restore original state, and multiple flips will probably tend limit() to become zero.
Can I "unflip" somehow to reuse bytes went out of a limit?
Can I concatenate tail to be flipped with some other data?
One fairly common use case for the ByteBuffer is to construct some data structure piece-by-piece and then write that whole structure to disk. flip is used to flip the ByteBuffer from "reading from I/O" (putting) to "writing to I/O" (getting): after a sequence of puts is used to fill the ByteBuffer, flip will set the limit of the buffer to the current position and reset the position to zero. This has the effect of making a future get or write from the buffer write all of what was put into the buffer and no more.
After finishing the put, you might want to reuse the ByteBuffer to construct another data structure. To "unflip" it, call clear. This resets the limit to the capacity (making all of the buffer usable), and the position to 0.
So, a typical usage scenario:
ByteBuffer b = new ByteBuffer(1024);
for(int i=0; i<N; i++) {
b.clear();
b.put(header[i]);
b.put(data[i]);
b.flip();
out.write(b);
}
Flip assigns the current position value to the limit property and sets the position property to 0. Flip is useful to drain only active elements from a buffer.
For example, below program prints "hello" instead of empty elements of the buffer. Method calls limit and position can be replaced with flip.
CharBuffer cbuff = CharBuffer.allocate(40);
cbuff.put("hello");
// These two lines below are what flip does
cbuff.limit(cbuff.position());
cbuff.position(0);
while(cbuff.hasRemaining()) {
System.out.println(cbuff.get());
}
See http://www.zoftino.com/java-nio-tutorial for more information on buffers and channels.
flip() method makes a buffer ready for a new sequence of channel-write or relative get operations: It sets the limit to the current position and then sets the position to zero.
Buffer keeps track of the data written into it. Post writing, flip() method is called to switch from writing to reading mode.
A buffer has a fixed capacity; it maintains 2 pointers: start and end. get() returns the byte at the start position and increments start. put() puts the byte at the end position and increments end. No flip()!
While writing a message on wire, I want to write down the number of bytes in the data followed by the data.
Message format:
{num of bytes in data}{data}
I can do this by writing the data to a temporary byteArrayOutput stream and then obtaining the byte array size from it, writing the size followed by the byte array. This approach involves a lot of overhead, viz. unnecessary creation of temporary byte arrays, creation of temporary streams, etc.
Do we have a better (considering both CPU and garbage creation) way of achieving this?
A typical approach would be to introduce a re-useable ByteBuffer. For example:
ByteBuffer out = ...
int oldPos = out.position(); // Remember current position.
out.position(oldPos + 2); // Leave space for message length (unsigned short)
out.putInt(...); // Write out data.
// Finally prepend buffer with number of bytes.
out.putShort(oldPos, (short)(out.position() - (oldPos + 2)));
Once the buffer is populated you could then send the data over the wire using SocketChannel.write(ByteBuffer) (assuming you are using NIO).
Here’s what I would do, in order of preference.
Don’t bother about memory consumption and stuff. Most likely this already is the optimal solution unless it takes a lot of time to create the byte representation of your data so that creating it twice is a noticable impact.
(Actually this would be more like #37 on my list, with #2 to #36 being empty.) Include a method in your all your data objects that can calculate the size of the byte representation and takes less resources than it would to create the byte representation.