We Keep Coding

java.lang.UnsupportedOperationException at java.nio.ByteBuffer.array(ByteBuffer.java:959)

The following Java code compiles, but there's an error at runtime:
# javac ByteBufTest.java
# java ByteBufTest
Exception in thread "main" java.lang.UnsupportedOperationException
at java.nio.ByteBuffer.array(ByteBuffer.java:959)
at ByteBufTest.<init>(ByteBufTest.java:12)
at ByteBufTest.main(ByteBufTest.java:33)
#
Why does this happen?
Note：Next, I need to use mDirectBuffer in JNI, so I have to use the ByteBuffer.allocateDirect(TEST_BUFFER_SIZE) function。
ByteBufTest.java:
import java.nio.ByteBuffer;
public class ByteBufTest {
public static final int TEST_BUFFER_SIZE = 128;
private ByteBuffer mDirectBuffer;
public ByteBufTest() {
mDirectBuffer = ByteBuffer.allocateDirect(TEST_BUFFER_SIZE);
byte[] buf = mDirectBuffer.array();
buf[1]=100;
}
public void test() {
printBuffer("nativeInitDirectBuffer",mDirectBuffer.array());
}
private void printBuffer( String tag, byte[] buffer ) {
StringBuffer sBuffer = new StringBuffer();
for( int i=0; i<buffer.length; i++ ) {
sBuffer.append(buffer[i]);
sBuffer.append(" ");
}
//System.out.println(tag+sBuffer);
}
public static void main(String[] args) throws Exception {
ByteBufTest item = new ByteBufTest();
item.test();
}
}

This is the expected behaviour. The Javadoc states
throws UnsupportedOperationException - If this buffer is not backed by an accessible array
You should try another approach or search for another implementation, e.g.
mDirectBuffer = ByteBuffer.wrap(new byte[TEST_BUFFER_SIZE]);

This exception occurs at runtime if the resulting buffer is not backed by an accessible array. You can try allocate() method.

You should call java.nio.ByteBuffer.hasArray() to ensure that java.nio.ByteBuffer.array() will succeed in order to write clean and portable code as stated in the Java API documentation:
If this method returns true then the array and arrayOffset methods may safely be invoked
You can allocate a direct writable NIO byte buffer in your Java source code by calling java.nio.ByteBuffer.allocateDirect(int) as you already do and call java.nio.ByteBuffer.get(byte[]) to store the content of the buffer into an array, this method is supported by Android too. Keep in mind that it's a relative operation that affects the position of the NIO buffer.
Maybe another approach would consist in using the NIO buffer as is without doing any conversion but I'm not sure that it suits your needs.

WriteObject not properly writing a Set?

I hope I didn't just find a bug in Java! I am running JDK 7u11 (mostly because that is the sanctioned JVM allowed by my employer) and I am noticing a very odd issue.
Namely, I am chunking data into a LinkedHashSet and writing it to a file using the ObjectOutputStream daisy changed through the GZIpOutputStream (mentioning this just in case it matters).
Now, when I get to the other side of the program and readObject() I notice that the size always reads 68, which I is the first size. The underlying table can have many more or less than 68, but the .size() method always returns 68. More troubling, when I try to manually iterate the underlying Set, it also stops at 68.
while(...) {
oos.writeInt(p_rid);
oos.writeObject(wptSet);
wptSet.clear();
// wptSet = new LinkedHashSet<>(); // **This somehow causes the heapsize to increase dramatically, but it does solve the problem**
}
And when reading it
Set<Coordinate> coordinates = (Set<Coordinate>) ois.readObject();
the coordinates.size() always returns 68. Now, I could make a workaround by also .writeInt() the size, but I can only iterate through 68 members!
Notice the wptSet = new LinkedHashSet<>() line actually solves the issue. The main problem with that is it makes my heapsize skyrocket when looking at the program in JVisualVM.
Update:
I actually just found a viable workaround that fixes the memory leak of re-instantiating wptSet... System.gc() Calling that after each call to .clear() actually keeps the memory leak away.
Either way, I shouldn't have to do this and shipping out the LinkedHashSet should not exhibit this behavior.

Alright, I think I understand what you are asking.
Here is an example to reproduce...
import java.util.*;
import java.io.*;
class Example {
public static void main(String[] args) throws Exception {
Set<Object> theSet = new LinkedHashSet<>();
final int size = 3;
for(int i = 0; i < size; ++i) {
theSet.add(i);
}
ByteArrayOutputStream bytesOut = new ByteArrayOutputStream();
ObjectOutputStream objectsOut = new ObjectOutputStream(bytesOut);
for(int i = 0; i < size; ++i) {
objectsOut.writeObject(theSet);
theSet.remove(i); // mutate theSet for each write
}
ObjectInputStream objectsIn = new ObjectInputStream(
new ByteArrayInputStream(bytesOut.toByteArray()));
for(;;) {
try {
System.out.println(((Set<?>)objectsIn.readObject()).size());
} catch(EOFException e) {
break;
}
}
}
}
The output is
3
3
3
What is going on here is that ObjectOutputStream detects that you are writing the same object every time. Each time theSet is written, a "shared reference" to the object is written so that the same object is deserialized each time. This is explained in the documentation:
Multiple references to a single object are encoded using a reference sharing mechanism so that graphs of objects can be restored to the same shape as when the original was written.
In this case you should use writeUnshared(Object) which will bypass this mechanism, instead of writeObject(Object).

How to Declare a Byte Array of Infinite Size/Dynamic in Java?

I am declaring a byte array which is of unknown size to me as it keeps on updating, so how can I declare the byte array of infinite size/variable size?

You cannot declare an array of infinite size, as that would require infinite memory. Additionally, all the allocation calls deal with numbers, not infinite quantities.
You can allocate a byte buffer that resizes on demand. I believe the easiest choice would be a ByteArrayOutputStream.
ByteBuffer has an API which makes manipulation of the buffer easier, but you would have to build the resize functionality yourself. The easiest way will be to allocate a new, larger array, copy the old contents in, and swap the new buffer for the old.
Other answers have mentioned using a List<Byte> of some sort. It is worth noting that if you create a bunch of new Byte() objects, you can dramatically increase memory consumption. Byte.valueOf sidesteps this problem, but you have to ensure that it is consistently used throughout your code. If you intend to use this list in many places, I might consider writing a simple List decorator which interns all the elements. For example:
public class InterningList extends AbstractList<Byte>
{
...
#Override
public boolean add(Byte b) {
return super.add(Byte.valueOf(b));
}
...
}
This is not a complete (or even tested) example, just something to start with...

Arrays in Java are not dynamic. You can use list instead.
List<Byte> list = new ArrayList<Byte>();
Due to autoboxing feature you can freely add either Byte objects or primitive bytes to this list.

To define a bytearray of varying length just use the apache commons.io.IOUtils library instead of assigning manual length like
byte[] b=new byte[50];
You can pass your input stream to IOUtils function which will perform a read function on this inputstream thus byte array will have exact length of bytes as required.
ex.
byte[] b = IOUtils.toByteArray(inpustream);
Chaos..

ByteArrayOutputStream will allow for writing to a dynamic byte array. However, methods such as remove, replace and insert are not available. One has to extract the byte array and then manipulate it directly.

Your best bet is to use an ArrayList. As it resizes as you fill it.
List<Byte> array = new ArrayList<Byte>();

The obvious solution would be to use an ArrayList.
But this is a bad solution if you need performance or are constrained in memory, as it doesn't really store bytes but Bytes (that is, objects).
For any real application, the answer is simple : you have to manage yourself the byte array, by using methods making it grow as necessary. You may embed it in a specific class if needed :
public class AlmostInfiniteByteArray {
private byte[] array;
private int size;
public AlmostInfiniteByteArray(int cap) {
array = new byte[cap];
size = 0;
}
public int get(int pos) {
if (pos>=size) throw new ArrayIndexOutOfBoundsException();
return array[pos];
}
public void set(int pos, byte val) {
if (pos>=size) {
if (pos>=array.length) {
byte[] newarray = new byte[(pos+1)*5/4];
System.arraycopy(array, 0, newarray, 0, size);
array = newarray;
}
size = pos+1;
}
array[pos] = val;
}
}

Use an ArrayList of any subtype of List
The different implementations of List can allow you to do different things on the list (eg different traversal strategy, different performance etc)

Initial capacity of ArrayList is 10. You can change it by ArrayList(5000).
ArrayList will double it's size when needed (it will create new array and copy the old one to to the new one).

I would tweak slightly other people's answers.
Create a LargeByteArray class to manage your array. It will have get and set methods, etc, whatever you will need.
Behind the scenes that class will use a long to hold the current length and use an ArrayList to store the contents of the array.
I would pick to store byte[8192] or byte[16384] arrays in the ArrayList. That will give a reasonable trade off in terms of size wasted and reduce the need for resizing.
You can even make the array 'sparse' ie only allocate the list.get(index/8192) entry if there is a non-zero value stored in that box.
Such a structure can give you significantly more storage in some cases.
Another strategy you can use is to compress the byte[] boxes after write and uncompress before read (use a LRU cache for reading) which can allow storing twice or more than available ram... Though that depends on the compression strategy.
After that you can look at paging some boxes out to disk...
That's as close to an infinite array as I can get you ;-)

You can make use of IOUtils from piece, as Prashant already told.
Here's a little piece from it which can solve the task (you will need IOUtils.toByteArray):
public class IOUtils {
private static final int DEFAULT_BUFFER_SIZE = 1024 * 4;
public static byte[] toByteArray(InputStream input) throws IOException {
ByteArrayOutputStream output = new ByteArrayOutputStream();
copy(input, output);
return output.toByteArray();
}
public static int copy(InputStream input, OutputStream output)
throws IOException {
long count = copyLarge(input, output);
if (count > Integer.MAX_VALUE) {
return -1;
}
return (int) count;
}
public static long copyLarge(InputStream input, OutputStream output)
throws IOException {
byte[] buffer = new byte[DEFAULT_BUFFER_SIZE];
long count = 0;
int n = 0;
while (-1 != (n = input.read(buffer))) {
output.write(buffer, 0, n);
count += n;
}
return count;
}
}

JNI and Shared memory

I use JNI to access shared memory segments previously created. I use a Java/Jni read/write function which could either take the address of the shared buffer, or the id of the mapped region.
Unfortunately, it seems like every time I call the read/write functions I have to remap the segment using the function below. Is there any way to get round this? I would like (i have tried this, but it didn't work, to only have to map the segment once, and simply use the void* addr parameter in the future, as opposed to every time I call the JNI method read or write, to have to call map_shared_memory. The current code works well, but remains relatively slow. Hence the desire to minimise seemingly unnecessary operations.
void* map_shared_memory(int id) {
void* addr;
if (id == 0)
return NULL;
addr = shmat(id, NULL, 0);
shmctl(id, IPC_RMID, NULL);
if(addr<=0)
perror("Error Mapping Shared Memory: ");
return addr;
}

I suggest a collaboration between a class ShmFactory which opens (or create) the Shared Memory and and ShmAccess which offer read/write methods.
interface ShmAccess
{
void read( shmId, byte[] bytes );
void write( shmId, byte[] bytes );
}
interface ShmFactory
{
public native ShmAccess open( int shmId );
}

An Iterator which mutates and returns the same object. Bad practice?

I'm writing GC friendly code to read and return to the user a series of byte[] messages. Internally I reuse the same ByteBuffer which means I'll repeatedly return the same byte[] instance most of the time.
I'm considering writing cautionary javadoc and exposing this to the user as a Iterator<byte[]>. AFAIK it won't violate the Iterator contract, but the user certainly could be surprised if they do Lists.newArrayList(myIterator) and get back a List populated with the same byte[] in each position!
The question: is it bad practice for a class that may mutate and return the same object to implement the Iterator interface?
If so, what is the best alternative? "Don't mutate/reuse your objects" is an easy answer. But it doesn't address the cases when reuse is very desirable.
If not, how do you justify violating the principle of least astonishment?
Two minor notes:
I'm using Guava's AbstractIterator so remove() isn't really of concern.
In my use case the user is me and the visibility of this class will be limited, but I've tried to ask this generally enough to apply more broadly.
Update: I'm accepting Louis' answer because it has 3x more votes than Keith's, but note that in my use case I'm planning to take the code that I left in a comment on Keith's answer to production.

EnumMap did essentially exactly this in its entrySet() iterator, which causes confusing, crazy, depressing bugs to this day.
If I were you, I just wouldn't use an Iterator -- I'd write a different API (possibly quite dissimilar from Iterator, even) and implement that. For example, you might write a new API that takes as input the ByteBuffer to write the message into, so users of the API could control whether or not the buffer gets reused. That seems reasonably intuitive (the user can write code that obviously and cleanly reuses the ByteBuffer), without creating unnecessarily cluttered code.

I would define an intermediate object which you can invalidate. So your function would return an Iterator<ByteArray>, and ByteArray is something like this:
class ByteArray {
private byte[] data;
ByteArray(byte[] d) { data = d; }
byte[] getData() {
if (data == null) throw new BadUseOfIteratorException();
return data;
}
void invalidate() { data = null; }
}
Then your iterator can invalidate the previously returned ByteArray so that any future access (via getData, or any other accessor you provide) will fail. Then at least if someone does something like Lists.newArrayList(myIterator), they will at least get an error (when the first invalid ByteArray is accessed) instead of silently returning the wrong data.
Of course, this won't catch all possible bad uses, but probably the common ones. If you're happy with never returning the raw byte[] and providing accessors like byte get(int idx) instead, then it should catch all cases.
You will have to allocate a new ByteArray for each iterator return, but hopefully that's a lot less expensive than copying your byte[] for each iterator return.

Just like Keith Randall I'd also create Iterator<ByteArray>, but working quite differently (the annotations below come from lombok):
#RequiredArgsConstructor
public class ByteArray {
#Getter private final byte[] data;
private final ByteArrayIterable source;
void allowReuse() {
source.allowReuse();
}
}
public class ByteArrayIterable implements Iterable<ByteArray> {
private boolean allowReuse;
public allowReuse() {
allowReuse = true;
}
public Iterator<ByteArray> iterator() {
return new AbstractIterator<ByteArray>() {
private ByteArray nextElement;
public ByteArray computeNext() {
if (noMoreElements()) return endOfData();
if (!allowReuse) nextElement =
new ByteArray(new byte[length], ByteArrayIterable.this);
allowReuse = false;
fillWithNewData(lastElement.getData());
}
}
}
}
Now in calls like Lists.newArrayList(myIterator) always a new byte array gets allocated, so everything works. In your loops like
for (ByteArray a : myByteArrayIterable) {
a.allowReuse();
process(a.getData());
}
the buffer gets reused. No harm may result, unless you call allowReuse() by mistake. If you forget to call it, then you get worse performance but correct behavior.
Now I see it could work without ByteArray, the important thing is that myByteArrayIterable.allowReuse() gets called, which could be done directly.