We Keep Coding

Java uint32 (stored as long) to 4 byte array

I'm writing to a storage format that has uint32, with a max allowed value of "4294967295".
Integer in Java is, of course, just under half that at "2147483647". So internally, I have to use either Long or Guava's UnsignedInteger.
To write to this format, the byte array length needs to be 4, which fits Integer just fine, but converting Long to a byte array requires an array of length 8.
How can I convert a Long or UnsignedInteger representing a max value of "4294967295" as a 4 byte array?

Simply convert it to an 8 byte array and then take only the last 4 bytes:
public static byte[] fromUnsignedInt(long value)
{
byte[] bytes = new byte[8];
ByteBuffer.wrap(bytes).putLong(value);
return Arrays.copyOfRange(bytes, 4, 8);
}
To reverse this you can use the following method:
public static long toUnsignedInt(byte[] bytes)
{
ByteBuffer buffer = ByteBuffer.allocate(8).put(new byte[]{0, 0, 0, 0}).put(bytes);
buffer.position(0);
return buffer.getLong();
}
Note that this method CAN take a negative long or a long that exceed the range of an unsigned int and won't throw a exception in such a case!

You can just cast it to an int and do whatever you do that turns ints into arrays, such as this: (not tested)
public static byte[] getUnsignedInt(long value)
{
byte[] bytes = new byte[4];
ByteBuffer.wrap(bytes).putInt((int)value);
return bytes;
}
Of course if you're putting these things in a ByteBuffer anyway, you might as well do that directly.
The "meaning" or "interpretation" of the top bit is irrelevant if all you're doing it storing it. For example, 4294967295 would be interpreted as -1, but it's really the same number: 0xFFFFFFFF in hexadecimal, so you will get the byte array { 0xFF, 0xFF, 0xFF, 0xFF }.
To reverse it, you could do something like this (not tested)
public static long toUnsignedInt(byte[] bytes)
{
ByteBuffer buffer = ByteBuffer.allocate(4).put(bytes);
buffer.position(0);
return buffer.getInt() & 0xFFFFFFFFL;
}

An answer without the object creation and array copying of the accepted answer... It's easy to do yourself w/ shift operations. See:
import java.io.*;
public class TestUINT32 {
public static void writeLongAsUINT32(long value, OutputStream os) throws IOException {
for (int i=0; i<4; ++i) {
os.write((byte) value);
value = value >> 8;
}
}
public static void main(String[] args) throws IOException {
ByteArrayOutputStream os = new ByteArrayOutputStream();
long value = 0xFFEEDDBB;
writeLongAsUINT32(value, os);
byte[] ba = os.toByteArray();
for (int i=ba.length; i>0; --i) {
System.out.print(String.format("%02X", ba[i-1]));
}
System.out.println();
}
}
Example run:
$ java TestUINT32
FFEEDDBB

Convert array of doubles to byte array: What is the Java way of C# Buffer.BlockCopy?

I need to serialize an array of doubles to base64 in Java. I have following method from C#
public static string DoubleArrayToBase64( double[] dValues ) {
byte[] bytes = new byte[dValues.Length * sizeof( double )];
Buffer.BlockCopy( dValues, 0, bytes, 0, bytes.Length );
return Convert.ToBase64String( bytes );
}
How do I do that in Java? I tried
Byte[] bytes = new Byte[abundaceArray.length * Double.SIZE];
System.arraycopy(abundaceArray, 0, bytes, 0, bytes.length);
abundanceValues = Base64.encodeBase64String(bytes);
however this leads to an IndexOutofBoundsException.
How can I achieve this in Java?
EDIT:
Buffer.BlockCopy copies on byte level, the last paramter is number of bytes. System.arraycopy last parameter is number of elements to copy. So yes it should be abundaceArray.length but then a ArrayStoreException is thrown.
EDIT2:
The base64 string must be the same as the ine created with the c# code!

You get an ArrayStoreException when the array types on the method are not the same primitive, so double to byte will not work. Here is a workaround i patched up that seems to work. I do not know of any method in the java core that does automatic conversion from primitive to byte block :
public class CUSTOM {
public static void main(String[] args) {
double[] arr = new double[]{1.1,1.3};
byte[] barr = toByteArray(arr);
for(byte b: barr){
System.out.println(b);
}
}
public static byte[] toByteArray(double[] from) {
byte[] output = new byte[from.length*Double.SIZE/8]; //this is reprezented in bits
int step = Double.SIZE/8;
int index = 0;
for(double d : from){
for(int i=0 ; i<step ; i++){
long bits = Double.doubleToLongBits(d); // first transform to a primitive that allows bit shifting
byte b = (byte)((bits>>>(i*8)) & 0xFF); // bit shift and keep adding
int currentIndex = i+(index*8);
output[currentIndex] = b;
}
index++;
}
return output;
}
}

The Double.SIZE get 64 which is number of bits I suggest to initialize the array like this
Byte[] bytes = new Byte[abundaceArray.length * 8];

Not sure what this C# function does, but I suspect you should replace this line
System.arraycopy(abundaceArray, 0, bytes, 0, bytes.length);
with this
System.arraycopy(abundaceArray, 0, bytes, 0, abundaceArray.length);

I'm guessing you're using the apache commons Base64 class. That only has methods accepting an array of bytes (the primitive type), not Bytes (object wrapper around primitive type).
It's not clear what type your 'abundaceArray' is - whether it's doubles or Doubles.
Either way, you can't use System.arraycopy to copy between arrays of difference primitive types.
I think your best bet is to serialise your array object to a byte array, then base64 encode that.
eg:
ByteArrayOutputStream b = new ByteArrayOutputStream(); // to store output from serialization in a byte array
ObjectOutputStream o = new ObjectOutputStream(b); // to do the serialization
o.writeObject(abundaceArray); // arrays of primitive types are serializable
String abundanceValues = Base64.encodeBase64String(b.toByteArray());
There is of course an ObjectInputStream for going in the other direction at the other end.

Concat two ByteBuffers in Java

How can I concat two ByteBuffers to one ByteBuffer?
The following doesn't work:
ByteBuffer bb = ByteBuffer.allocate(100);
ByteBuffer bb2 = ByteBuffer.allocate(200);
bb.allocate(200).put(bb2);
System.out.println(bb.array().length);
The length of bb is still 100.

Something like
bb = ByteBuffer.allocate(300).put(bb).put(bb2);
should do the job: Create a buffer that is large enough to hold the contents of both buffers, and then use the relative put-methods to fill it with the first and the second buffer. (The put method returns the instance that the method was called on, by the way)

We'll be copying all data. Remember that this is why string concatenation is expensive!
public static ByteBuffer concat(final ByteBuffer... buffers) {
final ByteBuffer combined = ByteBuffer.allocate(Arrays.stream(buffers).mapToInt(Buffer::remaining).sum());
Arrays.stream(buffers).forEach(b -> combined.put(b.duplicate()));
return combined;
}

you can use the method here
https://github.com/ata4/ioutils/blob/047e401d73c866317af2e12f7803b3ee43eec80a/src/main/java/info/ata4/io/buffer/ByteBufferUtils.java#L289
and for example:
ByteBuffer concat() {
int length = 0;
for (ByteBuffer bb : buffers) {
bb.rewind();
length += bb.remaining();
}
ByteBuffer bbNew = ByteBuffer.allocateDirect((int) length);
// put all buffers from list
for (ByteBuffer bb : buffers) {
bb.rewind();
bbNew.put(bb);
}
bbNew.rewind();
return bbNew;
}

Probably because on line3 i.e. bb.allocate(200).put(bb2); ,
bb.allocate(200) is returning a new Byte buffer (See https://docs.oracle.com/javase/7/docs/api/java/nio/ByteBuffer.html#allocate(int) ). That is not actually changing bb itself. So its still the bytebuffer of capacity 100 from line1.

Try the following code:
//store both ByteBuffer object as an array
byte[] array1 = ByteBuffer.allocate(100).array();
byte[] array2 = ByteBuffer.allocate(200).array();
//create a ByteBuffer which is big enough
ByteBuffer bigenough = ByteBuffer.allocate(array1.length + array2.length);
//put the two arrays in one ByteBuffer
ByteBuffer after1 = bigenough.put(array1, 0, array1.length);
ByteBuffer result = after1.put(array2, array1.length, array2.length);
//print the length of the combined array.
System.out.println(result.array().length);

Buffered short I/O

How can read/write in a short buffer?
I'm trying to implement a BufferedReader and Writer for short values. Each times will be passed a short[] and will be read a short[].
But the java API doesn't have this interface, only byte[].
What's the best way to implement this feature?

Well, for your BufferedInputStream (not reader), you could try reading 2 bytes at the same time:
public synchronized int read(short[] s, int off, int len) throws IOException {
byte[] b = new byte[s.length * 2];
int read = read(b, off * 2, len * 2);
for (int i = 0; i < read; i+=2) {
int b1 = b[i];
int b2 = b[i+1];
s[i/2] = (short) ((b1 << 8) | b2);
}
return read / 2;
}
For your BufferedOutputStream (not writer), you could try the reverse operation for writing 2 bytes at the same time.

You could read/write the bytes and convert sets of two into shorts using ByteBuffer of length 2:
ByteBuffer put() to put the bytes into or putShort() when going the other way.
ByteBuffer.getShort() to convert back into shorts.

You could implement the Reader interface, and then extend the writer class to implement a writer that excepts short[].

Just wrap a DataOutputStream around a BufferedOutputStream, and implement a method writeShortArray(short[]) that calls writeShort() iteratively over the array argument. Similarly for input.

Reading from socket input stream returns data in the wrong order

I am sending byte arrays over a socket. The sent data starts off with 4 bytes indicating the length of the following byte array.
// get the amount of data being sent
byte[] lengthOfReplyAsArray = new byte[4];
forceRead(inputStream, lengthOfReplyAsArray);
int lengthOfReply = byteArrayToInt(lengthOfReplyAsArray);
// read the data into a byte array
byte[] reply = new byte[lengthOfReply];
forceRead(inputStream, reply);
The method used to read data from an InputStream:
private byte[] forceRead(InputStream inputStream, byte[] result)
throws IOException {
int bytesRead = 0;
int total = result.length;
int remaining = total;
while (remaining > 0)
remaining -= inputStream.read(result, bytesRead, remaining);
return result;
}
The method used to convert a byte array to an integer:
private int byteArrayToInt(byte[] byteArray) {
int result = 0;
for (int i = 0; (i<byteArray.length) && (i<8); i++) {
result |= (byteArray[3-i] & 0xff) << (i << 3);
}
return result;
}
The problem is, that the data is not read in the order it arrives. The first 4 bytes are being read just fine. The rest is mixed up. I made a TCP dump to ensure the data correctly arrives at the client. It seems as if the data is split up into 4 TCP packets. The InputStream returns the first 4 bytes of the first packet, then the entire data of the fourth packet, the last part (starting from "length of last packet") of the second packet and the entire data of the third packet. In this order.
Does anyone have a clue what might cause this issue?

Your logic for reading the byte array is not quite right:
From to the docs:
Reads up to len bytes of data from the input stream into an array of
bytes. An attempt is made to read as many as len bytes, but a smaller
number may be read. The number of bytes actually read is returned as
an integer.
and
The first byte read is stored into element b[off], the next one into
b[off+1], and so on. The number of bytes read is, at most, equal to
len. Let k be the number of bytes actually read; these bytes will be
stored in elements b[off] through b[off+k-1], leaving elements
b[off+k] through b[off+len-1] unaffected.
However, as your bytesRead variable stays at 0 for the whole loop, any data from the inputstream is always written to the beginning of your buffer, overwriting the data already in there.
What will work better (checking for -1 will also ensure that you don't subtract -1 from remaining if the stream runs out of data prematurely which would result in remaining increase, which would mean the loop would run unnecessarily until a buffer overrun would make remaining negative):
while ((bytesRead = inputStream.read(result, total - remaining, remaining)) != -1
&& remaining > 0) {
remaining -= bytesRead;