I'm writing to a storage format that has uint32, with a max allowed value of "4294967295".
Integer in Java is, of course, just under half that at "2147483647". So internally, I have to use either Long or Guava's UnsignedInteger.
To write to this format, the byte array length needs to be 4, which fits Integer just fine, but converting Long to a byte array requires an array of length 8.
How can I convert a Long or UnsignedInteger representing a max value of "4294967295" as a 4 byte array?
Simply convert it to an 8 byte array and then take only the last 4 bytes:
public static byte[] fromUnsignedInt(long value)
{
byte[] bytes = new byte[8];
ByteBuffer.wrap(bytes).putLong(value);
return Arrays.copyOfRange(bytes, 4, 8);
}
To reverse this you can use the following method:
public static long toUnsignedInt(byte[] bytes)
{
ByteBuffer buffer = ByteBuffer.allocate(8).put(new byte[]{0, 0, 0, 0}).put(bytes);
buffer.position(0);
return buffer.getLong();
}
Note that this method CAN take a negative long or a long that exceed the range of an unsigned int and won't throw a exception in such a case!
You can just cast it to an int and do whatever you do that turns ints into arrays, such as this: (not tested)
public static byte[] getUnsignedInt(long value)
{
byte[] bytes = new byte[4];
ByteBuffer.wrap(bytes).putInt((int)value);
return bytes;
}
Of course if you're putting these things in a ByteBuffer anyway, you might as well do that directly.
The "meaning" or "interpretation" of the top bit is irrelevant if all you're doing it storing it. For example, 4294967295 would be interpreted as -1, but it's really the same number: 0xFFFFFFFF in hexadecimal, so you will get the byte array { 0xFF, 0xFF, 0xFF, 0xFF }.
To reverse it, you could do something like this (not tested)
public static long toUnsignedInt(byte[] bytes)
{
ByteBuffer buffer = ByteBuffer.allocate(4).put(bytes);
buffer.position(0);
return buffer.getInt() & 0xFFFFFFFFL;
}
An answer without the object creation and array copying of the accepted answer... It's easy to do yourself w/ shift operations. See:
import java.io.*;
public class TestUINT32 {
public static void writeLongAsUINT32(long value, OutputStream os) throws IOException {
for (int i=0; i<4; ++i) {
os.write((byte) value);
value = value >> 8;
}
}
public static void main(String[] args) throws IOException {
ByteArrayOutputStream os = new ByteArrayOutputStream();
long value = 0xFFEEDDBB;
writeLongAsUINT32(value, os);
byte[] ba = os.toByteArray();
for (int i=ba.length; i>0; --i) {
System.out.print(String.format("%02X", ba[i-1]));
}
System.out.println();
}
}
Example run:
$ java TestUINT32
FFEEDDBB
Related
I have an array byte[] arr;
ByteArrayOutputStream out = new ByteArrayOutputStream();
byte[] arr = out.toByteArray();
How can I measure the data size in arr (if it was written to disk or transferred via network)?
Are below approaches are correct - they suppose that sizeof(byte) = 1B
int byteCount = out.size();
int byteMsgCount = arr.length;
Yes, by definition the size of a variable of type byte is one byte. So the length of your array is indeed array.length bytes.
out.size() will give you the same value, i.e. the number of bytes that you wrote into the output stream.
[Edit] From cricket_007 comment: if you look at the implementation of size and toByteArray
public synchronized byte toByteArray()[] {
return Arrays.copyOf(buf, count);
}
public synchronized int size() {
return count;
}
... so toByteArray basically copies the current output buffer, up to count bytes. So using size is a better solution.
I need to serialize an array of doubles to base64 in Java. I have following method from C#
public static string DoubleArrayToBase64( double[] dValues ) {
byte[] bytes = new byte[dValues.Length * sizeof( double )];
Buffer.BlockCopy( dValues, 0, bytes, 0, bytes.Length );
return Convert.ToBase64String( bytes );
}
How do I do that in Java? I tried
Byte[] bytes = new Byte[abundaceArray.length * Double.SIZE];
System.arraycopy(abundaceArray, 0, bytes, 0, bytes.length);
abundanceValues = Base64.encodeBase64String(bytes);
however this leads to an IndexOutofBoundsException.
How can I achieve this in Java?
EDIT:
Buffer.BlockCopy copies on byte level, the last paramter is number of bytes. System.arraycopy last parameter is number of elements to copy. So yes it should be abundaceArray.length but then a ArrayStoreException is thrown.
EDIT2:
The base64 string must be the same as the ine created with the c# code!
You get an ArrayStoreException when the array types on the method are not the same primitive, so double to byte will not work. Here is a workaround i patched up that seems to work. I do not know of any method in the java core that does automatic conversion from primitive to byte block :
public class CUSTOM {
public static void main(String[] args) {
double[] arr = new double[]{1.1,1.3};
byte[] barr = toByteArray(arr);
for(byte b: barr){
System.out.println(b);
}
}
public static byte[] toByteArray(double[] from) {
byte[] output = new byte[from.length*Double.SIZE/8]; //this is reprezented in bits
int step = Double.SIZE/8;
int index = 0;
for(double d : from){
for(int i=0 ; i<step ; i++){
long bits = Double.doubleToLongBits(d); // first transform to a primitive that allows bit shifting
byte b = (byte)((bits>>>(i*8)) & 0xFF); // bit shift and keep adding
int currentIndex = i+(index*8);
output[currentIndex] = b;
}
index++;
}
return output;
}
}
The Double.SIZE get 64 which is number of bits I suggest to initialize the array like this
Byte[] bytes = new Byte[abundaceArray.length * 8];
Not sure what this C# function does, but I suspect you should replace this line
System.arraycopy(abundaceArray, 0, bytes, 0, bytes.length);
with this
System.arraycopy(abundaceArray, 0, bytes, 0, abundaceArray.length);
I'm guessing you're using the apache commons Base64 class. That only has methods accepting an array of bytes (the primitive type), not Bytes (object wrapper around primitive type).
It's not clear what type your 'abundaceArray' is - whether it's doubles or Doubles.
Either way, you can't use System.arraycopy to copy between arrays of difference primitive types.
I think your best bet is to serialise your array object to a byte array, then base64 encode that.
eg:
ByteArrayOutputStream b = new ByteArrayOutputStream(); // to store output from serialization in a byte array
ObjectOutputStream o = new ObjectOutputStream(b); // to do the serialization
o.writeObject(abundaceArray); // arrays of primitive types are serializable
String abundanceValues = Base64.encodeBase64String(b.toByteArray());
There is of course an ObjectInputStream for going in the other direction at the other end.
Suppose I have a bit stream which spits one bit at each time, and the stream can stop at any time. What is the idiomatic way to record the output? Assuming my main usage for this data structure is to convert it to 8-bit block ASCII string later. List<Boolean> doesn't sound right because it's messy to convert to 8-bit block bit array. BitSet can't grow dynamically. List<Char> having a problem when the stream stops after spit bits whose number is not a multiple of 8. Any ideas?
I recommend using a ByteBuffer. http://docs.oracle.com/javase/7/docs/api/java/nio/ByteBuffer.html
You can simply construct a BitList using an long[] array and an integer nBits that keeps track of the number of bits:
public class BitList {
private int nBits = 0;
private long[] data = new long[2];
//0 or 1
public void add (byte data) {
if(nBits >= 64*data.length) {
long[] newdata = new long[2*data.length];
for(int i = 0; i < data.length; i++) {
newdata[i] = data[i];
}
this.data = newdata;
}
data[nBits/64] |= data<<(nBits&0x3f);
nBits++;
}
public byte get (int index) {
long val = data[index/64]>>(index&0x3f);
return (byte) (val&0x01);
}
//and so on.
}
Or you might wait until the system has spit out a multiple of eight by packing them into bytes:
public class Packer {
private byte data;
public byte getData () {
byte result = this.data;
this.data = 0;
return result;
}
//only last bit counts (thus bit is 0 or 1)
public void addBit (byte bit) {
this.data <<= 0x01;
this.data |= bit;
}
}
In that case the Packer can be used to ease the implementation since you can use an ArrayList<Byte> and use an integer to keep track of the number of bits (without having to implement add/remove/etc. methods yourself.
You might want to try
List<Byte>
This would most fluidly convert into a byte string, and the bits in each byte are identifiable.
Given a integer value from a string, I want to convert it to 2 byte signed integer.
BigInteger does the job, but I don't know how to grant 2 bytes...
public void handleThisStringValue(String x, String y){
BigInteger bi_x = new BigInteger(x, 10);
BigInteger bi_y = new BigInteger(y, 10);
byte[] byteX = bi_x.toByteArray();
byte[] byteY = bi_y.toByteArray();
}
I noticed that BigInteger.toByteArray() handles negative values which is suitable for me.
Then I need to read those values (negative and positive ones), or saying convert byte[2] to signed int. Any suggestion?
Well, your questions still lacks certain information.
First, Java integers are 32-bit long, so they will not fit into a 2-byte array, you need a 4-byte array, otherwise you are actually dealing with short data type, which is 16-bit long.
Also, not sure if you need to deal with any kind of byte ordering (little endian, big endian).
At any rate, assuming that you are using integers that only fit in 16-bits and big endian byte ordering, you could do something as follows to create the byte array:
public static byte[] toByteArray(String number){
ByteBuffer buffer = ByteBuffer.allocate(4);
buffer.putInt(Integer.parseInt(number));
return Arrays.copyOfRange(buffer.array(), 2, 4); //asumming big endian
}
And as follows to convert it back:
public static int toInteger(byte[] payload){
byte[] data = new byte[4];
System.arraycopy(payload, 0, data, 2, 2);
return ByteBuffer.wrap(data).getInt();
}
You can also change the byte order of the ByteBuffer with the ByteBuffer.order method.
I used it as follows:
byte[] payload = toByteArray("255");
int number = toInteger(payload);
System.out.println(number);
Output is 255
int x = bs[0] | ((int)bs[1] << 8);
if (x >= 0x8000) x -= 0x10000;
// Reverse
bs[0] = (byte)(x & 0xFF);
bs[1] = (byte)((x >> 8) & 0xFF);
You can make the inverse:
new BigInteger(byteX);
new BigInteger(byteY);
It's exactly what you want, and then you can use .intvalue() to get it as an int
The solution is simple, based in posts I found here (thank you for all):
Remember that I wanted a 2 byte integer... so it is a Short!
String val= "-32";
short x = Short.parseShort(val);
byte[] byteX = ByteBuffer.allocate(2).putShort(x).array();
... and it works!
Then, I'm using BigInteger to read it back!
int x1 = new BigInteger(byteX).intValue();
or
short x2 = new BigInteger(x).shortValue();
I want to change a values in byte array to put a long timestamp value in in the MSBs. Can someone tell me whats the best way to do it. I do not want to insert values bit-by-bit which I believe is very inefficient.
long time = System.currentTimeMillis();
Long timeStamp = new Long(time);
byte[] bArray = new byte[128];
What I want is something like:
byte[0-63] = timeStamp.byteValue();
Is something like this possible . What is the best way to edit/insert values in this byte array. since byte is a primitive I dont think there are some direct implementations I can make use of?
Edit:
It seems that System.currentTimeMillis() is faster than Calendar.getTimeInMillis(), so replacing the above code by it.Please correct me if wrong.
There are multiple ways to do it:
Use a ByteBuffer (best option - concise and easy to read):
byte[] bytes = ByteBuffer.allocate(Long.SIZE / Byte.SIZE).putLong(someLong).array();
You can also use DataOutputStream (more verbose):
ByteArrayOutputStream baos = new ByteArrayOutputStream();
DataOutputStream dos = new DataOutputStream(baos);
dos.writeLong(someLong);
dos.close();
byte[] longBytes = baos.toByteArray();
Finally, you can do this manually (taken from the LongSerializer in Hector's code) (harder to read):
byte[] b = new byte[8];
for (int i = 0; i < size; ++i) {
b[i] = (byte) (l >> (size - i - 1 << 3));
}
Then you can append these bytes to your existing array by a simple loop:
// change this, if you want your long to start from
// a different position in the array
int start = 0;
for (int i = 0; i < longBytes.length; i ++) {
bytes[start + i] = longBytes[i];
}
If you want to really get under the hood...
public byte[] longToByteArray(long value) {
return new byte[] {
(byte) (value >> 56),
(byte) (value >> 48),
(byte) (value >> 40),
(byte) (value >> 32),
(byte) (value >> 24),
(byte) (value >> 16),
(byte) (value >> 8),
(byte) value
};
}
For me ByteBuffer and other utils are expensive from time perspective. Here are 2 methods that you can use:
// first method (that is using the second method), it return the array allocated and fulfilled
public byte[] longToByteArray(long value)
{
byte[] array = new byte[8];
longToByteArray(value,array,0);
return array;
}
// this method is useful if you have already allocated the buffer and you want to write the long a specific location in the array.
public void longToByteArray(long value, byte[] array, int startFrom)
{
for (int i=7; i>=0; i--)
{
array[startFrom+7-i] = (byte) (value >> i*8);
}
}
It doesn't look like you can slice a byte array to insert something into a subset without doing it byte by byte. Look at Grab a segment of an array in Java without creating a new array on heap . Basically what I would do is set create a 64 byte array and set the time to it then append a blank 64 byte array to it. Or just do it byte by byte.
I am updating this post because I have just announced a pre-release version of a library that will convert longs to byte arrays (and back again). The library is very small and will convert any java primitive to a byte array.
http://rschilling.wordpress.com/2013/09/26/pre-release-announcement-pend-oreille/
http://code.google.com/p/pend-oreille/
If you use it you can do things like convert long arrays to byte arrays:
Double[] doubles = new Double[1000];
for (int i = 2; i < 1002; i++) {
doubles[i - 2] = (double) i;
}
byte[] resultBytes1 = (byte[]) new PrimitiveHelper(PrimitiveUtil.unbox(doubles))
.asType(byte[].class);
You can also convert a single long value as well.
byte[] resultBytes1 = (byte[]) new PrimitiveHelper(1000l)
.asType(byte[].class);
Feel free to provide some feedback.
Update on October 4, 2013:
I've now released the production of the library http://rschilling.wordpress.com/2013/10/04/pend-oreille-official-1-0-release/