I'm trying to figure out a way of converting a 4D vector into a bounded long. However, the vector and the resultant long have certain restrictions. The vector itself is composed of 4 integers: The first integer can be anything within Java's capability (so Integer.MIN_VALUE all the way to Integer.MAX_VALUE). The second and fourth integers are always between -2999984 and 2999984 (both inclusive). And finally, the third is always between 0 and 255 (again, both inclusive). So it follows this format:
([Integer min - Integer max], [-2999984 - 2999984], [0 - 255], [-2999984 - 2999984])
That vector needs to be converted to a long between -824629322721380016 and 824629339968358064.
I'm aware that there is probably no function that results in a 1 to 1 matching, but I'm trying to figure out a function that will result in as few collisions as possible.
If you are wondering, these bounds for the vector and long are not arbitrary. As I've tagged the post with Minecraft, I ought to explain why. I'm trying to match a certain blockpos in one dimension with a blockpos in another. The 4D vector is [dimension id, x pos, y pos, z pos] and the resultant long is the serialized form of BlockPos (BlockPos#fromLong). You can see this forums post that sparked my inquiry. I'm asking here because my queston is necessarily MC specific, as it's mainly mathematical and code-based.
I would recommend converting your 4d vector into bits, making that bit representation into a BigInteger and hashing that integer using a hashing algorithm designed for low collisions.
Your number of 'buckets' is effectively the range of the long.
According to this Murmur2 appears to be the best hashing for numbers.:
https://softwareengineering.stackexchange.com/questions/49550/which-hashing-algorithm-is-best-for-uniqueness-and-speed
You can Google for Murmur2 java implementations, but here is one such example at the time of the writing of this answer:
https://github.com/sangupta/murmur
Worth noting that if you could limit your number of dimensions to 65536 (16 bits) - you could have a 1 to 1 hash on bits alone. Possibly could do this by limiting the number of virtual worlds that a user can go into?
Unfortunately this can't be done. A long only holds 64 bits, but your 4D-Vector needs 32 + 23 + 8 + 23 > 64.
If you could restrict your input a little bit to make it fit, you could convert it similar to the following code (example of a 2D-int-Vector <-> long conversion):
long toLong(int int1, int int2) {
return ((long) int1 << 32) | (int2 & (-1L >>> 32));
}
int[] toInts(long l) {
int[] ints = new int[2];
int[0] = (int) (both >> 32);
int[1] = (int) both;
return ints;
}
Related
I need to multiply two 8 byte (64 bit) arrays in the fastest way possible. The byte arrays are little endian. The arrays can be wrapped in a ByteBuffer and treated as little endian to easily resolve a java "long" value that correctly represents the bytes (but not the real nominal value since java longs are 2s compliment).
Java's standard way to handle large math is BigInteger. But that implementation is slow and unnecessary since im very strictly working with 64 bits x 64 bits. In addition, you can't throw the "long" value into one because the nominal value is incorrect, and I can't use the byte array directly because it's little endian. I need to be able to do this without having to use up more memory / CPU to reverse the array. This type of multiplication should be able to execute 1m+ times per second. BigInteger doesn't really come close to meeting that requirement anyway, so I'm trying to do it via splitting the high order bits from the low order bits, but I can't get it working consistently.
The high-order-bits-only code is only working for a subset of longs because even the intermediate addition can overflow. I got my current code from this answer....
high bits of long multiplication in Java?
Is there a more generic pattern for getting hi/lo order bits from 128 bit multiplication? That works for the largest long values?
Edit:
FWIW I'm prepared for the answer to be.. "cant do that in java, do it in c++ and call via JNI". Though I'm hoping someone can give a java solution before it comes to that.
As of Java 9 (which was a bit too new at the time this question was asked), there is now a trivial way to get the upper half of the 128-bit product of two signed 64-bit integers: Math.multiplyHigh
There is a relatively simple conversion from "upper half of signed product" to "upper half unsigned product" (see Hacker's Delight chapter 8), which can be used to implement an unsigned multiply high like this:
static long multiplyHighUnsigned(long x, long y) {
long signedUpperHalf = Math.multiplyHigh(x, y);
return signedUpperHalf + ((x >> 63) & y) + ((y >> 63) & x);
}
This has the potential to be more efficient (on platforms on which multiplyHigh is treated as an intrinsic function by the JIT) than the more manual approach used by the old answer, which I will leave below the line.
It can be done manually without BigInteger by splitting the longs up into two halves, creating the partial products, and then summing them up. Naturally the low half of the sum can be left out.
The partial products overlap, like this:
LL
LH
HL
HH
So the high halves of LH and HL must be added to the high result, and furthermore the low halves of LH and HL together with the high half of LL may carry into the bits of the high half of the result. The low half of LL is not used.
So something like this (only slightly tested):
static long hmul(long x, long y) {
long m32 = 0xffffffffL;
// split
long xl = x & m32;
long xh = x >>> 32;
long yl = y & m32;
long yh = y >>> 32;
// partial products
long t00 = xl * yl;
long t01 = xh * yl;
long t10 = xl * yh;
long t11 = xh * yh;
// resolve sum and carries
// high halves of t10 and t01 overlap with the low half of t11
t11 += (t10 >>> 32) + (t01 >>> 32);
// the sum of the low halves of t10 + t01 plus
// the high half of t00 may carry into the high half of the result
long tc = (t10 & m32) + (t01 & m32) + (t00 >>> 32);
t11 += tc >>> 32;
return t11;
}
This of course treats the input as unsigned, which does not mean they have to be positive in the sense that Java would treat them as positive, you can absolutely input -1501598000831384712L and -735932670715772870L and the right answer comes out, as confirmed by wolfram alpha.
If you are prepared to interface with native code, in C++ with MSVC you could use __umulh, and with GCC/Clang you can make the product as an __uint128_t and just shift it right, the codegen for that is actually fine, it doesn't cause a full 128x128 multiply.
I'm implementing Karatsuba multiplication in Scala (my choice) for an online course. Considering the algorithm is meant to multiply large numbers, I chose the BigInt type which is backed by Java BigInteger. I'd like to implement the algorithm efficiently, which using base 10 arithmetic is copied below from Wikipedia:
procedure karatsuba(num1, num2)
if (num1 < 10) or (num2 < 10)
return num1*num2
/* calculates the size of the numbers */
m = max(size_base10(num1), size_base10(num2))
m2 = floor(m/2)
/* split the digit sequences in the middle */
high1, low1 = split_at(num1, m2)
high2, low2 = split_at(num2, m2)
/* 3 calls made to numbers approximately half the size */
z0 = karatsuba(low1, low2)
z1 = karatsuba((low1 + high1), (low2 + high2))
z2 = karatsuba(high1, high2)
return (z2 * 10 ^ (m2 * 2)) + ((z1 - z2 - z0) * 10 ^ m2) + z0
Given that BigInteger is internally represented as an int[], if I can calculate m2 in terms of the int[], I can use bit shifting to extract the lower and higher halves of the number. Similarly, the last step can be achieved by bit shifting too.
However, it's easier said than done, as I can't seem to wrap my head around the logic. For example, if the max number is 999, the binary representation is 1111100111, lower half is 99 = 1100011, upper half is 9 = 1001. How do I get the above split?
Note:
There is an existing question that shows how to implement using arithmetic on BigInteger, but not bit shifting. Hence, my question is not a duplicate.
To be able to use bit shifting to do the splits and recombination, the base needs to be a power of two. Using two itself, as in the linked answer, is probably reasonable. Then the "length" of the inputs can be found directly with bitLength, and the split could be implemented as:
// x = a + 2^N b
BigInteger b = x.shiftRight(N);
BigInteger a = x.subtract(b.shiftLeft(N));
Where N is the size that a will have in bits.
Given that BigInteger is implemented with 32bit limbs, it makes sense to use 2³² as the base, ensuring that the big shifts involve only the movement of whole integers, and not also the slower code path where the BigInteger is shifted by a value between 1 and 31. This could be accomplished by rounding N to a multiple of 32.
The specific constant in this line,
if (N <= 2000) return x.multiply(y); // optimize this parameter
Should probably not be trusted too much, given that comment. For performance there should be some bound though, otherwise the recursive splitting goes too deeply. For example, when the size of the numbers is 32 or less, it's clearly better to just multiply, but probably a good cut-off is much higher. In this source of BigInteger itself, the cutoff is expressed in terms of the number of limbs instead of bits, and set to 80 (so 2560 bits) - it also has an other threshold above which it switches to 3-way Toom-Cook multiplication instead of Karatsuba multiplication.
I need to multiply two 8 byte (64 bit) arrays in the fastest way possible. The byte arrays are little endian. The arrays can be wrapped in a ByteBuffer and treated as little endian to easily resolve a java "long" value that correctly represents the bytes (but not the real nominal value since java longs are 2s compliment).
Java's standard way to handle large math is BigInteger. But that implementation is slow and unnecessary since im very strictly working with 64 bits x 64 bits. In addition, you can't throw the "long" value into one because the nominal value is incorrect, and I can't use the byte array directly because it's little endian. I need to be able to do this without having to use up more memory / CPU to reverse the array. This type of multiplication should be able to execute 1m+ times per second. BigInteger doesn't really come close to meeting that requirement anyway, so I'm trying to do it via splitting the high order bits from the low order bits, but I can't get it working consistently.
The high-order-bits-only code is only working for a subset of longs because even the intermediate addition can overflow. I got my current code from this answer....
high bits of long multiplication in Java?
Is there a more generic pattern for getting hi/lo order bits from 128 bit multiplication? That works for the largest long values?
Edit:
FWIW I'm prepared for the answer to be.. "cant do that in java, do it in c++ and call via JNI". Though I'm hoping someone can give a java solution before it comes to that.
As of Java 9 (which was a bit too new at the time this question was asked), there is now a trivial way to get the upper half of the 128-bit product of two signed 64-bit integers: Math.multiplyHigh
There is a relatively simple conversion from "upper half of signed product" to "upper half unsigned product" (see Hacker's Delight chapter 8), which can be used to implement an unsigned multiply high like this:
static long multiplyHighUnsigned(long x, long y) {
long signedUpperHalf = Math.multiplyHigh(x, y);
return signedUpperHalf + ((x >> 63) & y) + ((y >> 63) & x);
}
This has the potential to be more efficient (on platforms on which multiplyHigh is treated as an intrinsic function by the JIT) than the more manual approach used by the old answer, which I will leave below the line.
It can be done manually without BigInteger by splitting the longs up into two halves, creating the partial products, and then summing them up. Naturally the low half of the sum can be left out.
The partial products overlap, like this:
LL
LH
HL
HH
So the high halves of LH and HL must be added to the high result, and furthermore the low halves of LH and HL together with the high half of LL may carry into the bits of the high half of the result. The low half of LL is not used.
So something like this (only slightly tested):
static long hmul(long x, long y) {
long m32 = 0xffffffffL;
// split
long xl = x & m32;
long xh = x >>> 32;
long yl = y & m32;
long yh = y >>> 32;
// partial products
long t00 = xl * yl;
long t01 = xh * yl;
long t10 = xl * yh;
long t11 = xh * yh;
// resolve sum and carries
// high halves of t10 and t01 overlap with the low half of t11
t11 += (t10 >>> 32) + (t01 >>> 32);
// the sum of the low halves of t10 + t01 plus
// the high half of t00 may carry into the high half of the result
long tc = (t10 & m32) + (t01 & m32) + (t00 >>> 32);
t11 += tc >>> 32;
return t11;
}
This of course treats the input as unsigned, which does not mean they have to be positive in the sense that Java would treat them as positive, you can absolutely input -1501598000831384712L and -735932670715772870L and the right answer comes out, as confirmed by wolfram alpha.
If you are prepared to interface with native code, in C++ with MSVC you could use __umulh, and with GCC/Clang you can make the product as an __uint128_t and just shift it right, the codegen for that is actually fine, it doesn't cause a full 128x128 multiply.
I'm currently trying to convert a piece of matlab code to java. The purpose of the code is to invert and normalize the image pixels of an image file. In java, the pixels are stored in a byte array. Below is the Matlab code of importance:
inp2=1024.-inp.-min; %inp is the input array (double precision). min is the minimum value in that matrix.
The image is 16 bit, but is using only 10 bits for storage, so that's where the 1024 comes from (2^10). I know definitively that this code works in matlab. However, I'm personally not proficient in matlab, and my java translation isn't behaving the same way as its counterpart.
Below is the method where I've tried inverting the image matrix:
//bitsStored is the bit depth. In this test, it is 10.
//imageBytes is the pixel data in a byte array
public static short[] invert(int bitsStored) {
short min = min(imageBytes);//custom method. Gets the minimum value in the byte array.
short range = (short) (2 << bitsStored);
short[] holder = new short[imageBytes.length];
for (int i = 0; i < imageBytes.length; i++) {
holder[i] = (short) (range - imageBytes[i] - min);
}
imageBytes = holder;
return imageBytes;
}
However, instead of inverting the color channels, the image loses some data and becomes much harsher looking (higher contrast, less blend, etc). What am I doing wrong here?
Let me know if I can make anything clearer for you. Thank you.
UPDATE:
Hi, I have another question regarding this code. Can the above code (fixed to short[] not byte[]) be used in reverse on the same file? As in, if I rerun through this code using an inverted version of the original image, should I get the original input/image from the start of the program? The only problem with it I think is that the min value changes between runs.
byte has range from -128 to 127, it cannot hold 1024 different values. So either you need to use a wider type (like short) to model your points, or your byte array has to be unpacked before processing.
One more thing: double is floating point and it does not play well with integers used in the rest of your code. The following seems better:
short range = 1 << bitsStored; // 2^bitsStored
Correct equation for inversion is:
newValue[i] = maxPossibleValue - currentValue[i]
Your maxPossibleValue is 1024.
Other thing is that you can't have image with depth of 10 bits in array of bytes (cause they've 8 bits)
On your second question about the reversibility of your algorithm.
Your formula looks like result[i] = 1024 - min(data) - data[i] where data ranges from 0 to 1023. Let's imagine that all your data points are 1023. Then min is 1023, so all the result[i] will be -1022.
So the result does not even fit in the same range as the data.
Then, if you run your algorithm with that result array to produce result1, all its points will be 1024 - (-1022) - (-1022) i.e. 3068, and not the original 1023.
So the answer is not, double application of this algorithm does not produce result equal to the input.
Please note that the algorithm mentioned in another answer (maxPossibleValue - currentValue[i]) keeps range and it is reverses when applied twice.
BTW, it should be
short range = (short) (1 << bitsStored);
instead of
short range = (short) (2 << bitsStored);
to produce 2^bitsStored.
I'm tying to make a Linear Congruential random number generator, and I want to select, say the first 16 bits of a 64-bit length hexadecimal value. How Can I do this in java? I've already created a (very) basic generated number based on the time of day.
My formula:
seed = 0x5D588B656C078965L * cal.get(Calendar.HOUR_OF_DAY) + 0x0000000000269EC3;
I just want to select the first 16 bits of this, I tried to think of how I would do this with an integer but the I don't think i can apply the same concepts here. Thanks!
If you want a long that has the first 16 bits and zeroes in the other positions, you can either use a bit mask or shifting.
Assuming by "first 16 bits" you mean the highest-order bits, then the mask looks like this:
long mask = 0xffff000000000000;
so that a '1' is in each bit position you want to retain, and a 0 elsewhere. Then do a logical 'and' of this with your original integer:
long result = seed & mask;
The other way is to shift your original to the right 48 bits, then left again 48 bits.
Bit shift:
long value = seed >>> 48;
or you can save it in an int:
int value = (int)(seed >> 48);
Usually, you would use the last bits of the seed to produce the random number, which makes for a one-way mod operation (ie "more random"), so:
seed & 0xFFFF