There are special formats (base-128) designed for transmitting integers used in protobufs and elsewhere. They're advantageous when most the integers are small (they need a single byte for smallest numbers and may waste one byte for others).
I wonder if there's something similar for floating point numbers under the assumption that most of them are actually small integers?
To address the answer by Alice: I was thinking about something like
void putCompressedDouble(double x) {
int n = (int) x;
boolean fits = (n == x);
putBoolean(fits);
if (fits) {
putCompressedInt(n);
} else {
putUncompressedLong(Double.doubleToLongBits(x));
}
}
This works (except for the negative zero, which I really don't care about), but it's wasteful in case of fits == true.
It depends on the distribution of your numbers. Magnitude doesn't really matter that much, since its expressed through the exponent field of a float. Its usually the mantissa that contributes the most "weight" in terms of storage.
If your floats are mainly integers, you may gain something by converting to int (via Float.floatToIntBits()), and checking how many trailing zeros there are (for small int values there should be up to 23 trailing zeros). When using a simple scheme to encode small int's, you may implement encoding floats simply as:
int raw = Float.floatToIntBits(f);
raw = Integer.reverse(raw);
encodeAsInt(raw);
(Decoding is simply reversing the process).
What this does is simply move the trailing zeros in the mantissa to the most significant bits of the int representation, which is friendly to encoding schemes devised for small integers.
Same can be applied to double<->long.
Probably not, and this is almost certainly not something you want.
As noted at this stack overflow post, floating point numbers are not stored in a platform independent way in protocol buffers; they are essentially bit for bit representations that are then cast using a union. This means float will take 4 bytes and double 8 bytes. This is almost certainly what you want.
Why? Floating points are not integers. The integers are a well formed group; each number is valid, every bit pattern represents a number, and they exactly represent the integer they are. Floating points cannot represent many important numbers exactly: most floats can't represent 0.1 exactly, for example. The problem of infinities, NAN's, etc etc, all make a compressed format a non-trivial task.
If you have small integers in a float, then convert them to small integers or some fixed point precision format. For example, if you know you only have....4 sigfigs, you can convert from floating point to a fixed point short, saving 2 bytes. Just make sure each end knows how to deal with this type, and you'll be golden.
But any operation that google could do to try and save space in this instance would be both reinventing the wheel and potentially dangerous. Which is probably why they try not to mess with floats at all.
I really like Durandal's solution. Despite its simplicity, it performs pretty well, at least for floats. For doubles with their exponent longer than one byte, some additional bit rearrangement might help. The following table gives the encoding length for numbers with up to D digits, negative numbers are also considered. In each column the first number given the maximum bytes needed while the parenthesized number is the average.
D AS_INT REV_FLOAT REV_DOUBLE BEST
1: 1 (1.0) 2 (1.8) 3 (2.2) 1 (1.0)
2: 2 (1.4) 3 (2.4) 3 (2.8) 2 (1.7)
3: 2 (1.9) 3 (2.9) 4 (3.2) 2 (2.0)
4: 3 (2.2) 4 (3.3) 4 (3.8) 3 (2.6)
5: 3 (2.9) 4 (3.9) 5 (4.1) 3 (3.0)
6: 3 (3.0) 5 (4.2) 5 (4.8) 4 (3.5)
7: 4 (3.9) 5 (4.8) 6 (5.1) 4 (3.9)
8: 4 (4.0) 5 (4.9) 6 (5.8) 5 (4.3)
9: 5 (4.9) 5 (4.9) 6 (6.0) 5 (4.9)
Four different methods were tested:
AS_INT: Simply convert the number to int. This is unusable but gives us a lower bound.
REV_FLOAT: The method by Durandal applied to floats.
REV_DOUBLE: The method by Durandal applied to doubles.
BEST: An improvement of my own method as described in the question. Rather complicated.
Related
This question already has answers here:
Retain precision with double in Java
(24 answers)
Closed 1 year ago.
Is there a data type that can handle/store numbers exceeding type double in Java ?
As far as I know, Java type double can store numbers in range 1.0E+38 to 1.0E-45.
What if I need Java program to read numbers exceedeing digit precision of 1.0E+38 and 1.0E-45? for example 1.0E-180. Currently those numbers are recognized as 0.0 .
Don't ask why I need such ridiculous numbers. I got what was given to me.
As far as I know, Java type double can store numbers in range 1.0E+38 to 1.0E-45.
Incorrect; it can go as far as 1.7976931348623157 * 10^308 and 4.9406564584124654 x 10^-324.
However, there are only at most 2^64 numbers that can be stored in a double (reason: Think about it. Pigeon hole principle). There are an infinite amount of numbers between 0 and 1, let alone between those 2 extremes. doubles work by silently rounding, all the time, to the nearest number that is one of the chosen few 2^64. Let's call those blessed numbers.
These numbers are not equally distributed. Near 0, there are A LOT of these, as you move away from 0 there are fewer and fewer. Eventually (at around 2^52), the distance between any 2 blessed numbers is more than 1.0.
BigDecimal is one solution. There are others (for starters, double CAN represent 1e-108 - you're doing something wrong in your 'translate input data to a double value' code), but keep in mind that doubles at those extremes are incredibly inaccurate.
BD with numbers like that are incredibly slow, and out of the box, BDs cant divide out of the box (for the same reason you can't divide 1 by 3 and get a perfect result: 0.333333... and where does that stop?) - you need to configure it so it cuts off at some point. They're hard to use, but perhaps your only option.
I try to send sensor data continuously from a stm32wb55 to my own android app.
I receive two bytes from a acceleration sensor and convert those correctly on my stm32wb55 to a float with format (XX.XXXXX, float can be negative).
Now I want to send exactly this float to my own android app.
Before, I have send two bytes from type "int or uint" to my android app and tried to convert those the same way I have done already on the stm32wb55. But the values on my screen are up to 50% of cases false. So now I try to send the float value directly, so that no more conversion on my phone is needed.
EDIT: After your contributions, I have forget my poor idea to send a float to my android app. I tried again to send the two byte integers and convert those the right way on my app. Now it works how it should. I have found the solution I needed on this post:2 Chars to Short in C.
By combining the two bytes to a 16-Bit Integer, I just needed 0x00ff & for my LSB like it be used in the answer of the referenced post.
to a float with format (XX.XXXXX, float can be negative).
This is impossible.
floats are a 32-bit IEEE754 floating point number. They do not work like you (apparently) think they do.
Specifically, they are binary constructs with a mantissa system. I'll try to explain why it doesn't work like you think they do: What is 1/3, in decimal? You'll find that you can't write it in decimal no matter how many 'bits' (digits) you use. You'll never quite get it. There's always another 3 to add.
floating point works the same way, but it's in binary (base 2) and not decimal (base 10). That means there are numbers that work great in decimal (such as 1/10th which in decimal is 0.1, with perfect accuracy) but which are like 1/3 in binary: No matter how many bits you use, you'll never get it.
Here's another way to think about it: 32-bit, so, there are only 2^32 (about 4 billion) different values. In other words, of all the numbers in existence (and there is an infinite infinity of them: There are infinite numbers, and within any 2 consecutive numbers, another infinity of numbers), only at most 4 billion are blessed: 4 billion of all numbers in existence are representable by a float value. If you try to represent a number that isn't blessed with a float, then java / your CPU will just round it to the nearest blessed number and gives you no real opportunity to deal with the error (after all, how would you represent the error? It is rather unlikely to be blessed, either).
Thus, say, '12.34567'? That's not a blessed number - therefore, your float cannot possibly represent that. It'll instead be a number very close to it, and probably a number that would round to 12.34567 if you round it to 5 digits.
send exactly this float to my own android app.
So, no, you don't want to do that. You want to send 12.34567 to your android app, not the 32 bits that represent it. Unless you intend for the android side of the app to do the rounding, which you probably should. Note that I bet there are numbers that fit the 'XX.YYYYY' pattern that just do not 'work' as a float (they round such that you're off by 1). If that's a problem, don't use floats (use doubles where I doubt that you'll find an XX.YYYYY that doesn't have a blessed number such that it rounds correctly due to having more bits to work with, or use a string, or use 2 ints, or use a single int, and have an agreement that both sides know that the int 1234567 represents 12.34567).
That last one sounds like the most convenient trick for you here, but it's hard to tell as you haven't provided much detail.
Something like (but note that the float may be off by 1 or so!):
sender side:
double v = theFloat; // doubles have less error
int z = (int) (v * 100000);
sendToPhone(z);
receiver side:
int z = getFromDevice();
double v = z;
v /= 100000;
float theFloat = (float) v;
The above will end up automatically rounding off (rounding down for positive numbers and up for negative numbers) any digits after the floating point beyond the 5 you want), and can deal with numbers up to plus or minus 21473.99999. Sounds like that'll easily cover your needs.
NB: You'll have to write the 'multiply by 100000 and then convert to an int32' code for your stm32wb55, the above is how you'd write it if the stm32wb55 was programmed in java, which I would assume it isn't. The 'go to double before multiplying by 100000 is a probably irrelevant optimization, I wouldn't be too worried if you can't do that. Note that CPUs are not guaranteed to use the exact same IEEE754 representation for floats/doubles that java does, which is why you should definitely not attempt to send the value as a float/double across the bluetooth channel, but as something universally agreed upon, such as 'a 2's complement 32-bit integer value'.
A question of mine was recently closed as a duplicate, but that didn't help me completely. My new and a more specific question is:
Can all values(whole numbers, without any decimal part) smaller than 1.7e308 and greater than 0 be stored in a double data type, as 1.7e308 is the maximum value of a double data type? Because I don't want a decimal numeral, but a large, non-decimal number so large that can't be represented even by long long.
Can all whole numbers smaller than 1.7e308 and greater than 0 be stored in a double data type
The simple answer is No.
There are various ways to come to this conclusion, but here is one that doesn't even depend on an understanding of floating point number formats.
We know that double is a 64 bit representation.
Therefore, there can be at most 264 distinct double values: that is about 1.8 x 1019
You are asking if a double can represent all integers between zero and 1.7 x 10308.
That is 1.7 x 10308 distinct values.
1.7 x 10308 is greater than 1.8 x 1019.
Therefore what you are asking is impossible.
Your best simple option is to use BigInteger.
But you said this:
... but due to slow operations on BigIntegers, I'm keeping it a last choice. It takes about a second to multiply two 4-digit numbers.
That is simply not true. If you have come to that conclusion by benchmarking, then there is something very wrong with your methodology.
Multiplying 2 x 4 digit numbers using BigInteger should take less than a microsecond.
All floating type numbers (halfs/floats/doubles/long doubles/etc) are composed of a mantissa and an exponent.
Suppose you have 1.7e308, 1.7 is the mantissa while 308 is the exponent. You can't exactly separate the two in a float. This is because every float is represented as a composition of the aforementioned in memory. Hence you can't have a "non-decimal" float.
This question already has answers here:
Java float 123.129456 to 123.12 without rounding
(5 answers)
How to round a number to n decimal places in Java
(39 answers)
Closed 5 years ago.
Can I reduce the precision of a float number?
In all the searching I've been doing I saw only how to reduce the precision for printing the number. I do not need to print it.
I want, for example, to convert 13.2836 to 13.28. Without even rounding it.
Is it possible?
The suggested answer from the system is not what I am looking for. It also deals with printing the value and I want to have a float.
There isn't really a way to do it, with good reason. While john16384's answer alludes to this, his answer doesn't make the problem clear... so probably you'll try it, it won't do what you want, and perhaps you still won't know why...
The problem is that while we think in decimal and expect that the decimal point is controlled by a power-of-10 exponent, typical floating point implementations (including Java float) use a power-of-2 exponent. Why does it matter?
You know that to represent 1/3 in decimal you'd say 0.3(repeating) - so if you have a limited number of decimal digits, you can't really represent 1/3. When the exponent is 2 instead of 10, you can't really represent 1/5 either, or a lot of other numbers that you could represent exactly in decimal.
As it happens .28 is one of those numbers. So you could multiply by 100, pass the result to floor, and divide by 100, but when this gets converted back to a float, the resulting value will be a little different from .28 and so, if you then check its value, you'll still see more than 2 decimal places.
The solution would be to use something like BigDecimal that can exactly represent decimal values of a given precision.
The standard warnings about doing precision arithmetic with floats applies, but you can do this:
float f = 13.2836;
f = Math.floor(f * 100) / 100;
if you need to save memory in some part of your calculation, And your numbers are smaller than 2^15/100 (range short), you can do the following.
Part of this taken from this post https://stackoverflow.com/a/25201407/7256243.
float number = 1.2345667f;
number= (short)(100*number);
number=(float)(number/100);
You only need to rememeber that the short's are 100 times larger.
Most answers went straight to how do represent floats more accurately, which is strange because you're asking:
Can I reduce the precision of a float number
Which is the exact opposite. So I'll try to answer this.
However there are several way to "reduce precision":
Reduce precision to gain performance
Reduce memory footprint
Round / floor arbitrarily
Make the number more "fuzzy"
Reduce the number of digits after the coma
I'll tackle those separately.
Reduce precision to gain performance
Just to get it out of the way: simply because you're dropping precision off of your calculations on a float, doesn't mean it'll be any faster. Quite the contrary. This answer by #john16384:
f = Math.floor(f * 100) / 100;
Only adds up computation time. If you know the number of significant digits from the result is low, don't bother removing them, just carry that information with the number:
public class Number WithSignificantDigits {
private float value;
private int significantdigits;
(implement basic operations here, but don't floor/round anywhere)
}
If you're doing this because you're worried about performance: stop it now, just use the full precision. If not, read on.
Reduce memory footprint
To actually store a number with less precision, you need to move away from float.
One such representation is using an int with a fixed point convention (i.e. the last 2 digits are past the coma).
If you're trying to save on storage space, do this. If not, read on.
Round / floor arbitrarily
To keep using float, but drop its precision, several options exist:
#john16384 proposed:
`f = Math.floor(f * 100) / 100;`
Or even
f = ((int) (f*100)) / 100.;
If the answer is this, your question is a duplicate. If not, read on.
Make the number more "fuzzy"
Since you just want to lose precision, but haven't stated how much, you could do with bitwise shifts:
float v = 0;
int bits = Float.floatToIntBits(v);
bits = bits >> 7; // Precision lost here
float truncated = Float.intBitsToFloat(bits);
Use 7 bitshifts to reduce precision to nearest 1/128th (close enough to 1/100)
Use 10 bitshifts to reduce precision to nearest 1/1024th (close enough to 1/1000)
I haven't tested performance of those, but If your read this, you did not care.
If you want to lose precision, and you don't care about formatting (numbers may stil have a large number of digits after the coma, like 0,9765625 instead of 1), do this. If you care about formatting and want a limited number of digits after the coma, read on.
Reduce the number of digits after the coma
For this you can:
Follow #Mark Adelsberger's suggestion of BigDecimals, or
Store as a String (yuk)
Because floats or doubles won't let you do this in most cases.
I understand that using a short in Java we can store a minimum value of -32,768 and a maximum value of 32,767 (inclusive).
And using an int we can store a minimum value of -2^31 and a maximum value of 2^31-1
Question: If I have an int[] numbers and the numbers I can store are positive up to 10Million.
Is it possible to somehow store these numbers without having to use 4 bytes for each? I am wondering if for a specific "small" range there might be some "hack/trick" so that I could use less memory than numbers.length*4
You could attempt to use a smaller number of bits by using masking or bit-operations to represent each number, and then perform a sign-extension later on if you wish to get the full number of bits. This kind of operation is done on a system-architecture level in nearly all computer systems today.
It may help you to research 2's Complement, which seems to be what you are going for... And possibly Sign Extension for good measure.
Typically, in high-level languages an int is represented by the basic size of the processor register. ex) 8, 16, 32, or 64-bits.
If you use a 2's-Complement method, you could easily account for the full spectrum of positive and negative numbers if needed. This is also very easy on the hardware, because you only have to invert all the bits and then add 1, which may prove to give you a big performance increase over other possible methods.
How 2's Complement Works:
Get –N by inversing all bits and then
add 1
That is, get 1-complement of N and then add 1 to it.
For example with 8-bit words:
9 = 00001001
-9 = 11110111 (11110110 + 1)
Easily and efficiently in hardware
(inverse and then +1)
• An n-bit word can be used to represent numbers
from -2^(N-1) to +(2^(N-1) - 1)
UPDATE: Bit-operations to represent larger numbers.
If you are trying to get a larger number, say 1,000,000 as in your comment, then you can use a Bitwise left-shift operation to then extract the number by increasing your current number by the appropriate power of 2.
9 (base 10): 00000000000000000000000000001001 (base 2)
--------------------------------
9 << 2 (base 10): 00000000000000000000000000100100 (base 2) = 36 (base 10)
You could also try:
(Zero-fill right shift)
This operator shifts the first operand the specified number of bits to the right. Excess bits shifted off to the right are discarded. Zero bits are shifted in from the left. The sign bit becomes 0, so the result is always non-negative.
For non-negative numbers, zero-fill right shift and sign-propagating right shift yield the same result. For example, 9 >>> 2 yields 2, the same as 9 >> 2:
9 (base 10): 00000000000000000000000000001001 (base 2)
--------------------------------
9 >>> 2 (base 10): 00000000000000000000000000000010 (base 2) = 2 (base 10)
However, this is not the case for negative numbers. For example, -9 >>> 2 yields 1073741821, which is different than -9 >> 2 (which yields -3):
-9 (base 10): 11111111111111111111111111110111 (base 2)
--------------------------------
-9 >>> 2 (base 10): 00111111111111111111111111111101 (base 2) = 1073741821 (base 10)
As others have stated in the comments, you could actually hamper your overall performance in the long-run if you are attempting to manipulate data that is not specifically word/double/etc-aligned. This is because your hardware will have to work a bit harder to try and piece together what you truly need.
Just another thought. One parameter is the range of numbers you have. But also other properties can help save storage. For example, when you know that each number will be divisible by some multiple of 8, you need not store the lower 3 bits, since you know they are 0 all the time. (This is how the JVM stores "compressed" references.)
Or, to take another possible scenario: When you store prime numbers, then all of them (except 2) will be odd. So no need to store the lowest bit, as it is always 1. Of course you need to handle 2 separately. A similar trick is used in floating point representations: since the first bit of the mantissa of a non-null number is always 1, it is not stored at all, thus increasing precision by 1 bit.
One solution is to use bit manipulation and use a number of bits of your choosing to store a single number. Say you select to use 5 bits. You can then store 4 such numbers in 4 bytes. You need to pack and unpack the bits into an integer when operations need to be done.
You need to decide if you want to deal with negative numbers in which case you need to store a sign bit.
To make it easier to use, you need to create a class that will conceal the nitty-gritty details via get and store operations.
In light of the questions about performance, as is often the case, we are trading space for performance or vise versa. Depending on the situation various optimization techniques can be used to minimize the number of CPU cycles.
That said, is there a need for such optimization in the first place? If so, is it at the memory level or storage level? Could we use a generic mechanism such as compression to take care of this instead of using special techniques?