BigDecimal.setScale(..) changes precisions to 0 - java

I'm using java Corretto 11 and trying to serialize BigDecimal value into avro format.
The value:
BigDecimal v = BigDecimal.valueOf(16858.7109375)) //scale = 7, precision = 12
then round it
v.setScale(5, RoundingMode.HALF_UP) //scale = 5, precision = 0
It happens because BigDecimal's internals call static factory method
public static BigDecimal valueOf(long unscaledVal, int scale) {
if (scale == 0)
return valueOf(unscaledVal);
else if (unscaledVal == 0) {
return zeroValueOf(scale);
}
return new BigDecimal(unscaledVal == INFLATED ?
INFLATED_BIGINT : null,
unscaledVal, scale, 0); //HERE it sets precision = 0
}
Not a big deal.. however avro's validator and converter has different opinion about it :)
//org.apache.avro.Conversions (lib avro-1.10.0)
private static BigDecimal validate(final LogicalTypes.Decimal decimal, BigDecimal value) { //scale = 5, precision = 0
... //nothing happens here
int precision = decimal.getPrecision(); // 9 as defined in avro schema
int valuePrecision = value.precision(); // this changes internal precision to 10 (total digits count after scaling)
if (valuePrecision > precision) { //this is true, and so serialization fails
if (scaleAdjusted) {
throw new AvroTypeException("Cannot encode decimal with precision " + valuePrecision + " as max precision "
+ precision + ". This is after safely adjusting scale from " + valueScale + " to required " + scale);
} else {
throw new AvroTypeException(
"Cannot encode decimal with precision " + valuePrecision + " as max precision " + precision);
}
}
I'm not sure how to fix it.. Is this a bug of BigDecimals internal functions or avro converter? Can I hack it somehow?

BigDecimal a = BigDecimal.valueOf(16858.7109375); final
BigDecimal b = a.setScale(5, RoundingMode.HALF_UP);
System.out.println("Scale: " + b.scale() + ", precision: " + b.precision());
//Scale: 5, precision: 10
It is true, that internally BigDecimal b is showing precision zero, but for each public contract, you will see 10.
In your Avro, if you have "precision": 9 (max precision) Avro validation will throw an exception because it cannot do downsizing in any way.
You can change your Avro to add bigger max precision (but you have to start from consumers first), or you can just ensure that bigger number will not occur there

Related

Unable to parse very long binary string to numeric type

Folks, I am trying to parse an extremely long binary String to its decimal equivalent, but its throwing the NumberFormatException.
String s = "1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111" +
"1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111" +
"1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111" +
"1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111" +
"1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111" +
"1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111" +
"1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111" +
"1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111" +
"1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111" +
"1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111" +
"1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111" +
"1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111" +
"1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111" +
"1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111" +
"111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111" +
"1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111" +
"1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111" +
"1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111" +
"1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111" +
"1111111111111111"; //String length is 1969
long n = Long.parseLong(s, 2); //line no. 25
System.out.println(n);
But it's giving the below-mentioned Runtime error:
Exception in thread "main" java.lang.NumberFormatException: For input string: "1111[...]111"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:592)
at Check2.main(Check2.java:25)
Tried using BigInteger,valueOf() methods too, but all efforts are going in vain.
Please do let me know if there's any other way to achieve the desired result.
The java long data type has a minimum value of -9,223,372,036,854,775,808 and a maximum value of 9,223,372,036,854,775,807. The number you have is much greater than the maximum limit allowed.
You can use BigInteger for your String instead.
BigInteger result = new BigInteger(inputString, 2);
Both int and long have a maximum (and miminum) value, see the documentation for int and long accordingly.
The decimal representation of your binary string exceeds these limits.
You might use a BigDecimal instead to parse the String.
It provides a constructor accepting a String with the number and the radix to use. As your string is in binary format, you should pass the radix value of 2.
BigInteger myValue = new BigInteger(s, 2);
An answer to the amendment:
If you want to calculate with BigInteger you have to do all operations with BigInteger. In the code fragment
BigInteger n = new BigInteger(s, 2);
int count = 0;
while (n.intValue() != 0) {
if (n.intValue() % 2 == 0) {
n = BigInteger.valueOf(n.intValue() / 2);
count++;
} else {
n = BigInteger.valueOf(n.intValue() - 1);
count++;
}
}
System.out.println(count);
you loose important information every time you call n.intValue() and every time you create a new BigInteger from an int (actually a long, the public BigInteger.valueOf() only accepts a long.)
This is because an int can only store 32 bits, the value you start with however has 1969 bits. n.intValue() extracts the lowest 32 bits, and after BigInteger.valueOf(n.intValue() - 1) / BigInteger.valueOf(n.intValue() / 2) all but those last 32 bits are lost.
The code works if you replace it with
BigInteger n = new BigInteger(s, 2);
int count = 0;
while (!n.equals(BigInteger.ZERO)) {
count++;
if (!n.testBit(0)) {
n = n.divide(BigInteger.TWO);
} else {
n = n.subtract(BigInteger.ONE);
}
}
return count;
Why does your code lead to an endless loop?
The endless loop arises from the fact that the int value that has all bits set is -1.
Your original loop produces these values:
-1 is odd, therefore it subtracts 1 which gives -2
-2 is even, the value is divided by 2 which gives -1
You can use this following method BigInteger() for integer conversion:
BigInteger bi = new BigInteger(inputstring, 2);
or you can use BigDecimal() like so:
BigDecimal bd1 = new BigDecimal(inputstring.charAt(0)=='1'?1:0);
BigDecimal two = new BigDecimal(2);
for (int i = 1; i<inputstring.length(); i++) {
bd1 = bd1.multiply(two);
bd1 = bd1.add(new BigDecimal(inputstring.charAt(i)=='1'?1:0));
}
System.out.println("Big decimal number is"+ bd1);

Convert double into BigRational (two BigInteger for numerator/denominator)

I have a custom made BigRational class in java.
It is implemented as two BigInteger, representing numerator and denominator.
I have a "from string" method that take input in the form "-1234/43"
but I would like to implement a from double/from float;
I'm not scare of generating a very large number, but I would like to keep all the precision present in the floating point representation; thus if I converted them in some decimal representation I would lose precision thanks to rounding.
-How do I generate a pair of BigIntegers that interpreted as numerator/denominator represents the same exact number as a given float/double?
(Hopefully by being in Java I do not need to worry about bigendian/littleendian, but I would like a confermation too)
So, thanks to a good friend I have found a good solution, so I will post it here for anyone in need.
It is not using any string representation so it should also be quite on the fast side.
I have tested it "reasonably" and It seams to work and to keep the exact representation.
Of course, we should still add some 'if' to handle NANs.
final static int mantissaBits=53;
public static BigRational from(double num){
int exponent=Math.getExponent(num);
long man=Math.round(Math.scalb(num, mantissaBits-exponent));
long den=Math.round(Math.scalb(1.0, mantissaBits-exponent));
return new BigRational(BigInteger.valueOf(man),BigInteger.valueOf(den));
}
Caveat: Not all numbers are rational, e.g. PI is not a rational number. However, given that double (and float) have limited precision, there are a limited number of digits in a floating-point value, so you can always find a rational number for that. E.g. Math.PI is a double with the value 3.141592653589793. That number is the rational number 3_141_592_653_589_793 / 1_000_000_000_000_000.
Understanding the caveat that floating-point values aren't accurate, you can find the rational number with the help of BigDecimal, then normalize the rational number using BigInteger.gcd().
Like this:
static void printAsRational(double value) {
printAsRational(BigDecimal.valueOf(value));
}
static void printAsRational(float value) {
printAsRational(new BigDecimal(Float.toString(value)));
}
static void printAsRational(BigDecimal value) {
BigInteger numerator, denominator;
if (value.signum() == 0) {
// Zero is 0 / 1
numerator = BigInteger.ZERO;
denominator = BigInteger.ONE;
} else {
BigDecimal bd = value.stripTrailingZeros(); // E.g. 1.20 -> 1.2
if (bd.scale() < 0)
bd = bd.setScale(0); // E.g. 1.7e3 -> 1700
numerator = bd.unscaledValue(); // E.g. 1.25 -> 125
denominator = BigDecimal.valueOf(1, -bd.scale()).toBigInteger(); // E.g. 1.25 -> 100
// Normalize, e.g. 12/8 -> 3/2
BigInteger gcd = numerator.gcd(denominator);
if (! gcd.equals(BigInteger.ONE)) {
numerator = numerator.divide(gcd);
denominator = denominator.divide(gcd);
}
}
System.out.println(value + " = " + numerator + " / " + denominator);
}
Tests
printAsRational(Math.PI);
printAsRational(Math.E);
printAsRational(1.25);
printAsRational(1);
printAsRational(0);
printAsRational(-1.25);
printAsRational(1.25e9);
printAsRational(1.25e-9);
Output
3.141592653589793 = 3141592653589793 / 1000000000000000
2.718281828459045 = 543656365691809 / 200000000000000
1.25 = 5 / 4
1.0 = 1 / 1
0.0 = 0 / 1
-1.25 = -5 / 4
1.25E+9 = 1250000000 / 1
1.25E-9 = 1 / 800000000

How to determine whether number is exactly representable in floating-point?

Since float numbers are base-2 numeral system then it's not possible to represent 0.24F directly as the same it's not possible to represent 1/3 in decimal system without recurring decimal period i.e. 1/3=0.3333... or 0.(3).
So the float number 0.24F when printed back to decimal representation is shown as 0.23 with a change due to rounding:
println(0.24F) => 0.23999999463558197021484375
while 0.25F can be shown directly:
println(0.25F) => 0.25
But how can I determine that a number is exactly representable?
isExactFloat(0.25F) ==> true
isExactFloat(0.24F) ==> false
Maybe Java API has already some function to do that?
UPD
Here is a code which shows float numbers in range [-4, 4] with their internal representation:
public class FloatDestructure {
public static void main(String[] args) {
BigDecimal dec = BigDecimal.valueOf(-4000L, 3);
BigDecimal incr = BigDecimal.valueOf(1L, 3);
for (int i = 0; i <= 8000; i++) {
double dbl = dec.doubleValue();
floatDestuct(dbl, dec);
dec = dec.add(incr);
}
}
static boolean isExactFloat(double d) { return d == (float) d; }
static void floatDestuct(double val, BigDecimal dec) {
float value = (float) val;
int bits = Float.floatToIntBits(value);
int sign = bits >>> 31;
int exp = (bits >>> 23 & ((1 << 8) - 1)) - ((1 << 7) - 1);
int mantissa = bits & ((1 << 23) - 1);
float backToFloat = Float.intBitsToFloat((sign << 31) | (exp + ((1 << 7) - 1)) << 23 | mantissa);
boolean exactFloat = isExactFloat(val);
boolean exactFloatStr = Double.toString(value).length() <= 7;
System.out.println(dec.toString() + " " + (double) val + " " + (double) value + " sign: " + sign + " exp: " + exp + " mantissa: " + mantissa + " " + Integer.toBinaryString(mantissa) + " " + (double) backToFloat + " " + exactFloat + " " + exactFloatStr);
}
}
When mantissa is zero then the float is definitely exact. But in other cases like -0.375 or -1.625 it's not so clear.
In general, this is not possible.
As soon as the number is converted to a float or double, it is just an approximation of the number. So your input to isexactfloat() would not be exact...
If you have the exact version of floating point number in e.g. string format, then it would be possible to devise a function that could tell you if the float or double exactly represents the string formatted number or not. See the comment below by Carlos Heurberger on how to implement such a function.
I would like to share this function here.
// Determine whether number is exactly representable in double.
// i.e., No rounding to an approximation during the conversion.
// Results are valid for numbers in the range [2^-24, 2^52].
public static boolean isExactFloat(double val) {
int exp2 = Math.getExponent(val);
int exp10 = (int) Math.floor(Math.log10(Math.abs(val)));
// check for any mismatch between the exact decimal and
// the round-trip representation.
int rightmost_bits = (52 - exp2) - (16 - exp10);
// create bitmask for rightmost bits
long mask = (1L << rightmost_bits) - 1;
// test if all rightmost bits are 0's (i.e., no rounding)
return (Double.doubleToLongBits(val) & mask) == 0;
}
Edit: the above function could be even shorter
public static boolean isExactFloat(double val) {
int exp2 = Math.getExponent(val);
int exp10 = (int) Math.floor(Math.log10(Math.abs(val)));
long bits = Double.doubleToLongBits(val);
// test if at least n rightmost bits are 0's (i.e., no rounding)
return Long.numberOfTrailingZeros(bits) >= 36 - exp2 + exp10;
}
Demo
Create a BigDecimal from it and catch java.lang.ArithmeticException which it will throw if there is a non-terminating decimal expansion.
Java double can only represent terminating binary fractions. Doing the conversion to double may hide issues, so I think it is better to work from the String representation. The conversion to BigDecimal is exact if the String represents a number. So is conversion from float or double to BigDecimal. Here are test functions for exact representation as float or double:
public static boolean isExactDouble(String data) {
BigDecimal rawBD = new BigDecimal(data);
double d = rawBD.doubleValue();
BigDecimal cookedBD = new BigDecimal(d);
return cookedBD.compareTo(rawBD) == 0;
}
public static boolean isExactFloat(String data) {
BigDecimal rawBD = new BigDecimal(data);
float d = rawBD.floatValue();
BigDecimal cookedBD = new BigDecimal(d);
return cookedBD.compareTo(rawBD) == 0;
}
It is not clear whether your issue has to do with precision (representing 0.24 accurately) or recurring numbers, like 1 / 3.0.
In general precision issues will always creep in if you use the conventional floating point representations.
If precision is a real problem for you, you should look at using BigDecimal. While not as flexible as double it has other advantages like arbitrary precision, and you can also control the rounding behaviour in non-exact calculations (like recurring decimal values).
If all you are after is precision control, you might want to look at the Apache Commons Math Precision class.
You could just compare the double and the float?
public static boolean isExactFloat(double d, float f) {
return d == f;
}
Demo

How do I format double input in Java WITHOUT rounding it?

I have read this question Round a double to 2 decimal places It shows how to round number. What I want is just simple formatting, printing only two decimal places.
What I have and what I tried:
double res = 24.695999999999998;
DecimalFormat df = new DecimalFormat("####0.00");
System.out.println("Value: " + df.format(res)); //prints 24.70 and I want 24.69
System.out.println("Total: " + String.format( "%.2f", res )); //prints 24.70
So when I have 24.695999999999998 I want to format it as 24.69
You need to take the floor of the double value first - then format it.
Math.floor(double)
Returns the largest (closest to positive infinity) double value that is less than or equal to the argument and is equal to a mathematical integer.
So use something like:
double v = Math.floor(res * 100) / 100.0;
Other alternatives include using BigDecimal.
public void test() {
double d = 0.29;
System.out.println("d=" + d);
System.out.println("floor(d*100)/100=" + Math.floor(d * 100) / 100);
System.out.println("BigDecimal d=" + BigDecimal.valueOf(d).movePointRight(2).round(MathContext.UNLIMITED).movePointLeft(2));
}
prints
d=0.29
floor(d*100)/100=0.28
BigDecimal d=0.29
Multiply the number by 100 and cast it to an integer. This cuts off all the decimal spaces except the two you want. Divide the result by 100.00. (24.69).
int temp = (int)(res * 100);
double result = temp / 100.00;
or the same thing in one line of code:
double result = ((int)(res * 100)) / 100.00;
In addition to using Math.floor(double) and calculating a scale (e.g. * 100 and then / 100.0 for two decimal points) you could use BigDecimal, then you can invoke setScale(int, int) like
double res = 24.695999999999998;
BigDecimal bd = BigDecimal.valueOf(res);
bd = bd.setScale(2, RoundingMode.DOWN);
System.out.println("Value: " + bd);
Which will also give you (the requested)
Value: 24.69

Determine Number of Decimal Place using BigDecimal

I was interested to have the following getNumberOfDecimalPlace function:
System.out.println("0 = " + Utils.getNumberOfDecimalPlace(0)); // 0
System.out.println("1.0 = " + Utils.getNumberOfDecimalPlace(1.0)); // 0
System.out.println("1.01 = " + Utils.getNumberOfDecimalPlace(1.01)); // 2
System.out.println("1.012 = " + Utils.getNumberOfDecimalPlace(1.012)); // 3
System.out.println("0.01 = " + Utils.getNumberOfDecimalPlace(0.01)); // 2
System.out.println("0.012 = " + Utils.getNumberOfDecimalPlace(0.012)); // 3
May I know how can I implement getNumberOfDecimalPlace, by using BigDecimal?
The following code doesn't work as expected:
public static int getNumberOfDecimalPlace(double value) {
final BigDecimal bigDecimal = new BigDecimal("" + value);
final String s = bigDecimal.toPlainString();
System.out.println(s);
final int index = s.indexOf('.');
if (index < 0) {
return 0;
}
return s.length() - 1 - index;
}
The following get printed :
0.0
0 = 1
1.0
1.0 = 1
1.01
1.01 = 2
1.012
1.012 = 3
0.01
0.01 = 2
0.012
0.012 = 3
However, for case 0, 1.0, it doesn't work well. I expect, "0" as result. But they turned out to be "0.0" and "1.0". This will return "1" as result.
Combining Turismo, Robert and user1777653's answers, we've got:
int getNumberOfDecimalPlaces(BigDecimal bigDecimal) {
return Math.max(0, bigDecimal.stripTrailingZeros().scale());
}
stripTrailingZeros() ensures that trailing zeros are not counted (e.g. 1.0 has 0 decimal places).
scale() is more efficient than String.indexOf().
A negative scale() represents zero decimal places.
There you have it, the best of both worlds.
This code:
int getNumberOfDecimalPlaces(BigDecimal bigDecimal) {
String string = bigDecimal.stripTrailingZeros().toPlainString();
int index = string.indexOf(".");
return index < 0 ? 0 : string.length() - index - 1;
}
... passes these tests:
assertThat(getNumberOfDecimalPlaces(new BigDecimal("0.001")), equalTo(3));
assertThat(getNumberOfDecimalPlaces(new BigDecimal("0.01")), equalTo(2));
assertThat(getNumberOfDecimalPlaces(new BigDecimal("0.1")), equalTo(1));
assertThat(getNumberOfDecimalPlaces(new BigDecimal("1.000")), equalTo(0));
assertThat(getNumberOfDecimalPlaces(new BigDecimal("1.00")), equalTo(0));
assertThat(getNumberOfDecimalPlaces(new BigDecimal("1.0")), equalTo(0));
assertThat(getNumberOfDecimalPlaces(new BigDecimal("1")), equalTo(0));
assertThat(getNumberOfDecimalPlaces(new BigDecimal("10")), equalTo(0));
assertThat(getNumberOfDecimalPlaces(new BigDecimal("10.1")), equalTo(1));
assertThat(getNumberOfDecimalPlaces(new BigDecimal("10.01")), equalTo(2));
assertThat(getNumberOfDecimalPlaces(new BigDecimal("10.001")), equalTo(3));
... if that is indeed what you want. The other replies are correct, you have to use BigDecimal all the way through for this rather than double/float.
Without having to convert to String, it should be more efficient to use the scale directly:
private int getNumberOfDecimalPlaces(BigDecimal bigDecimal)
{
int scale = bigDecimal.stripTrailingZeros().scale();
return scale>0?scale:0;
}
That should do it
int getNumberOfDecimalPlace(BigDecimal number) {
int scale = number.stripTrailingZeros().scale();
return scale > 0 ? scale : 0;
}
If you really get doubles i recommend formating them first as strings before creating the BigDecimal. At least that has worked for me: How to check if a double has at most n decimal places?
Depending on how many digits you expect you can either use standard formating like
String.valueOf(doubleValue);
or you could use specialised formatting to avoid exponential format
DecimalFormat decimalFormat = new DecimalFormat();
decimalFormat.setMaximumIntegerDigits(Integer.MAX_VALUE);
// don't use grouping for numeric-type cells
decimalFormat.setGroupingUsed(false);
decimalFormat.setDecimalFormatSymbols(new DecimalFormatSymbols(Locale.US));
value = decimalFormat.format(numericValue);
When you have a BigDecimal you can simply call scale() to get the number of decimal places.
It's not your code that's wrong, but your expectations. double is based on a binary floating point representation and completely unfit for accurately representing decimal fractions. Decimal 0.1 e.g. has an infinite number of digits when represented in binary, thus it gets truncated and when converted back to decimal, you get erros in the least significant digits.
If you use BigDecimal exclusively, your code will work as expected.
Try this:
Math.floor(Math.log(x) / Math.log(10))
0.001 = -3
0.01 = -2
0.1 = -1
1 = 0
10 = 1
100 = 2
How about having a look at the javadoc of BigDecimal. I'm not sure, but I'd give getScale and getPercision a try.
The best way to get a BigDecimal with a specified number of decimal places is by using the setscale method over it. Personally I like to also use the rounding version of the method (see the link below):
http://java.sun.com/j2se/1.5.0/docs/api/java/math/BigDecimal.html#setScale(int,%20int)
If you're wanting to get the number of decimal positions that a BigDecimal is currently set at call the associated scale() method.
Best option I have found so far (not needing toString + index):
public static int digitsStripTrailingZero(BigDecimal value)
{
return digits(value.stripTrailingZeros());
}
public static int digits(BigDecimal value)
{
return Math.max(0, value.scale());
}
Michael Borgwardt answer is the correct one. As soon as you use any double or float, your values are already corrupted.
To provide a code example:
System.out.println("0 = " + BigDecimalUtil.getNumberOfDecimalPlace("0")); // 0
System.out.println("1.0 = " + BigDecimalUtil.getNumberOfDecimalPlace("1.0")); // 0
System.out.println("1.01 = " + BigDecimalUtil.getNumberOfDecimalPlace(new BigDecimal("1.01"))); // 2
System.out.println("1.012 = " + BigDecimalUtil.getNumberOfDecimalPlace(new BigDecimal("1.012"))); // 3
System.out.println("0.01 = " + BigDecimalUtil.getNumberOfDecimalPlace("0.01")); // 2
System.out.println("0.012 = " + BigDecimalUtil.getNumberOfDecimalPlace("0.012")); // 3
System.out.println("0.00000000000000000012 = " + BigDecimalUtil.getNumberOfDecimalPlace("0.00000000000000000012")); // 20
And an overloaded version of getNumberOfDecimalPlace so you could use it with BigDecimal or String:
public static int getNumberOfDecimalPlace(String value) {
final int index = value.indexOf('.');
if (index < 0) {
return 0;
}
return value.length() - 1 - index;
}
public static int getNumberOfDecimalPlace(BigDecimal value) {
return getNumberOfDecimalPlace(value.toPlainString());
}
Why not just change your code to get a doubles decimal places?
public static int getNumberOfDecimalPlace(double value) {
//For whole numbers like 0
if (Math.round(value) == value) return 0;
final String s = Double.toString(value);
System.out.println(s);
final int index = s.indexOf('.');
if (index < 0) {
return 0;
}
return s.length() - 1 - index;
}

Categories