I have a need to sum all of the money from the underlying DB.
I have 2 options to do that.
Use the SQL SUM() function to sum it up or
Get the list of all the money and sum in memory.
I think it would be better to use the first option considering better performance but I want to know if SUM function can be safely used to sum the monetary amount without loosing the precision.
If it were the 2nd option I would have used Java's BigDecimal which is used to represent money.
It depends on the datatype you are using.
if you're using floats, there is some precision lost in mysql (see here and here), if you are using decimals, then you should be ok.
If you use DECIMAL(m, 2) (assuming 2 decimal places for your currency), MySQL's SUM() will be exact and faster than fetching all the rows, shoveling them into the client, then doing the equivalent arithmetic with a similar library.
AVG has precision issues in any situation.
https://msdn.microsoft.com/en-us/library/ms187810.aspx
Returns the summation of all expression values in the most precise expression data type.
SUM will not lose precision any more than your local processor would, you are safe to use it.
Edit:
I mistakenly posted some T-SQL docs, here are the mySQL docs:
http://dev.mysql.com/doc/refman/5.7/en/group-by-functions.html
The SUM() and AVG() functions return a DECIMAL value for exact-value arguments (integer or DECIMAL), and a DOUBLE value for approximate-value arguments (FLOAT or DOUBLE).
So identically to T-SQL, it will choose the most precise data type.
Related
I have a question about big numbers in groovy, I write at script that call to DB and put budget value in variable called toub_start_budget, the value stored is 2570000000.
since I want to do arithmetic operation I created another variable called toub_budget and put in it the value of the first variable as float.
The problem is that the new variable not saved the data as float but as a number as 2.56999987E9.
and the arithmetic that I do are wrong, for example divide by 1000000, will bring 2569.9 and not the accurate results 2570 (accuracy is important).
can someone please advise how to handle big numbers, with arithmetic?
regards
Do not use float, it is by definition inaccurate. If you need to do accurate calculations and arbitrary big numbers, use BigDecimal. If you use Groovy, then just do the calculation, Groovy will automatically use BigDecimal when appropriate as you can see by executing (2570000000 / 1000).getClass() and (2570000000 / 1001).getClass()
Oracle (and some other DB's) have a datatype NUMBER, with which one can optionally set the precision and scale.
Suppose the below query:
SELECT agent_code,
AVG (opening_amt)
FROM customer
GROUP BY agent_code;
If both fields in above query were defined as NUMBER(12,0), the result in JDBC is indeed that for agent_code, but on "AVG(opening_amt)" both precision and scale return 0 (via java.sql.ResultSetMetaData.getPrecision(col) and java.sql.ResultSetMetaData.getScale(col) .
That's basically the same as NUMBER, without any precision or scale specification, and according to oracle, would equal NUMBER(38,12).
The above loss of precision gives me a problem to determine if the sql type should be converted to Double or Integer.
So, I was wondering if this is actually a bug in Oracle's JDBC driver, or how this should be handled? (and no, using BigDecimal as corresponsting java type is not an option for me).
This is speculation based on similar behaviour in the Postgres driver postgresql-9.4-1204-jdbc42.jar.
For an unspecified NUMERIC the database doesn't seem to store any particular information on the precision and scale of the column. This allows the database to internally store the value in any way it seems fit. From https://www.postgresql.org/docs/current/static/datatype-numeric.html
without any precision or scale creates a column in which numeric values of any precision and scale can be stored, up to the implementation limit on precision (up to 131072 digits before the decimal point; up to 16383 digits after the decimal point)
Since the driver doesn't know what the implementation specific maximum of the server is, it can't return the actual values. It returns 0 to indicate that it doesn't know the actual values, and doesn't want to make any educated guesses.
Seems like the situation is the same with Oracle. The max precision may be higher, but portability is guaranteed only up to 38 digits.
Numbers of virtually any magnitude can be stored and are guaranteed portable among different systems operating Oracle Database, up to 38 digits of precision.
As for solving the issue in the question, like StanislavL indicated you can force the value to a specific precision/scale by casting.
I think you can cast to any desired type
CAST(AVG(opening_amt) AS DECIMAL(12,2))
See the example
The SQL AVG() function returns the average value with default decimal places. The CAST() is used to increase or decrease the decimal places of a value. The CAST() function is much better at preserving the decimal places when converting decimal and numeric data types. The 'AS DECIMAL' followed by the format specification is used with CAST() for making a numeric value to a specific decimal place value.
Is there a simple way that i can manipulate high precision decimal numbers in java, without a limit on the number of decimal places, and the ability to output the number in a println or write it to a file? I want to work with one of the identities of PI that involves a sum of fractions, where k starts at 0 and goes to infinity. I know that most systems use limited decimals, but couldn't i use some pre-designed class which stores the value as a linked list of massive memory blocks if the number gets long enough? Please keep in mind i do need to do arithmetic with this class as well. Addition, subtraction, multiplication, and division should be sufficient.
I believe that you are looking for the java.lang.BigDecimal class.
Look at java.lang.BigDecimal, may solve your problem.
http://docs.oracle.com/javase/7/docs/api/java/math/BigDecimal.html
I know the problem with double/float, and it's recommended to use BigDecimal instead of double/float to represent monetary fields. But double/float is more effective and space-saving. Then my question is:
It's acceptable to use double/float to represent monetary fields in Java class, but use BigDecimal to take care of the arithmetic (i.e. convert double/float to BigDecimal before any arithmetic) and equal-checking?
The reason is to save some space. And I really see lots of projects are using double/float to represent the monetary fields.
Is there any pitfall for this?
Thanks in advance.
No, you can't.
Suppose double is enough to store two values x and y. Then you convert them to safe BigDecimal and multiple them. The result is accurate, however if you store the multiplication result back in double, chances are you will loose the precision. Proof:
double x = 1234567891234.0;
double y = 1234567891234.0;
System.out.println(x);
System.out.println(y);
BigDecimal bigZ = new BigDecimal(x).multiply(new BigDecimal(y));
double z = bigZ.doubleValue();
System.out.println(bigZ);
System.out.println(z);
Results:
1.234567891234E12 //precise 'x'
1.234567891234E12 //precise 'y'
1524157878065965654042756 //precise 'x * y'
1.5241578780659657E24 //loosing precision
x and y are accurate, as well as the multiplication using BigDecimal. However after casting back to double we loose least significant digits.
I would also recommend that you use nothing but BigDecimal for ALL arithmetic that may involve currency.
Make sure that you always use the String constructor of BigDecimal. Why? Try the following code in a JUnit test:
assertEquals(new BigDecimal("0.01").toString(), new BigDecimal(0.01).toString());
You get the following output:
expected:<0.01[]> but was <0.01[000000000000000020816681711721685132943093776702880859375]>
The truth is, you cannot store EXACTLY 0.01 as a 'double' amount. Only BigDecimal stores the number you require EXACTLY as you want it.
And remember that BigDecimal is immutable. The following will compile:
BigDecimal amount = new BigDecimal("123.45");
BigDecimal more = new BigDecimal("12.34");
amount.add(more);
System.out.println("Amount is now: " + amount);
but the resulting output will be:
Amount is now: 123.45
That's because you need to assign the result to a new (or the same) BigDecimal variable.
In other words:
amount = amount.add(more)
What is acceptable depends on your project. You can use double and long in some projects may be expected to do so. However in other projects, this is considered unacceptable. As a double you can represent values up to 70,000,000,000,000.00 to the cent (larger than the US national debt), with fixed place long you can represent 90,000,000,000,000,000.00 accurately.
If you have to deal with hyper-inflationary currencies (a bad idea in any case) but for some reason still need to account for every cent, use BigDecimal.
If you use double or long or BigDecimal, you must round the result. How you do this varies with each data type and BigDecimal is the least error prone as you are requires to specify what rounding and the precision for different operations. With double or long, you are left to your own devices.
long will be much better choice than double/float.
Are you sure that using BigDecimal type will be a real bottleneck?
Pit fall is that floats/doubles can not store all values without losing precision. Even if you do your use BigDecimal and preserve precision during calculations, you are still storing the end product as a float/double.
The "proper" solution to this, in my experience, is to store monetary values as integers (e.g. Long) representing thousands of a dollar. This gives sufficient resolution for most tasks, e.g. interest accruement, while side stepping the problem of using floats/doubles. As an added "bonus", this requires about the same amount of storage as floats/doubles.
If the only use of double is to store decimal values, then yes, you can under some conditions: if you can guarantee that your values have no more than 15 decimal digits, then converting a value to double (53 bits of precision) and converting the double back to decimal with 15-digit precision (or less) will give you the original value, i.e. without any loss, from an application of David Matula's theorem proved in his article In-and-out conversions. Note that for this result to be applicable, the conversions must be done with correct rounding.
Note however that a double may not be the best choice: monetary values are generally expressed not in floating point, but in fixed point with a few digits (p) after the decimal point, and in this case, converting the value to an integer with a scaling by 10^p and storing this integer (as others suggested) is better.
Should we use double or BigDecimal for calculations in Java?
How much is the overhead in terms of performance for BigDecimal as compared to double?
For a serious financial application BigDecimal is a must.
Depends on how many digits you need you can go with a long and a decimal factor for visualization.
For general floating point calculations, you should use double. If you are absolutely sure that you really do need arbitrary precision arithmetic (most applications don't), then you can consider BigDecimal.
You will find that double will significantly outperform BigDecimal (not to mention being easier to work with) for any application where double is sufficient precision.
Update: You commented on another answer that you want to use this for a finance related application. This is one of the areas where you actually should consider using BigDecimal, otherwise you may get unexpected rounding effects from double calculations. Also, double values have limited precision, and you won't be able to accurately keep track of pennies at the same time as millions of dollars.
How much is the overhead in terms of performance for BigDecimal as compared to double?
A lot. For example, a multiplication of two doubles is a single machine instruction. Multiplying two BigDecimals is probably a minimum of 50 machine instructions, and has complexity of O(N * M) where M and N are the number of bytes used to represent the two numbers.
However, if your application requires the calculation to be "decimally correct", then you need to accept the overhead.
However (#2) ... even BigDecimal can't do this calculation with real number accuracy:
1/3 + 1/3 + 1/3 -> ?
To do that computation precisely you would need to implement a Rational type; i.e. a pair of BigInteger values ... and some thing to reduce the common factors.
However (#3) ... even a hypothetical Rational type won't give you a precise numeric representation for (say) Pi.
As always: it depends.
If you need the precision (even for "small" numbers, when representing amounts for example) go with BigDecimal.
In some scientific applications, double may be a better choice.
Even in finance we can't answer without knowing what area. For instance if you were doing currency conversions of $billions, where the conversion rate could be to 5 d.p. you might have problems with double. Whereas for simply adding and subtracting balances you'd be fine.
If you don't need to work in fractions of a cent/penny, maybe an integral type might be more appropriate, again it depends on the size of numbers involved.