Related
I am trying to create a logistic regression algorithm in java but when I calculate the logarithm of the likelihood it is always returning NaN. My method which calculates the logarithm looks like this :
//Calculate log likelihood on given data
private double getLogLikelihood(double cat, double[] x) {
return cat * Math.log(findProbability(x))
+ (1 - cat) * Math.log(1 - findProbability(x));
}
And the findProbability method is just take an instance from the dataset and returning the sigmoid funcion result which is between 0 and 1.
//Calculate the sum of w * x for each weight and attribute
//call the sigmoid function with that s
public double findProbability(double[] x){
double s = 0;
for(int i = 0; i < this.weights.length; i++){
if(i >= x.length) break;
s += this.weights[i] * x[i];
}
return sigmoid(s);
}
private double sigmoid(double s){
return 1 / (1 + Math.exp(-s));
}
Moreover, my starting weights are :
[-0.2982955509135178, -0.4984900460081106, -1.816880187922516, -2.7325608512266073, 0.12542715714800834, 0.1516078084483485, 0.27631147403449774, 0.1371611094778011, 0.16029832096058613, 0.3117065974657231, 0.04262385176091778, 0.1948263133838624, 0.10788353525185314, 0.770608588466501, 0.2697281907888033, 0.09920694325563077, 0.003224073601703939, 0.021573742410541247, 0.21528348692817675, 0.3275511757298476, -0.1500597314893408, -0.7221692528386277, -2.062544912370121, 1.4315146889363015, 0.2522133355419722, 0.23919315019065995, 0.3200037377021523, 0.059466770771758076, 0.04012493980772944, 0.2553236501265919]
Finally, an instance from my dataset is :[M,17.99,10.38,122.8,1001,0.1184,0.2776,0.3001,0.1471,0.2419,0.07871,1.095,0.9053,8.589,153.4,0.006399,0.04904,0.05373,0.01587,0.03003,0.006193,25.38,17.33,184.6,2019,0.1622,0.6656,0.7119,0.2654,0.4601,0.1189]
I tried to initialize the starting weightss with different random numbers but thats didnt solve the problem.
The arithematic is causing a rounding error leaving you with 1.
double b = 1 + Math.exp(-3522);
b will be equal to 1, because otherwise you will need too many sig figs. You'll have to approximate the value to keep the precision. 1/(1+s) ~= 1 - s; Which means you need to calculate log(1) and log(s).
edit: sorry, I made a mistake, it appears Math.exp(-3522) is evaluated as 0 after rounding. Ill leave this answer because Math.exp(-x) might be too small to add to 1, or it might just be zero.
NaN is a result of dividing by zero or calling Math.log on a non-positive number, so u should try and find where exactly this happens. I suggest debugging or adding code to print the values of which u take the logarithm/dividy by.
EDIT: it seems it is a rounding error: exp(-s) will return a result so small that added with 1 it will still remain 1. This causes the logarithm to return -Inf. I'd suggest u try and find a mathematical way to solve this by trying to perhaps to approximate the log of the exponential.
I found a solution to my problem so I post it here:
I added an overflow check:
private double sigmoid(double s){
if(s>20){
s=20;
}else if(s<-20){
s=-20;
}
double exp = Math.exp(s);
return exp/(1+exp);
}
Also changing 1/(1+Math.exp(s) to exp/(1+exp) proved to be more stable in small disturbances of inputs.
I have to replicate the luhn algorithm in Java, the problem I face is how to implement this in an efficient and elegant way (not a requirement but that is what I want).
The luhn-algorithm works like this:
You take a number, let's say 56789
loop over the next steps till there are no digits left
You pick the left-most digit and add it to the total sum. sum = 5
You discard this digit and go the next. number = 6789
You double this digit, if it's more than one digit you take apart this number and add them separately to the sum. 2*6 = 12, so sum = 5 + 1 = 6 and then sum = 6 + 2 = 8.
Addition restrictions
For this particular problem I was required to read all digits one at a time and do computations on each of them separately before moving on. I also assume that all numbers are positive.
The problems I face and the questions I have
As said before I try to solve this in an elegant and efficient way. That's why I don't want to invoke the toString() method on the number to access all individual digits which require a lot of converting. I also can't use the modulo kind of way because of the restriction above that states once I read a number I should also do computations on it right away. I could only use modulo if I knew in advance the length of the String, but that feels like I first have to count all digits one-for-once which thus is against the restriction. Now I can only think of one way to do this, but this would also require a lot of computations and only ever cares about the first digit*:
int firstDigit(int x) {
while (x > 9) {
x /= 10;
}
return x;
}
Found here: https://stackoverflow.com/a/2968068/3972558
*However, when I think about it, this is basically a different and weird way to make use of the length property of a number by dividing it as often till there is one digit left.
So basically I am stuck now and I think I must use the length property of a number which it does not really have, so I should find it by hand. Is there a good way to do this? Now I am thinking that I should use modulo in combination with the length of a number.
So that I know if the total number of digits is uneven or even and then I can do computations from right to left. Just for fun I think I could use this for efficiency to get the length of a number: https://stackoverflow.com/a/1308407/3972558
This question appeared in the book Think like a programmer.
You can optimise it by unrolling the loop once (or as many times are you like) This will be close to twice as fast for large numbers, however make small numbers slower. If you have an idea of the typical range of numbers you will have you can determine how much to unroll this loop.
int firstDigit(int x) {
while (x > 99)
x /= 100;
if (x > 9)
x /= 10;
return x;
}
use org.apache.commons.validator.routines.checkdigit.LuhnCheckDigit . isValid()
Maven Dependency:
<dependency>
<groupId>commons-validator</groupId>
<artifactId>commons-validator</artifactId>
<version>1.4.0</version>
</dependency>
Normally you would process the numbers from right to left using divide by 10 to shift the digits and modulo 10 to extract the last one. You can still use this technique when processing the numbers from left to right. Just use divide by 1000000000 to extract the first number and multiply by 10 to shift it left:
0000056789
0000567890
0005678900
0056789000
0567890000
5678900000
6789000000
7890000000
8900000000
9000000000
Some of those numbers exceed maximum value of int. If you have to support full range of input, you will have to store the number as long:
static int checksum(int x) {
long n = x;
int sum = 0;
while (n != 0) {
long d = 1000000000l;
int digit = (int) (n / d);
n %= d;
n *= 10l;
// add digit to sum
}
return sum;
}
As I understand, you will eventually need to read every digit, so what is wrong with convert initial number to string (and therefore char[]) and then you can easily implement the algorithm iterating that char array.
JDK implementation of Integer.toString is rather optimized so that you would need to implement your own optimalizations, e.g. it uses different lookup tables for optimized conversion, convert two chars at once etc.
final static int [] sizeTable = { 9, 99, 999, 9999, 99999, 999999, 9999999,
99999999, 999999999, Integer.MAX_VALUE };
// Requires positive x
static int stringSize(int x) {
for (int i=0; ; i++)
if (x <= sizeTable[i])
return i+1;
}
This was just an example but feel free to check complete implementation :)
I would first convert the number to a kind of BCD (binary coded decimal). I'm not sure to be able to find a better optimisation than the JDK Integer.toString() conversion method but as you said you did not want to use it :
List<Byte> bcd(int i) {
List<Byte> l = new ArrayList<Byte>(10); // max size for an integer to avoid reallocations
if (i == 0) {
l.add((byte) i);
}
else {
while (i != 0) {
l.add((byte) (i % 10));
i = i / 10;
}
}
return l;
}
It is more or less what you proposed to get first digit, but now you have all you digits in one single pass and can use them for your algorythm.
I proposed to use byte because it is enough, but as java always convert to int to do computations, it might be more efficient to directly use a List<Integer> even if it really wastes memory.
Suppose I have a method to calculate combinations of r items from n items:
public static long combi(int n, int r) {
if ( r == n) return 1;
long numr = 1;
for(int i=n; i > (n-r); i--) {
numr *=i;
}
return numr/fact(r);
}
public static long fact(int n) {
long rs = 1;
if(n <2) return 1;
for (int i=2; i<=n; i++) {
rs *=i;
}
return rs;
}
As you can see it involves factorial which can easily overflow the result. For example if I have fact(200) for the foctorial method I get zero. The question is why do I get zero?
Secondly how do I deal with overflow in above context? The method should return largest possible number to fit in long if the result is too big instead of returning wrong answer.
One approach (but this could be wrong) is that if the result exceed some large number for example 1,400,000,000 then return remainder of result modulo
1,400,000,001. Can you explain what this means and how can I do that in Java?
Note that I do not guarantee that above methods are accurate for calculating factorial and combinations. Extra bonus if you can find errors and correct them.
Note that I can only use int or long and if it is unavoidable, can also use double. Other data types are not allowed.
I am not sure who marked this question as homework. This is NOT homework. I wish it was homework and i was back to future, young student at university. But I am old with more than 10 years working as programmer. I just want to practice developing highly optimized solutions in Java. In our times at university, Internet did not even exist. Today's students are lucky that they can even post their homework on site like SO.
Use the multiplicative formula, instead of the factorial formula.
Since its homework, I won't want to just give you a solution. However a hint I will give is that instead of calculating two large numbers and dividing the result, try calculating both together. e.g. calculate the numerator until its about to over flow, then calculate the denominator. In this last step you can chose the divide the numerator instead of multiplying the denominator. This stops both values from getting really large when the ratio of the two is relatively small.
I got this result before an overflow was detected.
combi(61,30) = 232714176627630544 which is 2.52% of Long.MAX_VALUE
The only "bug" I found in your code is not having any overflow detection, since you know its likely to be a problem. ;)
To answer your first question (why did you get zero), the values of fact() as computed by modular arithmetic were such that you hit a result with all 64 bits zero! Change your fact code to this:
public static long fact(int n) {
long rs = 1;
if( n <2) return 1;
for (int i=2; i<=n; i++) {
rs *=i;
System.out.println(rs);
}
return rs;
}
Take a look at the outputs! They are very interesting.
Now onto the second question....
It looks like you want to give exact integer (er, long) answers for values of n and r that fit, and throw an exception if they do not. This is a fair exercise.
To do this properly you should not use factorial at all. The trick is to recognize that C(n,r) can be computed incrementally by adding terms. This can be done using recursion with memoization, or by the multiplicative formula mentioned by Stefan Kendall.
As you accumulate the results into a long variable that you will use for your answer, check the value after each addition to see if it goes negative. When it does, throw an exception. If it stays positive, you can safely return your accumulated result as your answer.
To see why this works consider Pascal's triangle
1
1 1
1 2 1
1 3 3 1
1 4 6 4 1
1 5 10 10 5 1
1 6 15 20 15 6 1
which is generated like so:
C(0,0) = 1 (base case)
C(1,0) = 1 (base case)
C(1,1) = 1 (base case)
C(2,0) = 1 (base case)
C(2,1) = C(1,0) + C(1,1) = 2
C(2,2) = 1 (base case)
C(3,0) = 1 (base case)
C(3,1) = C(2,0) + C(2,1) = 3
C(3,2) = C(2,1) + C(2,2) = 3
...
When computing the value of C(n,r) using memoization, store the results of recursive invocations as you encounter them in a suitable structure such as an array or hashmap. Each value is the sum of two smaller numbers. The numbers start small and are always positive. Whenever you compute a new value (let's call it a subterm) you are adding smaller positive numbers. Recall from your computer organization class that whenever you add two modular positive numbers, there is an overflow if and only if the sum is negative. It only takes one overflow in the whole process for you to know that the C(n,r) you are looking for is too large.
This line of argument could be turned into a nice inductive proof, but that might be for another assignment, and perhaps another StackExchange site.
ADDENDUM
Here is a complete application you can run. (I haven't figured out how to get Java to run on codepad and ideone).
/**
* A demo showing how to do combinations using recursion and memoization, while detecting
* results that cannot fit in 64 bits.
*/
public class CombinationExample {
/**
* Returns the number of combinatios of r things out of n total.
*/
public static long combi(int n, int r) {
long[][] cache = new long[n + 1][n + 1];
if (n < 0 || r > n) {
throw new IllegalArgumentException("Nonsense args");
}
return c(n, r, cache);
}
/**
* Recursive helper for combi.
*/
private static long c(int n, int r, long[][] cache) {
if (r == 0 || r == n) {
return cache[n][r] = 1;
} else if (cache[n][r] != 0) {
return cache[n][r];
} else {
cache[n][r] = c(n-1, r-1, cache) + c(n-1, r, cache);
if (cache[n][r] < 0) {
throw new RuntimeException("Woops too big");
}
return cache[n][r];
}
}
/**
* Prints out a few example invocations.
*/
public static void main(String[] args) {
String[] data = ("0,0,3,1,4,4,5,2,10,0,10,10,10,4,9,7,70,8,295,100," +
"34,88,-2,7,9,-1,90,0,90,1,90,2,90,3,90,8,90,24").split(",");
for (int i = 0; i < data.length; i += 2) {
int n = Integer.valueOf(data[i]);
int r = Integer.valueOf(data[i + 1]);
System.out.printf("C(%d,%d) = ", n, r);
try {
System.out.println(combi(n, r));
} catch (Exception e) {
System.out.println(e.getMessage());
}
}
}
}
Hope it is useful. It's just a quick hack so you might want to clean it up a little.... Also note that a good solution would use proper unit testing, although this code does give nice output.
You can use the java.math.BigInteger class to deal with arbitrarily large numbers.
If you make the return type double, it can handle up to fact(170), but you'll lose some precision because of the nature of double (I don't know why you'd need exact precision for such huge numbers).
For input over 170, the result is infinity
Note that java.lang.Long includes constants for the min and max values for a long.
When you add together two signed 2s-complement positive values of a given size, and the result overflows, the result will be negative. Bit-wise, it will be the same bits you would have gotten with a larger representation, only the high-order bit will be truncated away.
Multiplying is a bit more complicated, unfortunately, since you can overflow by more than one bit.
But you can multiply in parts. Basically you break the to multipliers into low and high halves (or more than that, if you already have an "overflowed" value), perform the four possible multiplications between the four halves, then recombine the results. (It's really just like doing decimal multiplication by hand, but each "digit" is, say, 32 bits.)
You can copy the code from java.math.BigInteger to deal with arbitrarily large numbers. Go ahead and plagiarize.
I have a requirement to calculate the average of a very large set of doubles (10^9 values). The sum of the values exceeds the upper bound of a double, so does anyone know any neat little tricks for calculating an average that doesn't require also calculating the sum?
I am using Java 1.5.
You can calculate the mean iteratively. This algorithm is simple, fast, you have to process each value just once, and the variables never get larger than the largest value in the set, so you won't get an overflow.
double mean(double[] ary) {
double avg = 0;
int t = 1;
for (double x : ary) {
avg += (x - avg) / t;
++t;
}
return avg;
}
Inside the loop avg always is the average value of all values processed so far. In other words, if all the values are finite you should not get an overflow.
The very first issue I'd like to ask you is this:
Do you know the number of values beforehand?
If not, then you have little choice but to sum, and count, and divide, to do the average. If Double isn't high enough precision to handle this, then tough luck, you can't use Double, you need to find a data type that can handle it.
If, on the other hand, you do know the number of values beforehand, you can look at what you're really doing and change how you do it, but keep the overall result.
The average of N values, stored in some collection A, is this:
A[0] A[1] A[2] A[3] A[N-1] A[N]
---- + ---- + ---- + ---- + .... + ------ + ----
N N N N N N
To calculate subsets of this result, you can split up the calculation into equally sized sets, so you can do this, for 3-valued sets (assuming the number of values is divisable by 3, otherwise you need a different divisor)
/ A[0] A[1] A[2] \ / A[3] A[4] A[5] \ // A[N-1] A[N] \
| ---- + ---- + ---- | | ---- + ---- + ---- | \\ + ------ + ---- |
\ 3 3 3 / \ 3 3 3 / // 3 3 /
--------------------- + -------------------- + \\ --------------
N N N
--- --- ---
3 3 3
Note that you need equally sized sets, otherwise numbers in the last set, which will not have enough values compared to all the sets before it, will have a higher impact on the final result.
Consider the numbers 1-7 in sequence, if you pick a set-size of 3, you'll get this result:
/ 1 2 3 \ / 4 5 6 \ / 7 \
| - + - + - | + | - + - + - | + | - |
\ 3 3 3 / \ 3 3 3 / \ 3 /
----------- ----------- ---
y y y
which gives:
2 5 7/3
- + - + ---
y y y
If y is 3 for all the sets, you get this:
2 5 7/3
- + - + ---
3 3 3
which gives:
2*3 5*3 7
--- + --- + ---
9 9 9
which is:
6 15 7
- + -- + -
9 9 9
which totals:
28
-- ~ 3,1111111111111111111111.........1111111.........
9
The average of 1-7, is 4. Obviously this won't work. Note that if you do the above exercise with the numbers 1, 2, 3, 4, 5, 6, 7, 0, 0 (note the two zeroes at the end there), then you'll get the above result.
In other words, if you can't split the number of values up into equally sized sets, the last set will be counted as though it has the same number of values as all the sets preceeding it, but it will be padded with zeroes for all the missing values.
So, you need equally sized sets. Tough luck if your original input set consists of a prime number of values.
What I'm worried about here though is loss of precision. I'm not entirely sure Double will give you good enough precision in such a case, if it initially cannot hold the entire sum of the values.
Apart from using the better approaches already suggested, you can use BigDecimal to make your calculations. (Bear in mind it is immutable)
IMHO, the most robust way of solving your problem is
sort your set
split in groups of elements whose sum wouldn't overflow - since they are sorted, this is fast and easy
do the sum in each group - and divide by the group size
do the sum of the group's sum's (possibly calling this same algorithm recursively) - be aware that if the groups will not be equally sized, you'll have to weight them by their size
One nice thing of this approach is that it scales nicely if you have a really large number of elements to sum - and a large number of processors/machines to use to do the math
Please clarify the potential ranges of the values.
Given that a double has a range ~= +/-10^308, and you're summing 10^9 values, the apparent range suggested in your question is values of the order of 10^299.
That seems somewhat, well, unlikely...
If your values really are that large, then with a normal double you've got only 17 significant decimal digits to play with, so you'll be throwing away about 280 digits worth of information before you can even think about averaging the values.
I would also note (since no-one else has) that for any set of numbers X:
mean(X) = sum(X[i] - c) + c
-------------
N
for any arbitrary constant c.
In this particular problem, setting c = min(X) might dramatically reduce the risk of overflow during the summation.
May I humbly suggest that the problem statement is incomplete...?
A double can be divided by a power of 2 without loss of precision. So if your only problem if the absolute size of the sum you could pre-scale your numbers before summing them. But with a dataset of this size, there is still the risk that you will hit a situation where you are adding small numbers to a large one, and the small numbers will end up being mostly (or completely) ignored.
for instance, when you add 2.2e-20 to 9.0e20 the result is 9.0e20 because once the scales are adjusted so that they numbers can be added together, the smaller number is 0. Doubles can only hold about 17 digits, and you would need more than 40 digits to add these two numbers together without loss.
So, depending on your data set and how many digits of precision you can afford to loose, you may need to do other things. Breaking the data into sets will help, but a better way to preserve precision might be to determine a rough average (you may already know this number). then subtract each value from the rough average before you sum it. That way you are summing the distances from the average, so your sum should never get very large.
Then you take the average delta, and add it to your rough sum to get the correct average. Keeping track of the min and max delta will also tell you how much precision you lost during the summing process. If you have lots of time and need a very accurate result, you can iterate.
You could take the average of averages of equal-sized subsets of numbers that don't exceed the limit.
divide all values by the set size and then sum it up
Option 1 is to use an arbitrary-precision library so you don't have an upper-bound.
Other options (which lose precision) are to sum in groups rather than all at once, or to divide before summing.
So I don't repeat myself so much, let me state that I am assuming that the list of numbers is normally distributed, and that you can sum many numbers before you overflow. The technique still works for non-normal distros, but somethings will not meet the expectations I describe below.
--
Sum up a sub-series, keeping track of how many numbers you eat, until you approach the overflow, then take the average. This will give you an average a0, and count n0. Repeat until you exhaust the list. Now you should have many ai, ni.
Each ai and ni should be relatively close, with the possible exception of the last bite of the list. You can mitigate that by under-biting near the end of the list.
You can combine any subset of these ai, ni by picking any ni in the subset (call it np) and dividing all the ni in the subset by that value. The max size of the subsets to combine is the roughly constant value of the n's.
The ni/np should be close to one. Now sum ni/np * ai and multiple by np/(sum ni), keeping track of sum ni. This gives you a new ni, ai combination, if you need to repeat the procedure.
If you will need to repeat (i.e., the number of ai, ni pairs is much larger than the typical ni), try to keep relative n sizes constant by combining all the averages at one n level first, then combining at the next level, and so on.
First of all, make yourself familiar with the internal representation of double values. Wikipedia should be a good starting point.
Then, consider that doubles are expressed as "value plus exponent" where exponent is a power of two. The limit of the largest double value is an upper limit of the exponent, and not a limit of the value! So you may divide all large input numbers by a large enough power of two. This should be safe for all large enough numbers. You can re-multiply the result with the factor to check whether you lost precision with the multiplication.
Here we go with an algorithm
public static double sum(double[] numbers) {
double eachSum, tempSum;
double factor = Math.pow(2.0,30); // about as large as 10^9
for (double each: numbers) {
double temp = each / factor;
if (t * factor != each) {
eachSum += each;
else {
tempSum += temp;
}
}
return (tempSum / numbers.length) * factor + (eachSum / numbers.length);
}
and dont be worried by the additional division and multiplication. The FPU will optimize the hell out of them since they are done with a power of two (for comparison imagine adding and removing digits at the end of a decimal numbers).
PS: in addition, you may want to use Kahan summation to improve the precision. Kahan summation avoids loss of precision when very large and very small numbers are summed up.
I posted an answer to a question spawned from this one, realizing afterwards that my answer is better suited to this question than to that one. I've reproduced it below. I notice though, that my answer is similar to a combination of Bozho's and Anon.'s.
As the other question was tagged language-agnostic, I chose C# for the code sample I've included. Its relative ease of use and easy-to-follow syntax, along with its inclusion of a couple of features facilitating this routine (a DivRem function in the BCL, and support for iterator functions), as well as my own familiarity with it, made it a good choice for this problem. Since the OP here is interested in a Java solution, but I'm not Java-fluent enough to write it effectively, it might be nice if someone could add a translation of this code to Java.
Some of the mathematical solutions here are very good. Here's a simple technical solution.
Use a larger data type. This breaks down into two possibilities:
Use a high-precision floating point library. One who encounters a need to average a billion numbers probably has the resources to purchase, or the brain power to write, a 128-bit (or longer) floating point library.
I understand the drawbacks here. It would certainly be slower than using intrinsic types. You still might over/underflow if the number of values grows too high. Yada yada.
If your values are integers or can be easily scaled to integers, keep your sum in a list of integers. When you overflow, simply add another integer. This is essentially a simplified implementation of the first option. A simple (untested) example in C# follows
class BigMeanSet{
List<uint> list = new List<uint>();
public double GetAverage(IEnumerable<uint> values){
list.Clear();
list.Add(0);
uint count = 0;
foreach(uint value in values){
Add(0, value);
count++;
}
return DivideBy(count);
}
void Add(int listIndex, uint value){
if((list[listIndex] += value) < value){ // then overflow has ocurred
if(list.Count == listIndex + 1)
list.Add(0);
Add(listIndex + 1, 1);
}
}
double DivideBy(uint count){
const double shift = 4.0 * 1024 * 1024 * 1024;
double rtn = 0;
long remainder = 0;
for(int i = list.Count - 1; i >= 0; i--){
rtn *= shift;
remainder <<= 32;
rtn += Math.DivRem(remainder + list[i], count, out remainder);
}
rtn += remainder / (double)count;
return rtn;
}
}
Like I said, this is untested—I don't have a billion values I really want to average—so I've probably made a mistake or two, especially in the DivideBy function, but it should demonstrate the general idea.
This should provide as much accuracy as a double can represent and should work for any number of 32-bit elements, up to 232 - 1. If more elements are needed, then the count variable will need be expanded and the DivideBy function will increase in complexity, but I'll leave that as an exercise for the reader.
In terms of efficiency, it should be as fast or faster than any other technique here, as it only requires iterating through the list once, only performs one division operation (well, one set of them), and does most of its work with integers. I didn't optimize it, though, and I'm pretty certain it could be made slightly faster still if necessary. Ditching the recursive function call and list indexing would be a good start. Again, an exercise for the reader. The code is intended to be easy to understand.
If anybody more motivated than I am at the moment feels like verifying the correctness of the code, and fixing whatever problems there might be, please be my guest.
I've now tested this code, and made a couple of small corrections (a missing pair of parentheses in the List<uint> constructor call, and an incorrect divisor in the final division of the DivideBy function).
I tested it by first running it through 1000 sets of random length (ranging between 1 and 1000) filled with random integers (ranging between 0 and 232 - 1). These were sets for which I could easily and quickly verify accuracy by also running a canonical mean on them.
I then tested with 100* large series, with random length between 105 and 109. The lower and upper bounds of these series were also chosen at random, constrained so that the series would fit within the range of a 32-bit integer. For any series, the results are easily verifiable as (lowerbound + upperbound) / 2.
*Okay, that's a little white lie. I aborted the large-series test after about 20 or 30 successful runs. A series of length 109 takes just under a minute and a half to run on my machine, so half an hour or so of testing this routine was enough for my tastes.
For those interested, my test code is below:
static IEnumerable<uint> GetSeries(uint lowerbound, uint upperbound){
for(uint i = lowerbound; i <= upperbound; i++)
yield return i;
}
static void Test(){
Console.BufferHeight = 1200;
Random rnd = new Random();
for(int i = 0; i < 1000; i++){
uint[] numbers = new uint[rnd.Next(1, 1000)];
for(int j = 0; j < numbers.Length; j++)
numbers[j] = (uint)rnd.Next();
double sum = 0;
foreach(uint n in numbers)
sum += n;
double avg = sum / numbers.Length;
double ans = new BigMeanSet().GetAverage(numbers);
Console.WriteLine("{0}: {1} - {2} = {3}", numbers.Length, avg, ans, avg - ans);
if(avg != ans)
Debugger.Break();
}
for(int i = 0; i < 100; i++){
uint length = (uint)rnd.Next(100000, 1000000001);
uint lowerbound = (uint)rnd.Next(int.MaxValue - (int)length);
uint upperbound = lowerbound + length;
double avg = ((double)lowerbound + upperbound) / 2;
double ans = new BigMeanSet().GetAverage(GetSeries(lowerbound, upperbound));
Console.WriteLine("{0}: {1} - {2} = {3}", length, avg, ans, avg - ans);
if(avg != ans)
Debugger.Break();
}
}
A random sampling of a small set of the full dataset will often result in a 'good enough' solution. You obviously have to make this determination yourself based on system requirements. Sample size can be remarkably small and still obtain reasonably good answers. This can be adaptively computed by calculating the average of an increasing number of randomly chosen samples - the average will converge within some interval.
Sampling not only addresses the double overflow concern, but is much, much faster. Not applicable for all problems, but certainly useful for many problems.
Consider this:
avg(n1) : n1 = a1
avg(n1, n2) : ((1/2)*n1)+((1/2)*n2) = ((1/2)*a1)+((1/2)*n2) = a2
avg(n1, n2, n3) : ((1/3)*n1)+((1/3)*n2)+((1/3)*n3) = ((2/3)*a2)+((1/3)*n3) = a3
So for any set of doubles of arbitrary size, you could do this (this is in C#, but I'm pretty sure it could be easily translated to Java):
static double GetAverage(IEnumerable<double> values) {
int i = 0;
double avg = 0.0;
foreach (double value in values) {
avg = (((double)i / (double)(i + 1)) * avg) + ((1.0 / (double)(i + 1)) * value);
i++;
}
return avg;
}
Actually, this simplifies nicely into (already provided by martinus):
static double GetAverage(IEnumerable<double> values) {
int i = 1;
double avg = 0.0;
foreach (double value in values) {
avg += (value - avg) / (i++);
}
return avg;
}
I wrote a quick test to try this function out against the more conventional method of summing up the values and dividing by the count (GetAverage_old). For my input I wrote this quick function to return as many random positive doubles as desired:
static IEnumerable<double> GetRandomDoubles(long numValues, double maxValue, int seed) {
Random r = new Random(seed);
for (long i = 0L; i < numValues; i++)
yield return r.NextDouble() * maxValue;
yield break;
}
And here are the results of a few test trials:
long N = 100L;
double max = double.MaxValue * 0.01;
IEnumerable<double> doubles = GetRandomDoubles(N, max, 0);
double oldWay = GetAverage_old(doubles); // 1.00535024998431E+306
double newWay = GetAverage(doubles); // 1.00535024998431E+306
doubles = GetRandomDoubles(N, max, 1);
oldWay = GetAverage_old(doubles); // 8.75142021696299E+305
newWay = GetAverage(doubles); // 8.75142021696299E+305
doubles = GetRandomDoubles(N, max, 2);
oldWay = GetAverage_old(doubles); // 8.70772312848651E+305
newWay = GetAverage(doubles); // 8.70772312848651E+305
OK, but what about for 10^9 values?
long N = 1000000000;
double max = 100.0; // we start small, to verify accuracy
IEnumerable<double> doubles = GetRandomDoubles(N, max, 0);
double oldWay = GetAverage_old(doubles); // 49.9994879713857
double newWay = GetAverage(doubles); // 49.9994879713868 -- pretty close
max = double.MaxValue * 0.001; // now let's try something enormous
doubles = GetRandomDoubles(N, max, 0);
oldWay = GetAverage_old(doubles); // Infinity
newWay = GetAverage(doubles); // 8.98837362725198E+305 -- no overflow
Naturally, how acceptable this solution is will depend on your accuracy requirements. But it's worth considering.
Check out the section for cummulative moving average
In order to keep logic simple, and keep performance not the best but acceptable, i recommend you to use BigDecimal together with the primitive type.
The concept is very simple, you use primitive type to sum values together, whenever the value will underflow or overflow, you move the calculate value to the BigDecimal, then reset it for the next sum calculation. One more thing you should aware is when you construct BigDecimal, you ought to always use String instead of double.
BigDecimal average(double[] values){
BigDecimal totalSum = BigDecimal.ZERO;
double tempSum = 0.00;
for (double value : values){
if (isOutOfRange(tempSum, value)) {
totalSum = sum(totalSum, tempSum);
tempSum = 0.00;
}
tempSum += value;
}
totalSum = sum(totalSum, tempSum);
BigDecimal count = new BigDecimal(values.length);
return totalSum.divide(count);
}
BigDecimal sum(BigDecimal val1, double val2){
BigDecimal val = new BigDecimal(String.valueOf(val2));
return val1.add(val);
}
boolean isOutOfRange(double sum, double value){
// because sum + value > max will be error if both sum and value are positive
// so I adapt the equation to be value > max - sum
if(sum >= 0.00 && value > Double.MAX - sum){
return true;
}
// because sum + value < min will be error if both sum and value are negative
// so I adapt the equation to be value < min - sum
if(sum < 0.00 && value < Double.MIN - sum){
return true;
}
return false;
}
From this concept, every time the result is underflow or overflow, we will keep that value into the bigger variable, this solution might a bit slowdown the performance due to the BigDecimal calculation, but it guarantee the runtime stability.
Why so many complicated long answers. Here is the simplest way to find the running average till now without any need to know how many elements or size etc..
long int i = 0;
double average = 0;
while(there are still elements)
{
average = average * (i / i+1) + X[i] / (i+1);
i++;
}
return average;
I have a Java method in which I'm summing a set of numbers. However, I want any negatives numbers to be treated as positives. So (1)+(2)+(1)+(-1) should equal 5.
I'm sure there is very easy way of doing this - I just don't know how.
Just call Math.abs. For example:
int x = Math.abs(-5);
Which will set x to 5.
Note that if you pass Integer.MIN_VALUE, the same value (still negative) will be returned, as the range of int does not allow the positive equivalent to be represented.
The concept you are describing is called "absolute value", and Java has a function called Math.abs to do it for you. Or you could avoid the function call and do it yourself:
number = (number < 0 ? -number : number);
or
if (number < 0)
number = -number;
You're looking for absolute value, mate. Math.abs(-5) returns 5...
Use the abs function:
int sum=0;
for(Integer i : container)
sum+=Math.abs(i);
Try this (the negative in front of the x is valid since it is a unary operator, find more here):
int answer = -x;
With this, you can turn a positive to a negative and a negative to a positive.
However, if you want to only make a negative number positive then try this:
int answer = Math.abs(x);
A little cool math trick! Squaring the number will guarantee a positive value of x^2, and then, taking the square root will get you to the absolute value of x:
int answer = Math.sqrt(Math.pow(x, 2));
Hope it helps! Good Luck!
This code is not safe to be called on positive numbers.
int x = -20
int y = x + (2*(-1*x));
// Therefore y = -20 + (40) = 20
Are you asking about absolute values?
Math.abs(...) is the function you probably want.
You want to wrap each number into Math.abs(). e.g.
System.out.println(Math.abs(-1));
prints out "1".
If you want to avoid writing the Math.-part, you can include the Math util statically. Just write
import static java.lang.Math.abs;
along with your imports, and you can refer to the abs()-function just by writing
System.out.println(abs(-1));
The easiest, if verbose way to do this is to wrap each number in a Math.abs() call, so you would add:
Math.abs(1) + Math.abs(2) + Math.abs(1) + Math.abs(-1)
with logic changes to reflect how your code is structured. Verbose, perhaps, but it does what you want.
When you need to represent a value without the concept of a loss or absence (negative value), that is called "absolute value".
The logic to obtain the absolute value is very simple: "If it's positive, maintain it. If it's negative, negate it".
What this means is that your logic and code should work like the following:
//If value is negative...
if ( value < 0 ) {
//...negate it (make it a negative negative-value, thus a positive value).
value = negate(value);
}
There are 2 ways you can negate a value:
By, well, negating it's value: value = (-value);
By multiplying it by "100% negative", or "-1": value = value *
(-1);
Both are actually two sides of the same coin. It's just that you usually don't remember that value = (-value); is actually value = 1 * (-value);.
Well, as for how you actually do it in Java, it's very simple, because Java already provides a function for that, in the Math class: value = Math.abs(value);
Yes, doing it without Math.abs() is just a line of code with very simple math, but why make your code look ugly? Just use Java's provided Math.abs() function! They provide it for a reason!
If you absolutely need to skip the function, you can use value = (value < 0) ? (-value) : value;, which is simply a more compact version of the code I mentioned in the logic (3rd) section, using the Ternary operator (? :).
Additionally, there might be situations where you want to always represent loss or absence within a function that might receive both positive and negative values.
Instead of doing some complicated check, you can simply get the absolute value, and negate it: negativeValue = (-Math.abs(value));
With that in mind, and considering a case with a sum of multiple numbers such as yours, it would be a nice idea to implement a function:
int getSumOfAllAbsolutes(int[] values){
int total = 0;
for(int i=0; i<values.lenght; i++){
total += Math.abs(values[i]);
}
return total;
}
Depending on the probability you might need related code again, it might also be a good idea to add them to your own "utils" library, splitting such functions into their core components first, and maintaining the final function simply as a nest of calls to the core components' now-split functions:
int[] makeAllAbsolute(int[] values){
//#TIP: You can also make a reference-based version of this function, so that allocating 'absolutes[]' is not needed, thus optimizing.
int[] absolutes = values.clone();
for(int i=0; i<values.lenght; i++){
absolutes[i] = Math.abs(values[i]);
}
return absolutes;
}
int getSumOfAllValues(int[] values){
int total = 0;
for(int i=0; i<values.lenght; i++){
total += values[i];
}
return total;
}
int getSumOfAllAbsolutes(int[] values){
return getSumOfAllValues(makeAllAbsolute(values));
}
Why don't you multiply that number with -1?
Like This:
//Given x as the number, if x is less than 0, return 0 - x, otherwise return x:
return (x <= 0.0F) ? 0.0F - x : x;
If you're interested in the mechanics of two's complement, here's the absolutely inefficient, but illustrative low-level way this is made:
private static int makeAbsolute(int number){
if(number >=0){
return number;
} else{
return (~number)+1;
}
}
Library function Math.abs() can be used.
Math.abs() returns the absolute value of the argument
if the argument is negative, it returns the negation of the argument.
if the argument is positive, it returns the number as it is.
e.g:
int x=-5;
System.out.println(Math.abs(x));
Output: 5
int y=6;
System.out.println(Math.abs(y));
Output: 6
String s = "-1139627840";
BigInteger bg1 = new BigInteger(s);
System.out.println(bg1.abs());
Alternatively:
int i = -123;
System.out.println(Math.abs(i));
To convert negative number to positive number (this is called absolute value), uses Math.abs(). This Math.abs() method is work like this
“number = (number < 0 ? -number : number);".
In below example, Math.abs(-1) will convert the negative number 1 to positive 1.
example
public static void main(String[] args) {
int total = 1 + 1 + 1 + 1 + (-1);
//output 3
System.out.println("Total : " + total);
int total2 = 1 + 1 + 1 + 1 + Math.abs(-1);
//output 5
System.out.println("Total 2 (absolute value) : " + total2);
}
Output
Total : 3
Total 2 (absolute value) : 5
I would recommend the following solutions:
without lib fun:
value = (value*value)/value
(The above does not actually work.)
with lib fun:
value = Math.abs(value);
I needed the absolute value of a long , and looked deeply into Math.abs and found that if my argument is less than LONG.MIN_VAL which is -9223372036854775808l, then the abs function would not return an absolute value but only the minimum value. Inthis case if your code is using this abs value further then there might be an issue.
Can you please try this one?
public static int toPositive(int number) {
return number & 0x7fffffff;
}
if(arr[i]<0)
Math.abs(arr[i]); //1st way (taking absolute value)
arr[i]=-(arr[i]); //2nd way (taking -ve of -ve no. yields a +ve no.)
arr[i]= ~(arr[i]-1); //3rd way (taking negation)
I see people are saying that Math.abs(number) but this method is not full proof.
This fails when you try to wrap Math.abs(Integer.MIN_VALUE) (see ref. https://youtu.be/IWrpDP-ad7g)
If you are not sure whether you are going to receive the Integer.MIN_VALUE in the input. It is always recommended to check for that number and handle it manually.
In kotlin you can use unaryPlus
input = input.unaryPlus()
https://kotlinlang.org/api/latest/jvm/stdlib/kotlin/-int/unary-plus.html
Try this in the for loop:
sum += Math.abs(arr[i])
dont do this
number = (number < 0 ? -number : number);
or
if (number < 0) number = -number;
this will be an bug when you run find bug on your code it will report it as RV_NEGATING_RESULT_OF