I need to find a median value of an array of doubles (in Java) without modifying it (so selection is out) or allocating a lot of new memory. I also don't care to find the exact median, but within 10% is fine (so if median splits the sorted array 40%-60% it's fine).
How can I achieve this efficiently?
Taking into account suggestions from rfreak, ILMTitan and Peter I wrote this code:
public static double median(double[] array) {
final int smallArraySize = 5000;
final int bigArraySize = 100000;
if (array.length < smallArraySize + 2) { // small size, so can just sort
double[] arr = array.clone();
Arrays.sort(arr);
return arr[arr.length / 2];
} else if (array.length > bigArraySize) { // large size, don't want to make passes
double[] arr = new double[smallArraySize + 1];
int factor = array.length / arr.length;
for (int i = 0; i < arr.length; i++)
arr[i] = array[i * factor];
return median(arr);
} else { // average size, can sacrifice time for accuracy
final int buckets = 1000;
final double desiredPrecision = .005; // in percent
final int maxNumberOfPasses = 10;
int[] histogram = new int[buckets + 1];
int acceptableMin, acceptableMax;
double min, max, range, scale,
medianMin = -Double.MAX_VALUE, medianMax = Double.MAX_VALUE;
int sum, numbers, bin, neighborhood = (int) (array.length * 2 * desiredPrecision);
for (int r = 0; r < maxNumberOfPasses; r ++) { // enter search for number around median
max = -Double.MAX_VALUE; min = Double.MAX_VALUE;
numbers = 0;
for (int i = 0; i < array.length; i ++)
if (array[i] > medianMin && array[i] < medianMax) {
if (array[i] > max) max = array[i];
if (array[i] < min) min = array[i];
numbers ++;
}
if (min == max) return min;
if (numbers <= neighborhood) return (medianMin + medianMax) / 2;
acceptableMin = (int) (numbers * (50d - desiredPrecision) / 100);
acceptableMax = (int) (numbers * (50d + desiredPrecision) / 100);
range = max - min;
scale = range / buckets;
for (int i = 0; i < array.length; i ++)
histogram[(int) ((array[i] - min) / scale)] ++;
sum = 0;
for (bin = 0; bin <= buckets; bin ++) {
sum += histogram[bin];
if (sum > acceptableMin && sum < acceptableMax)
return ((.5d + bin) * scale) + min;
if (sum > acceptableMax) break; // one bin has too many values
}
medianMin = ((bin - 1) * scale) + min;
medianMax = (bin * scale) + min;
for (int i = 0; i < histogram.length; i ++)
histogram[i] = 0;
}
return .5d * medianMin + .5d * medianMax;
}
}
Here I take into account the size of the array. If it's small, then just sort and get the true median. If it's very large, sample it and get the median of the samples, and otherwise iteratively bin the values and see if the median can be narrowed down to an acceptable range.
I don't have any problems with this code. If someone sees something wrong with it, please let me know.
Thank you.
Assuming you mean median and not average. Also assuming you are working with fairly large double[], or memory wouldn't be an issue for sorting a copy and performing an exact median. ...
With minimal additional memory overhead you could probably run a O(n) algorithm that would get in the ballpark. I'd try this and see how accurate it is.
Two passes.
First pass find the min and max. Create a set of buckets that represent evenly spaced number ranges between the min and max. Make a second pass and "count" how many numbers fall in each bin. You should then be able to make a reasonable estimate of the median. Using 1000 buckets would only cost 4k if you use int[] to store the buckets. The math should be fast.
The only question is accuracy, and I think you should be able to tune the number of buckets to get in the error range for your data sets.
I'm sure someone with a better math/stats background than I could provide a precise size to get the error range you are looking for.
Pick a small number of array elements at random, and find the median of those.
Following on from the OPs question about; how to extract N values from a much larger array.
The following code shows how long it takes to find the median of a large array and then shows how long it take to find the median of a fixed size selection of values. The fixed size selection has a fixed cost, but is increasingly inaccurate as the the size of the original array grows.
The following prints
Avg time 17345 us. median=0.5009231700563378
Avg time 24 us. median=0.5146687617507585
the code
double[] nums = new double[100 * 1000 + 1];
for (int i = 0; i < nums.length; i++) nums[i] = Math.random();
{
int runs = 200;
double median = 0;
long start = System.nanoTime();
for (int r = 0; r < runs; r++) {
double[] arr = nums.clone();
Arrays.sort(arr);
median = arr[arr.length / 2];
}
long time = System.nanoTime() - start;
System.out.println("Avg time " + time / 1000 / runs + " us. median=" + median);
}
{
int runs = 20000;
double median = 0;
long start = System.nanoTime();
for (int r = 0; r < runs; r++) {
double[] arr = new double[301]; // fixed size to sample.
int factor = nums.length / arr.length; // take every nth value.
for (int i = 0; i < arr.length; i++)
arr[i] = nums[i * factor];
Arrays.sort(arr);
median = arr[arr.length / 2];
}
long time = System.nanoTime() - start;
System.out.println("Avg time " + time / 1000 / runs + " us. median=" + median);
}
To meet your requirement of not creating objects, I would put the fixed size array in a ThreadLocal so there is no ongoing object creation. You adjust the size of the array to suit how fast you want the function to be.
1) How much is a lot of new memory? Does it preclude a sorted copy of the data, or of references to the data?
2) Is your data repetitive (are there many distinct values)? If yes, then your answer to (1) is less likely to cause problems, because you may be able to do something with a lookup map and an array: e.g. Map and an an array of short and a suitably tweaked comparison object.
3) The typical case for the your "close to the mean" approximation is more likely to be O(n.log(n)). Most sort algorithms only degrade to O(n^2) with pathological data. Additionally, the exact median is only going to be (typically) O(n.log(n)), assuming you can afford a sorted copy.
4) Random sampling (a-la dan04) is more likely to be accurate than choosing values near the mean, unless your distribution is well behaved. For example poisson distribution and log normal both have different medians to means.
Related
for(i=0; i<array.length; i++){
sum = 4 * 5;
}
What I'm trying to do is add ((array.length - 1) - i) 0's to the value of sum. For this example assume array length is 3. sum equals 20. So for the first iteration of the loop i want to add ((3 - 1) - 0) 0's to the value of sum, so sum would be 2000. The next iteration would be ((3 - 1) - 1) 0's. so sum would equal 200 and so on. I hope what I am trying to achieve is clear.
So my questions are:
Is it possible to just shift an int to add extra digits? My search thus far suggests it is not.
If not, how can i achieve my desired goal?
Thankyou for reading my question and any help would be greatly apreciated.
You can just multiply it by 10 however many times.
200 * 10 = 2000
etc
So in your case, you'd have to use a for loop until the end of the array and multiply sum every iteration. Be careful though, because the max value of an int is 2^31, so it of surpasses that, it will roll back to 0
You can add n zeroes to the end of a number, sum by multiplying sum by 10 * n.
int sum = 20;
for (int i = 0; i < ary.length; ++i) {
int zeroesToAdd = ary.length - 1 - i
sum *= (zeroesToAdd > 0) ? zeroesToAdd * 10 : 1
}
System.out.println("Sum after loop: " + sum);
for(int i=array.length; i>0; i--){
sum = 20;
for(int j=0; j < (i - 1); j++)
{
sum *= 10;
}
}
Use inner loop to multiply by 10 the number of times i is for that iteration. You would need to reset sum in your outer loop each time.
You will want to check your for-loop condition: i>array.length. Since i starts at 0, this loop will not run unless the array's length is also 0 (an empty array). The correct condition is i < array.length.
This "shift" you want can be achieved by creating a temporary variable inside the loop that is equal to the sum times 10i. In Java's Math library, there is a pow(a,b) function that computes ab. With that in mind, what you want is something like this:
int oldSum = 4 * 5;
for (int i = 0; i < array.length; i++) {
int newSum = oldSum * Math.pow(10,i);
}
Multiply by 10 instead, and use < (not >) like
int sum = 20;
int[] array = { 1, 2, 3 };
for (int i = 0; i < array.length; i++) {
int powSum = sum;
for (int j = array.length - 1; j > i; j--) {
powSum *= 10;
}
System.out.println(powSum);
}
Output is (as requested)
2000
200
20
For array of length = n; you will end up adding (n - 1) + (n - 2) + ... + 2 + 1 + 0 zeros for i = 0, 1, ... n-2, n-1 respectively.
Therefore, number of zeros to append (z) = n * (n-1) / 2
So the answer is sum * (10 ^ z)
[EDIT]
The above can be used to find the answer after N iteration. (I miss read the question)
int n = array.length;
long sum = 20;
long pow = Math.pow(10, n-1); // for i = 0
for (int i = 0; i < n; i++) {
System.out.println(sum*pow);
pow /= 10;
}
Hello i am trying to make a method to generate a random number within a range
where it can take a Bias that will make the number more likely to be higher/lower depending on the bias.
To do this currently i was using this
public int randIntWeightedLow(int max, int min, int rolls){
int rValue = 100;
for (int i = 0; i < rolls ; i++) {
int rand = randInt(min, max);
if (rand < rValue ){
rValue = rand;
}
}
return rValue;
}
This works okay by giving me a number in the range and the more rolls i add the likely the number will be low. However the problem i am running in to is that the there is a big difference between having 3 rolls and 4 rolls.
I am loking to have somthing like
public void randomIntWithBias(int min, int max, float bias){
}
Where giving a negative bias would make the number be low more often and
a positive bias make the number be higher more often but still keeping the number in the random of the min and max.
Currently to generate a random number i am using
public int randInt(final int n1, final int n2) {
if (n1 == n2) {
return n1;
}
final int min = n1 > n2 ? n2 : n1;
final int max = n1 > n2 ? n1 : n2;
return rand.nextInt(max - min + 1) + min;
}
I am new to java and coding in general so any help would be greatly appreciated.
Ok, here is quick sketch how it could be done.
First, I propose to use Apache commons java library, it has sampling for integers
with different probabilities already implemented. We need Enumerated Integer Distribution.
Second, two parameters to make distribution look linear, p0 and delta.
For kth value relative probability would be p0 + k*delta. For delta positive
larger numbers will be more probable, for delta negative smaller numbers will be
more probable, delta=0 equal to uniform sampling.
Code (my Java is rusty, please bear with me)
import org.apache.commons.math3.distribution.EnumeratedIntegerDistribution;
public int randomIntWithBias(int min, int max, double p0, double delta){
if (p0 < 0.0)
throw new Exception("Negative initial probability");
int N = max - min + 1; // total number of items to sample
double[] p = new double[N]; // probabilities
int[] items = new int[N]; // items
double sum = 0.0; // total probabilities summed
for(int k = 0; k != N; ++k) { // fill arrays
p[k] = p0 + k*delta;
sum += p[k];
items[k] = min + k;
}
if (delta < 0.0) { // when delta negative we could get negative probabilities
if (p[N-1] < 0.0) // check only last probability
throw new Exception("Negative probability");
}
for(int k = 0; k != N; ++k) { // Normalize probabilities
p[k] /= sum;
}
EnumeratedIntegerDistribution rng = new EnumeratedIntegerDistribution(items, p);
return rng.sample();
}
That's the gist of the idea, code could be (and should be) optimized and cleaned.
UPDATE
Of course, instead of linear bias function you could put in, say, quadratic one.
General quadratic function has three parameters - pass them on, fill in a similar way array of probabilities, normalize, sample
So below is a script that will generate a list of n amount of random numbers between 1-100. I need to get it to where I can also identify the max and min of the random numbers generated in the command prompt after it runs the script. I keep running into the problem where it will just duplicate the number 2 additional times. Example when n= 2: 12 12 12 43 43 43 22 22 22
I think my problem is that when I use int min = Math.min(b,b); the for loop wants to repeat that part too. But if I put it outside of the script then I no longer have the variable b to use.
int n = Integer.parseInt(args[0]);
for(int i = 0; i < n; i++)
{
int b = (int)(Math.random() * (100 - 1)) + 1;
System.out.println(b);
}
if you insist on using Math library, you should consider the first number as min and max.
int n = Integer.parseInt(args[0]);
int max = 0;
int min = 0;
for(int i = 0; i < n; i++){
int b = (int)(Math.random() * (100 - 1)) + 1;
if(i == 0){
min = b;
max = b;
}else{
min = Math.min(min,b);
max = Math.max(max,b);
}
System.out.println(b);
}
System.out.println(String.format("Min:%d , Max:%d",min,max));
The Random class can generate streams of random numbers (with
lower/upper bound).
IntSteam can be collected into IntSummaryStatistics, which provide min/max/avg/sum/count information on the streamed data.
When streaming, you can output (or process) the items over an identity function.
Combining all these results in a simple and elegant solution:
int n=Integer.parseInt(args[0]);
IntSummaryStatistics statistics = new Random()
.ints(1, 100)
.limit(n)
.map(i -> {
System.out.println(i);
return i;
}).summaryStatistics();
System.out.println("min: " + statistics.getMin());
System.out.println("max: " + statistics.getMax());
first the word script in your question means the programming language ( javascript) ?
or you mean the word script ( probably in java we call it a class or program not a script the word script is mainly used in python )
look easy fast way is to push the elements generated into an array and sort it
Vector x=new Vector();
for(int i=0;i<10;i++)
{
x.add(i,(Math.random() * (100 - 1)) + 1);
//System.out.println(x.elementAt(i));
}
Collections.sort(x);
System.out.println(x.elementAt(0));
System.out.println(x.elementAt(x.size()-1));
Fill array a from a[0] to a[n-1]: generate random numbers until you get one that is not already in the previous indexes.
This is my implementation:
public static int[] first(int n) {
int[] a = new int[n];
int count = 0;
while (count != n) {
boolean isSame = false;
int rand = r.nextInt(n) + 1;
for (int i = 0; i < n; i++) {
if(a[i] == rand) isSame = true;
}
if (isSame == false){
a[count] = rand;
count++;
}
}
return a;
}
I thought it was N^2 but it's apparently N^2logN and I'm not sure when the log function is considered.
The 0 entry is filled immediately. The 1 entry has probability 1 - 1 / n = (n - 1) / n of getting filled by a random number. So we need on average n / (n - 1) random numbers to fill the second position. In general, for the k entry we need on average n / (n - k) random numbers and for each number we need k comparisons to check if it's unique.
So we need
n * 1 / (n - 1) + n * 2 / (n - 2) + ... + n * (n - 1) / 1
comparisons on average. If we consider the right half of the sum, we see that this half is greater than
n * (n / 2) * (1 / (n / 2) + 1 / (n / 2 - 1) + ... + 1 / 1)
The sum of the fractions is known to be Θ(log(n)) because it's an harmonic series. So the whole sum is Ω(n^2*log(n)). In a similar way, we can show the sum to be O(n^2*log(n)). This means on average we need
Θ(n^2*log(n))
operations.
This is similar to the Coupon Collector problem. You pick from n items until you get one you don't already have. On average, you have O(n log n) attempts (see the link, the analysis is not trivial). and in the worst case, you examine n elements on each of those attempts. This leads to an average complexity of O(N^2 log N)
The algorithm you have is not O(n^2 lg n) because the algorithm you have may loop forever and not finish. Imagine on your first pass, you get some value $X$ and on every subsequent pass, trying to get the second value, you continue to get $X$ forever. We're talking worst case here, after all. That would loop forever. So since your worst case is never finishing, you can't really analyze.
In case you're wondering, if you know that n is always both the size of the array and the upper bound of the values, you can simply do this:
int[] vals = new int[n];
for(int i = 0; i < n; i++) {
vals[i] = i;
}
// fischer yates shuffle
for(int i = n-1; i > 0; i--) {
int idx = rand.nextInt(i + 1);
int t = vals[idx];
vals[idx] = vals[i];
vals[i] = t;
}
One loop down, one loop back. O(n). Simple.
If I'm not mistaken, the log N part comes from this part:
for(int i = 0; i < count; i++){
if(a[i] == rand) isSame = true;
}
Notice that I changed n for count because you know that you have only count elements in your array on each loop.
I have a range [min-max]. min and max are of type double. I want to divide this interval into n equal intervals.(n is an integer). How can I achieve this in Java?
For example :
say I have a range [10-50]. and n=4 .
output should be a list of ranges like [10-20] [20-30][30-40] [40-50]
So what you need here is a formula for the limits of the smaller ranges. First lets start off by computing the length of each small range:
// let range be [start, end]
// let the number of smaller ranges be n
double total_length = end - start;
double subrange_length = total_length/n;
After that do a simple cycle for the smaller ranges moving the left end of the current range with the value computed above on each step:
double current_start = start;
for (int i = 0; i < n; ++i) {
System.out.printl("Smaller range: [" + current_start + ", " + (current_start + subrange_length) + "]");
current_start += subrange_length;
}
If have the Range given in the form of an array with two elements (min and max)
double[] range = new double[] {min, max};
int n = 4;
you could try it this way. What you get from divideRange is a two-dimensional array with subranges of the given range, with each of them having the wanted length.
public double[][] divideRange(double[] range, n) {
double[][] ranges = new double[n][2];
double length = (range[1] - range[0])/n;
ranges[0][0] = range[0];
ranges[0][1] = range[0]+length;
for(int i = 1; i < n; i++) {
ranges[i][0] = ranges[i-1][1];
ranges[i][1] = ranges[i-1][1]+length;
}
return ranges;
}
What you can do is use what #Achintya used, double dist = (double)(max-min)/n; Then starting from the min, add dist to it and that is the max of your first interval.
So it'd be something like:
[min, min + dist], [min + dist, min + 2*dist]... until min + n*dist >= max.
int counter = 0;
while(true) {
CreateInterval(min + counter*dist, min + (counter+1)*dist);
if (min+(counter+1)*dist >= max) {
//if we have reached the max, we are done
break;
}
}