Being relatively new to Big-Oh Notation and complexity analysis, I have been asked to determine the Big-Oh (tightest upper bound) time complexity of the following code.
Now as far as I can tell, for every iteration after the very first one (for which it would only run exactly once) the inner loop runs x2 meaning x times.
In itself this would be O(x).
y is never changed throughout the entire execution of the algorithm. However, n is incremented in every iteration, which affects the exit condition of the outer loop.
Because of the constant incrementation of n, the fraction serving as the exit condition of the outer loop becomes smaller and smaller.
Now if there was something like y/=2, and not y/n, every single time, I would immediately go for O(log y) runtime of the outer loop, but because of the changing denominator I'm thinking that we could view this as a factor, which --according to what I know about Big-Oh-- can be ignored, hence O(y) complexity of the outer loop, meaning O(x*y) complexity of the whole method.
Could anybody provide me with some guidance or a few tips regarding this? Any help would be greatly appreciated.
Thanks a lot!
void a (long x, long y){
long n = 0, x2 = n;
do {
do {
x2--;
} while (x2 > 0);
n++;
x2 = x;
} while (n < y / n);
}
EDIT: Thanks everybody for helping me out. Just as a little follow-up question: What would be the big-o complexity if the code were instead written like this:
void a(int x, int y) {
int n = 1;
do {
x++;
n++;
} while(y/n > x);
}
I tried rearranging it a little (e.g. y > n*x) and thinking of n as a constant that could be dropped which led me to believe that this would be O(y), but I feel like there is something I just don't yet understand about how these fractional conditions can be expressed in big O notation.
Could anybody provide me with some guidance or a few tips regarding this?
When in doubt count the operations performed.
For the purposes of complexity analysis1, it is a safe assumption that any primitive operation (arithmetic, test, branch) takes a constant amount of time. For these purposes, you can further assume that all primitive operations take the same time. So just add them all up.
Then work out an algebraic formula for the number of operations performed as a function of your variables; e.g. your x and y.
The other thing is that in order to figure out the complexity you will need to understand how the code works. Which can be a bit tricky when you have a mystery function like this one.
But even this one can be understood with some analytical thinking. For example, you can work out how many times x2 is going to be decremented in the inner loop ... by looking at the places where x2 is assigned to.
1 - This only applies to complexity analysis. If you are trying to estimate performance, these assumptions are not correct. Different operations take different times, and in some cases the time taken for a given operation could vary depending on the context; e.g. whether or not you get a cache hit. This is part of the reason that even roughly accurate a priori estimates of performance are really hard work.
Assuming the function is actually executed as written, the outer loop runs proportionally to sqrt(y) times, so the complexity of the entire function is O(x * sqrt(y)).
(It's worth noting that in real life, since the function has no side-effects, an optimizing compiler would probably replace it with a function that does nothing.)
Related
I have currently learned the code of all sorting algorithms used and understood their functioning. However as a part of these, one should also be capable to find the time and space complexity. I have seen people just looking at the loops and deriving the complexity. Can someone guide me towards the best practice for achieving this. The given example code is for "Shell sort". What should be the strategy used to understand and calculate from code itself. Please help! Something like step count method. Need to understand how we can do asymptotic analysis from code itself. Please help.
int i,n=a.length,diff=n/2,interchange,temp;
while(diff>0) {
interchange=0;
for(i=0;i<n-diff;i++) {
if(a[i]>a[i+diff]) {
temp=a[i];
a[i]=a[i+diff];
a[i+diff]=temp;
interchange=1;
}
}
if(interchange==0) {
diff=diff/2;
}
}
Since the absolute lower bound on worst-case of a comparison-sorting algorithm is O(n log n), evidently one can't do any better. The same complexity holds here.
Worst-case time complexity:
1. Inner loop
Let's first start analyzing the inner loop:
for(i=0;i<n-diff;i++) {
if(a[i]>a[i+diff]) {
temp=a[i];
a[i]=a[i+diff];
a[i+diff]=temp;
interchange=1;
}
}
Since we don't know much (anything) about the structure of a on this level, it is definitely possible that the condition holds, and thus a swap occurs. A conservative analysis thus says that it is possible that interchange can be 0 or 1 at the end of the loop. We know however that if we will execute the loop a second time, with the same diff value.
As you comment yourself, the loop will be executed O(n-diff) times. Since all instructions inside the loop take constant time. The time complexity of the loop itself is O(n-diff) as well.
Now the question is how many times can interchange be 1 before it turns to 0. The maximum bound is that an item that was placed at the absolute right is the minimal element, and thus will keep "swapping" until it reaches the start of the list. So the inner loop itself is repeated at most: O(n/diff) times. As a result the computational effort of the loop is worst-case:
O(n^2/diff-n)=O(n^2/diff-n)
2. Outer loop with different diff
The outer loop relies on the value of diff. Starts with a value of n/2, given interchange equals 1 at the end of the loop, something we cannot prove will not be the case, a new iteration will be performed with diff being set to diff/2. This is repeated until diff < 1. This means diff will take all powers of 2 up till n/2:
1 2 4 8 ... n/2
Now we can make an analysis by summing:
log2 n
------
\
/ O(n^2/2^i-n) = O(n^2)
------
i = 0
where i represents *log2(diff) of a given iteration. If we work this out, we get O(n2) worst case time complexity.
Note (On the lower bound of worst-case comparison sort): One can proof no comparison sort algorithm exists with a worst-case time complexity of O(n log n).
This is because for a list with n items, there are n! possible orderings. For each ordering, there is a different way one needs to reorganize the list.
Since using a comparison can split the set of possible orderings into two equals parts at the best, it will require at least log2(n!) comparisons to find out which ordering we are talking about. The complexity of log2(n) can be calculated using the Stirling approximation:
n
/\
|
| log(x) dx = n log n - n = O(n log n)
\/
1
Best-case time complexity: in the best case, the list is evidently ordered. In that case the inner loop will never perform the if-then part. As a consequence, the interchange will not be set to 1 and therefore after executing the for loop one time. The outer loop will still be repeated O(log n) times, thus the time complexity is O(n log n).
Look at the loops and try to figure out how many times they execute. Start from the innermost ones.
In the given example (not the easiest one to begin with), the for loop (innermost) is excuted for i in range [0,n-diff], i.e. it is executed exactly n-diff times.
What is done inside that loop doesn't really matter as long as it takes "constant time", i.e. there is a finite number of atomic operations.
Now the outer loop is executed as long as diff>0. This behavior is complex because an iteration can decrease diff or not (it is decreased when no inverted pair was found).
Now you can say that diff will be decreased log(n) times (because it is halved until 0), and between every decrease the inner loop is run "a certain number of times".
An exercised eye will also recognize interleaved passes of bubblesort and conclude that this number of times will not exceed the number of elements involved, i.e. n-diff, but that's about all that can be said "at a glance".
Complete analysis of the algorithm is an horrible mess, as the array gets progressively better and better sorted, which will influence the number of inner loops.
I just wondering what is the big o for the code below:
I am thinking O(n). What do you guys think? Thank you for your help!
for( w = Length ; w >= 0 ; w = w / 2 ){
for( i = Length ; i >= 0 ; --i ){
if( randomNumber() == 4 )
return
}
}
Since you are asking for the Big O notation, which is the worst case time complexity, the answer is:
O(n^x) , where x is the denominator used in the outer-for loop.
This pretty much looks like a class assignment, so I do not answer it, but just give you some pointers (homework should not be done by copying the assignment to the web ;) ). Also the assignment is incomplete. I hope your teacher/lecturer did not give it like this.
The missing information is:
Are you looking for worst case runtime or average case runtime? Big-O can be used for both. [originally I included best case runtime, but this is done with big omega, as Jerry pointed out in the comments]
Another missing piece is the datatype of the variables. If they are doubles, it takes much longer until w = w/2 is 0 than with integers.
Worst-case runtime:
The inner loop has i = i-1, so it is executed length times. This gives you O(n) for the inner loop.
This already shows that your estimate is wrong. It has to be the number of executions of the outer loop TIMES the number of executions of the inner loop, so it must be more than linear (unless the outer loop has constant number of executions).
The outer loop has w = w/2, so, in terms of length, how long will this need to be 0? This gives you how often the outer loop is executed. And, by multiplication, the total number of executions.
Than there is this randomNumber(). As I said I am assuming worst-case analysis, the worst case is clearly that it is never 4 and thereby we can ignore this return.
Average-case runtime:
The analysis for the loops does not change. For the randomNumber(), we need to estimate how long it takes until the probability of NOT having 4 is sufficiently small. However, I do not have enough information about randomNumber() to do this.
Best-case runtime [should be big omega, not big o]:
In the best case, randomNumber() returns 4 on the first call. So the best case runtime is constant, O(1).
My question is if there are some smart ways of debugging complicated recursive algorithms.
Assume that we have a complicated one (not a simple case when recursion counter is decreased in each 'nested iteration').
I mean something like recursive traversing of a graph when loops are possible.
I need to check if I am not getting endless loop somewhere. And doing this just using a debugger gives not certain answer (because I am not sure if an algorithm is in endless loop or just process as it should).
It's hard to explain it without concrete example. But what I need is...
'to check if the endless loops don't occur in let's say complicated recursive algorithm'.
You need to form a theory for why you think the algorithm does terminate. Ideally, prove the theory as a mathematical theorem.
You can look for a function of the problem state that does reduce on each recursive call. For example, see the following discussion of Ackermann's function, from Wikipedia
It may not be immediately obvious that the evaluation of A(m, n) always terminates. However, the recursion is bounded because in each recursive application either m decreases, or m remains the same and n decreases. Each time that n reaches zero, m decreases, so m eventually reaches zero as well. (Expressed more technically, in each case the pair (m, n) decreases in the lexicographic order on pairs, which is a well-ordering, just like the ordering of single non-negative integers; this means one cannot go down in the ordering infinitely many times in succession.) However, when m decreases there is no upper bound on how much n can increase — and it will often increase greatly.
That is the type of reasoning you should be thinking of applying to your algorithm.
If you cannot find any way to prove your algorithm terminates, consider looking for a variation whose termination you can prove. It is not always possible to decide whether an arbitrary program terminates or not. The trick is to write algorithms you can prove terminate.
Best is proving finiteness by pre and post conditions, variants and invariants. If you can specify a (virtual) formula which value increases on every call you have a guarantee.
This is the same as proving loops to be finite. Furthermore it might make complex algorithms more tackable.
You need to count the depth of recursive calls ... and then throw an exception if the depth of recursive calls reaches a certain threshold.
For example:
void TheMethod(object[] otherParameters, int recursiveCallDepth)
{
if (recursiveCallDepth > 100) {
throw new Exception("...."); }
TheMethod(otherParameters, ++recursiveCallDepth);
}
if you want to check for endless loops,
write a System.out.println("no its not endless"); at the next line of calling the recursive function.
if the loop would be endless, this statement wont get print, if otherwise you will see the output
One suggestion is the following:
If you have endless loop then in the graph case you will obtain a path with number of vertices greater than the total number of vertices in the graph. Assuming that the number of vertices in the graph is a global variable (which, I think, is the most common case) you can do a conditional breakpoint in the beginning of the recursion if the depth is already above the total number of vertices.
Here is a link how you do conditional breakpoints for java in Eclipse.
I have given run-time functions for two algorithms solving the same problem. Let's say -
For First algorithm : T(n) = an + b (Linear in n)
For second Algorithm: T(n) = xn^2 + yn + z (Quadratic in n)
Every book says linear in time is better than quadratic and of course it is for bigger n (how big?). I feel definition of Big changes based on the constants a, b, x, y and z.
Could you please let me know how to find the threshold for n when we should switch to algo1 from algo2 and vice-versa (is it found only through experiments?). I would be grateful if someone can explain how it is done in professional software development organizations.
I hope I am able to explain my question if not please let me know.
Thanks in advance for your help.
P.S. - The implementation would be in Java and expected to run on various platforms. I find it extremely hard to estimate the constants a, b, x, y and z mathematically. How do we solve this dilemma in professional software development?
I would always use the O(n) one, for smaller n it might be slower, but n is small anyway. The added complexity in your code will make it harder to debug and maintain if it's trying to choose the optimal algorithm for each dataset.
It is impossible to estimate the fixed factors in all cases of practical interest. Even if you could, it would not help unless you could also predict how the size of the input is going to evolve in the future.
The linear algorithm should always be preferred unless other factors come into play as well (e.g. memory consumption). If the practical performance is not acceptable you can then look for alternatives.
Experiment. I also encountered a situation in which we had code to find a particular instance in a list of instances. The original code did a simple loop, which worked well for several years.
Once, one of our customers logged a performance problem. In his case the list contained several thousands of instances and the lookup was really slow.
The solution of my fellow developer was to add hashing to the list, which indeed solved the customer's problem. However, now other customers started to complain because they suddenly had a performance problem. It seemed that in most cases, the list only contained a few (around 10) entries, and the hashing was much slower than just looping over the list.
The final solution was to measure the time of both alternatives (looping vs. hashing) and determining the point at which the looping become slower than hashing. In our case this was about 70. So we changed the algorithm:
If the list contains less than 70 items we loop
If the list contains more then 70 items we hash
The solution will probably be similar in your case.
You are asking a maths question, not a programming one.
NB I am going to assume x is positive...
You need to know when
an+b < xn^2 + yn + z
ie
0 < xn^2 + (y-a)n + (z-b)
You can plug this into the standard equation for solving quadratics http://en.wikipedia.org/wiki/Quadratic_equation#Quadratic_formula
And take the larger 0, and then you know for all values greater than this (as x positive) O(n^2) is greater.
You end up with a horrible equation involving x, y, a, z, and b that I very much doubt is any use to you.
Just profile the code with the expected inputs size, it's even better if you also add in a worst case input. Don't waste your time solving the equation, which might be impossible to derive in the first place.
Generally, you can expect O(n2) to be significantly slower than O(n) from size of n = 10000. Significantly slower means that any human can notice it is slower. Depending on the complexity of the algorithm, you might notice the difference at smaller n.
The point is: judging an algorithm based on time complexity allows us to ignore some algorithms that is clearly too slow for any input at the largest input size. However, depending on the domain of the input data, certain algorithm with higher complexity will practically outperform other algorithm with lower time complexity.
When we write an algorithm for a large scale purpose, we want it to perform good for large 'n'. In your case, depending upon a, b, x, y and z, the second algorithm may perform better though its quadratic. But no matter what the values of a, b, x, y and z are, there would be some lower limit of n (say n0) beyond which first algo (linear one) will always be faster than the second.
If f(n) = O(g(n))
then it means for some value of n >= n0 (constant)
f(n) <= c1*g(n)
So
if g(n) = n,
then f(n) = O(n)
So choose the algo depending upon you usage of n
Ok, these are all pretty simple methods, and there are a few of them, so I didnt want to just create multiple questions when they are all the same thing. BigO is my weakness. I just cant figure out how they come up with these answers. Is there anyway you can give me some insight into your thinking for analyzing running times of some of these methods? How do you break it down? How should I think when I see something like these? (specifically the second one, I dont get how thats O(1))
function f1:
loop 3 times
loop n times
Therefore O(3*n) which is effectively O(n).
function f2:
loop 50 times
O(50) is effectively O(1).
We know it will loop 50 times because it will go until n = n - (n / 50) is 0. For this to be true, it must iterate 50 times (n - (n / 50)*50 = 0).
function f3:
loop n times
loop n times
Therefore O(n^2).
function f4:
recurse n times
You know this because worst case is that n = high - low + 1. Disregard the +1.
That means that n = high - low.
To terminate,
arr[hi] * arr[low] > 10
Assume that this doesn't occur until low is incremented to the highest it can go (high).
This means n = high - 0 and we must recurse up to n times.
function 5:
loops ceil(log_2(n)) times
We know this because of the m/=2.
For example, let n=10. log_2(10) = 3.3, the ceiling of which is 4.
10 / 2 =
5 / 2 =
2.5 / 2 =
1.25 / 2 =
0.75
In total, there are 4 iterations.
You get an n^2 analysis when performing a loop within a loop, such as the third method.
However, the first method doesn't a n^2 timing analysis because the first loop is defined as running three times. This makes the timing for the first one 3n, but we don't care about numbers for Big-O.
The second one, introduces an interesting paradigm, where despite the fact that you have a single loop, the timing analysis is still O(1). This is because if you were to chart the timing it takes to perform this method, it wouldn't behave as O(n) for smaller numbers. For larger numbers it becomes obvious.
For the fourth method, you have an O(n) timing because you're recursive function call is passing lo + 1. This is similar to if you were using a for loop and incrementing with lo++/++lo.
The last one has a O(log n) timing because your dividing your variable by two. Just remember than anything that reminds you of a binary search will have a log n timing.
There is also another trick to timing analysis. Say you had a loop within a loop, and within each of the two loops you were reading lines from a file or popping of elements from a stack. This actually would only be a O(n) method, because a file only has a certain number of lines you can read, and a stack only has a certain number of elements you can pop off.
The general idea of big-O notation is this: it gives a rough answer to the question "If you're given a set of N items, and you have to perform some operation repeatedly on these items, how many times will you need to perform this operation?" I say a rough answer, because it (most of the time) doesn't give a precise answer of "5*N+35", but just "N". It's like a ballpark. You don't really care about the precise answer, you just want to know how bad it will get when N gets large. So answers like O(N), O(N*N), O(logN) and O(N!) are typical, because they each represent sort of a "class" of answers, which you can compare to each other. An algorithm with O(N) will perform way better than an algorithm with O(N*N) when N gets large enough, it doesn't matter how lengthy the operation is itself.
So I break it down thus: First identify what the N will be. In the examples above it's pretty obvious - it's the size of the input array, because that determines how many times we will loop. Sometimes it's not so obvious, and sometimes you have multiple input data, so instead of just N you also get M and other letters (and then the answer is something like O(N*M*M)).
Then, when I have my N figured out, I try to identify the loop which depends on N. Actually, these two things often get identified together, as they are pretty much tied together.
And, lastly of course, I have to figure out how many iterations the program will make depending on N. And to make it easier, I don't really try to count them, just try to recognize the typical answers - O(1), O(N), O(N*N), O(logN), O(N!) or perhaps some other power of N. The O(N!) is actually pretty rare, because it's so inefficient, that implementing it would be pointless.
If you get an answer of something like N*N+N+1, then just discard the smaller ones, because, again, when N gets large, the others don't matter anymore. And ignore if the operation is repeated some fixed number of times. O(5*N) is the same as O(N), because it's the ballpark we're looking for.
Added: As asked in the comments, here are the analysis of the first two methods:
The first one is easy. There are only two loops, the inner one is O(N), and the outer one just repeats that 3 times. So it's still O(N). (Remember - O(3N) = O(N)).
The second one is tricky. I'm not really sure about it. After looking at it for a while I understood why it loops at most only 50 times. Since this is not dependant on N at all, it counts as O(1). However, if you were to pass it, say, an array of only 10 items, all positive, it would go into an infinite loop. That's O(∞), I guess. So which one is it? I don't know...
I don't think there's a formal way of determining the big-O number for an algorithm. It's like the halting problem. In fact, come to think of it, if you could universally determine the big-O for a piece of code, you could also determine if it ever halts or not, thus contradicting the halting problem. But that's just my musings.
Typically I just go by... dunno, sort of a "gut feeling". Once you "get" what the Big-O represents, it becomes pretty intuitive. But for complicated algorithms it's not always possible to determine. Take Quicksort for example. On average it's O(N*logN), but depending on the data it can degrade to O(N*N). The questions you'll get on the test though should have clear answers.
The second one is 50 because big O is a function of the length of the input. That is if the input size changes from 1 million to 1 billion, the runtime should increase by 1000 if the function is O(N) and 1 million if it's O(n^2). However the second function runs in time 50 regardless of the input length, so it's O(1). Technically it would be O(50) but constants don't matter for big O.