We Keep Coding

Time complexity for all subsets using backtracking

I am trying to understand the time complexity while using backtracking. The problem is
Given a set of unique integers, return all possible subsets.
Eg. Input [1,2,3] would return [[],[1],[2],[1,2],[3],[1,3],[2,3],[1,2,3]]
I am solving it using backtracking as this:
private List<List<Integer>> result = new ArrayList<>();
public List<List<Integer>> getSubsets(int[] nums) {
for (int length = 1; length <= nums.length; length++) { //O(n)
backtrack(nums, 0, new ArrayList<>(), length);
}
result.add(new ArrayList<>());
return result;
}
private void backtrack(int[] nums, int index, List<Integer> listSoFar, int length) {
if (length == 0) {
result.add(listSoFar);
return;
}
for (int i = index; i < nums.length; i++) { // O(n)
List<Integer> temp = new ArrayList<>();
temp.addAll(listSoFar); // O(2^n)
temp.add(nums[i]);
backtrack(nums, i + 1, temp, length - 1);
}
}
The code works fine, but I am having trouble understanding the time/space complexity.
What I am thinking is here the recursive method is called n times. In each call, it generates the sublist that may contain max 2^n elements. So time and space, both will be O(n x 2^n), is that right?
Is that right? If not, can any one elaborate?
Note that I saw some answers here, like this but unable to understand. When recursion comes into the picture, I am finding it a bit hard to wrap my head around it.

You're exactly right about space complexity. The total space of the final output is O(n*2^n), and this dominates the total space used by the program. The analysis of the time complexity is slightly off though. Optimally, time complexity would, in this case, be the same as the space complexity, but there are a couple inefficiencies here (one of which is that you're not actually backtracking) such that the time complexity is actually O(n^2*2^n) at best.
It can definitely be useful to analyze a recursive algorithm's time complexity in terms of how many times the recursive method is called times how much work each call does. But be careful about saying backtrack is only called n times: it is called n times at the top level, but this is ignoring all the subsequent recursive calls. Also every call at the top level, backtrack(nums, 0, new ArrayList<>(), length); is responsible for generating all subsets sized length, of which there are n Choose length. That is, no single top-level call will ever produce 2^n subsets; it's instead that the sum of n Choose length for lengths from 0 to n is 2^n:
Knowing that across all recursive calls, you generate 2^n subsets, you might then want to ask how much work is done in generating each subset in order to determine the overall complexity. Optimally, this would be O(n), because each subset varies in length from 0 to n, with the average length being n/2, so the overall algorithm might be O(n/2*2^n) = O(n*2^n), but you can't just assume the subsets are generated optimally and that no significant extra work is done.
In your case, you're building subsets through the listSoFar variable until it reaches the appropriate length, at which point it is appended to the result. However, listSoFar gets copied to a temp list in O(n) time for each of its O(n) characters, so the complexity of generating each subset is O(n^2), which brings the overall complexity to O(n^2*2^n). Also, some listSoFar subsets are created which never figure into the final output (you never check to see that there are enough numbers remaining in nums to fill listSoFar out to the desired length before recursing), so you end up doing unnecessary work in building subsets and making recursive calls which will never reach the base case to get appended to result, which might also worsen the asymptotic complexity. You can address the first of these inefficiencies with back-tracking, and the second with a simple break statement. I wrote these changes into a JavaScript program, leaving most of the logic the same but re-naming/re-organizing a little bit:
function getSubsets(nums) {
let subsets = [];
for (let length = 0; length <= nums.length; length++) {
// refactored "backtrack" function:
genSubsetsByLength(length); // O(length*(n Choose length))
}
return subsets;
function genSubsetsByLength(length, i=0, partialSubset=[]) {
if (length === 0) {
subsets.push(partialSubset.slice()); // O(n): copy partial and push to result
return;
}
while (i < nums.length) {
if (nums.length - i < length) break; // don't build partial results that can't finish
partialSubset.push(nums[i]); // O(1)
genSubsetsByLength(length - 1, ++i, partialSubset);
partialSubset.pop(); // O(1): this is the back-tracking part
}
}
}
for (let subset of getSubsets([1, 2, 3])) console.log(`[`, ...subset, ']');
The key difference is using back-tracking to avoid making copies of the partial subset every time you add a new element to it, such that each is built in O(length) = O(n) time rather than O(n^2) time, because there is now only O(1) work done per element added. Popping off the last character added to the partial result after each recursive call allows you to re-use the same array across recursive calls, thus avoiding the O(n) overhead of making temp copies for each call. This, along with the fact that only subsets which appear in the final output are built, allows you to analyze the total time complexity in terms of the total number of elements across all subsets in the output: O(n*2^n).

Your code works not efficiently.
Like first solution in the link, you only think about the number will be included or not. (like getting combination)
It means, you don't have to iterate in getSubsets and backtrack function.
"backtrack" function could iterate "nums" array with parameter
private List<List<Integer>> result = new ArrayList<>();
public List<List<Integer>> getSubsets(int[] nums) {
backtrack(nums, 0, new ArrayList<>(), new ArrayList<>());
return result;
}
private void backtrack(int[] nums, int index, List<Integer> listSoFar)
// This function time complexity 2^N, because will search all cases when the number included or not
{
if (index == nums.length) {
result.add(listSoFar);
return;
}
// exclude num[index] in the subset
backtrack(nums, index+1, listSoFar)
// include num[index] in the subset
backtrack(nums, index+1, listSoFar.add(nums[index]))
}

What counts as a comparison in algorithm analysis?

MAIN QUESTION: When keeping track of comparisons, what actually counts as a comparison? Should I only count comparisons between array items since that's what the algorithm is meant for or is it more widely accepted to count every single comparison?
Currently, I am trying to wrap my head around the fact that I'm told that the theoretical number of comparisons for the worst case bubble sort algorithm is as follows:
Amount of comparisons:
(N-1) + (N-2) + (N-3) + ... + 2 + 1 = (N*(N-1))/2 = (N^2-N)/2 < N^2
So according to the formula (N^2-N)/2, with an input size (N) of 10, I would get a total of 45 comparisons. However, it is mentioned that this calculation only applies to the comparison operation in the inner loop of this pseudo code:
for i:=1 to N-1 do
{
for j:=0 to N-i do
{
if A[j] > A[j+1] // This is the comparison that's counted.
{
temp := A[j]
A[j] := A[j+1]
A[j+1] := temp
}
}
}
Now in Java, my code looks like this:
public int[] bubble(int[] array)
{
int comparisons = 0;
int exchanges = 0;
int temp;
int numberOfItems = array.length;
boolean cont = true;
comparisons++; // When pass == numberOfItems, a comparison will be made by the for loop that wouldn't otherwise be counted.
for (int pass=1; pass != numberOfItems; pass++)
{
comparisons = comparisons + 2; // Counts both the outer for loop comparison and the if statement comparison.
if (cont) // If any exchanges have taken place, cont will be true.
{
cont = false;
comparisons++; // Counts the inner for loop comparison
for (int index = 0; index != (numberOfItems - pass); index++)
{
comparisons++; // Counts the if statement comparison.
if (array[index] > array[index+1])
{
temp = array[index];
array[index] = array[index+1];
array[index+1] = temp;
cont = true;
exchanges++;
} // end inner if
} // end inner for
}
else
{
break; // end outer if
}
}
System.out.println("Comparisons = " + comparisons + "\tExchanges = " + exchanges);
return array;
}
After performing the worst case scenario on my code (using an array with 10 elements that are in the reverse order), I have gotten a total of 73 comparisons. This seems like a crazy high overshoot of the theoretical result which was 45 comparisons. This feels right to me though since I've accounted for all for loops and if statements.
Any help is greatly appreciated!
EDIT: I have noticed an error in my total comparison count for my inner loop. I wound up counting the inner loop twice before, but now it is fixed. Instead of getting 118 comparisons, I now get 73. However, the question still stands.

When measuring the number of comparisons in a sort, you only count comparisons between the array items. You count them whether or not they're actually in the array when you compare them.
The idea is that, instead of simple integers, the array might contain things that take a long time to compare. An array of of strings, for example, can be bubble-sorted using N(N-1)/2 string comparions, even though a single string comparison might require many other operations, including many comparisons of individual characters.
Measuring the performance of a sorting algorithm in terms of the number of comparisons makes the measurement independent of the type of things being sorted.

In evaluating sorting algorithms, it is common to count all comparisons between array elements as having equivalent cost, while ignoring comparisons between things like array indices. The basic concept is that in order for sorting operations to remain distinctly different from radix partitioning, the size of the items being sorted would need to increase as the number of them increased. Suppose, for example, one had an array of 1,000,000,000 char values and wanted to sort them. While one could use Quicksort, bubble sort, or something else, a faster approach would simply be to use an int[65536] and count how many of each value there are. Even if one needed to sort items which had char keys, the best way to do that would be to determine where to place the last item with a key of 0 (the number of items with a key of zero, minus one), where to place the last item with a key of 1 (number of items with keys of 0 or 1, minus one), etc. All such operations would take time proportional to the number of items plus the number of possible key values, without any lg(N) factor.
Note that if one ignores "bookkeeping" costs, algorithms like Quicksort aren't quite optimal. A sorting algorithm which is designed to maximize the amount of information gained from each comparison may do slightly fewer comparisons. Unless comparisons are very expensive, however, such a sorting algorithm would likely waste more time being "smart" than it would have spent being "stupid".
One issue I haven't seen discussed much, though I would think it could offer significant benefit in many real-world cases, would be optimizing sequences of comparisons between items that are known to be in a narrow range. If while performing a Quicksort on a series of thousand-character path names, one is processing a partition whose entries are all known to between two names that share the first 950 characters, there would be no need to examine the first 950 characters of any names in that partition. Such optimizations would not likely be meaningful in big-O terms unless key length was a parameter, but in the real world I would expect it could sometimes have an order-of-magnitude impact.

the comparison variable should only be incremented after the if statement has been reached in the execution of the code. The if statement is only reached if the condition stated in the outer and inner for loop have been met therefore the code should be like this.
Also dont forget to change the condition in the for loops from using != to <= The new java code:
public int[] bubble(int[] array)
{
int comparisons = 0;
int exchanges = 0;
int temp;
int numberOfItems = array.length;
boolean cont = true;
for (int pass=1; pass <= numberOfItems; pass++)
{
if (cont) // If any exchanges have taken place, cont will be true.
{
cont = false;
for (int index = 0; index <= (numberOfItems - pass); index++)
{
if (array[index] > array[index+1])
{ comparison++;
temp = array[index];
array[index] = array[index+1];
array[index+1] = temp;
cont = true;
exchanges++;
} // end inner if
} // end inner for
}
}
comparison++; // here you increment by one because you must also count the comparison that failed
System.out.println("Comparisons = " + comparisons + "\tExchanges = " + exchanges);
return array;
}

Time complexity modified bubblesort

Have this java code for bubblesort:
public void sort() {
for(int i = 1; i < getElementCount() ; ++i) {
for(int j = getElementCount()-1; j >= i; j--) {
if (cmp(j,j-1) < 0) swap(j, j-1);
}
}
}
the method "cmp" and "swap" are as follows:
public int cmp(int i, int j) {
return get(i).intValue()-get(j).intValue();
}
public void swap(int i, int j) {
Integer tmp = get(i);
set(i, get(j));
set(j, tmp);
}
I have now written an improved version of the Bubblesort where the sorting method "sort()" looks like this:
public void sort() {
boolean done = false;
for(int i = 1; i < getElementCount() && !done; ++i) {
done = true;
for(int j = getElementCount()-1; j >= i; j--) {
if (cmp(j,j-1) < 0) {
swap(j, j-1);
done = false;
}
}
}
}
Can anyone explain how to compute the time complexity of the latter algorithm? I'm thinking it's comparing n elements one time, and therefore it has complexity O(1) in its best case, and O(n^2) in it's worst case scenario, but I don't know if I'm right and would like to know how to think on this issue.

The complexity tells the programmer how long time it takes to process the data.
The O(1) complexity says that no matter how many elements it will only take one operation.
Insert a value in an array would have O(1)
E.g.
array[100] = value;
In your best case you will have to loop throught the entire array and compare each element.
Your complexity code is then O(n) where n is number of elements in the array.
In the worst case you will have to run through the array once for each element, that would give a complexity of O(n*n)

I just looked over what you've done and it is exactly the same as the one you had previously listed. You have set a boolean condition done = false and then you are checking the negation of it which will always evaluate to true - exactly the same logic as before. You can remove done in your code and you will see that it runs exactly the same. Just like before, you will have a best case complexity of O(n) and a worst case complexity of O(n^2). There is no way any sorting algorithm is O(1) as at the very least you at least have to move through the list once which gives O(n).

Worst Case : If the array is sorted in the sorted reverse order (descending) , it will show the worst time complexity of O(N^2).
Best Case : If the array is in sorted order, then the inner loop will go through each element at least once so it is O(N) - > (If you break out of the loop using the information in done, which is not present in the code).
At no point can it be a O(1) (In fact it is mathematically impossible to get a lower function than O(N) as the lower bounds for sorting is Omega(N) for comparison based sorts)
Omega(N) is the lowest possible function, as for comparison you have to see all elements at least once.

The best way is to represent your loops using Sigma notation like the following (General Case):
'c' here refers to the constant time of if, cmp, and swap that execute inside the innermost loop.
For the best case (modified bubble sort), the running time should look like this:

Efficiently determine the parity of a permutation

I have an int[] array of length N containing the values 0, 1, 2, .... (N-1), i.e. it represents a permutation of integer indexes.
What's the most efficient way to determine if the permutation has odd or even parity?
(I'm particularly keen to avoid allocating objects for temporary working space if possible....)

I think you can do this in O(n) time and O(n) space by simply computing the cycle decomposition.
You can compute the cycle decomposition in O(n) by simply starting with the first element and following the path until you return to the start. This gives you the first cycle. Mark each node as visited as you follow the path.
Then repeat for the next unvisited node until all nodes are marked as visited.
The parity of a cycle of length k is (k-1)%2, so you can simply add up the parities of all the cycles you have discovered to find the parity of the overall permutation.
Saving space
One way of marking the nodes as visited would be to add N to each value in the array when it is visited. You would then be able to do a final tidying O(n) pass to turn all the numbers back to the original values.

I selected the answer by Peter de Rivaz as the correct answer as this was the algorithmic approach I ended up using.
However I used a couple of extra optimisations so I thought I would share them:
Examine the size of data first
If it is greater than 64, use a java.util.BitSet to store the visited elements
If it is less than or equal to 64, use a long with bitwise operations to store the visited elements. This makes it O(1) space for many applications that only use small permutations.
Actually return the swap count rather than the parity. This gives you the parity if you need it, but is potentially useful for other purposes, and is no more expensive to compute.
Code below:
public int swapCount() {
if (length()<=64) {
return swapCountSmall();
} else {
return swapCountLong();
}
}
private int swapCountLong() {
int n=length();
int swaps=0;
BitSet seen=new BitSet(n);
for (int i=0; i<n; i++) {
if (seen.get(i)) continue;
seen.set(i);
for(int j=data[i]; !seen.get(j); j=data[j]) {
seen.set(j);
swaps++;
}
}
return swaps;
}
private int swapCountSmall() {
int n=length();
int swaps=0;
long seen=0;
for (int i=0; i<n; i++) {
long mask=(1L<<i);
if ((seen&mask)!=0) continue;
seen|=mask;
for(int j=data[i]; (seen&(1L<<j))==0; j=data[j]) {
seen|=(1L<<j);
swaps++;
}
}
return swaps;
}

You want the parity of the number of inversions. You can do this in O(n * log n) time using merge sort, but either you lose the initial array, or you need extra memory on the order of O(n).
A simple algorithm that uses O(n) extra space and is O(n * log n):
inv = 0
mergesort A into a copy B
for i from 1 to length(A):
binary search for position j of A[i] in B
remove B[j] from B
inv = inv + (j - 1)
That said, I don't think it's possible to do it in sublinear memory. See also:
https://cs.stackexchange.com/questions/3200/counting-inversion-pairs
https://mathoverflow.net/questions/72669/finding-the-parity-of-a-permutation-in-little-space

Consider this approach...
From the permutation, get the inverse permutation, by swapping the rows and
sorting according to the top row order. This is O(nlogn)
Then, simulate performing the inverse permutation and count the swaps, for O(n). This should give the parity of the permutation, according to this
An even permutation can be obtained as the composition of an even
number and only an even number of exchanges (called transpositions) of
two elements, while an odd permutation be obtained by (only) an odd
number of transpositions.
from Wikipedia.
Here's some code I had lying around, which performs an inverse permutation, I just modified it a bit to count swaps, you can just remove all mention of a, p contains the inverse permutation.
size_t
permute_inverse (std::vector<int> &a, std::vector<size_t> &p) {
size_t cnt = 0
for (size_t i = 0; i < a.size(); ++i) {
while (i != p[i]) {
++cnt;
std::swap (a[i], a[p[i]]);
std::swap (p[i], p[p[i]]);
}
}
return cnt;
}

generating all unique pairs from a list of numbers, n choose 2

i have a list of elements (let's say integers), and i need to make all possible 2-pair comparisons. my approach is O(n^2), and i am wondering if there is a faster way. here is my implementation in java.
public class Pair {
public int x, y;
public Pair(int x, int y) {
this.x = x;
this.y = y;
}
}
public List<Pair> getAllPairs(List<Integer> numbers) {
List<Pair> pairs = new ArrayList<Pair>();
int total = numbers.size();
for(int i=0; i < total; i++) {
int num1 = numbers.get(i).intValue();
for(int j=i+1; j < total; j++) {
int num2 = numbers.get(j).intValue();
pairs.add(new Pair(num1,num2));
}
}
return pairs;
}
please note that i don't allow self-pairing, so there should be ((n(n+1))/2) - n possible pairs. what i have currently works, but as n increases, it is taking me an unbearable long amount of time to get the pairs. is there any way to turn the O(n^2) algorithm above to something sub-quadratic? any help is appreciated.
by the way, i also tried the algorithm below, but when i benchmark, empirically, it performs worst than what i had above. i had thought that by avoiding an inner loop this would speed things up. shouldn't this algorithm below be faster? i would think that it's O(n)? if not, please explain and let me know. thanks.
public List<Pair> getAllPairs(List<Integer> numbers) {
int n = list.size();
int i = 0;
int j = i + 1;
while(true) {
int num1 = list.get(i);
int num2 = list.get(j);
pairs.add(new Pair(num1,num2));
j++;
if(j >= n) {
i++;
j = i + 1;
}
if(i >= n - 1) {
break;
}
}
}

Well, you can't, right?
The result is a list with n*(n-1)/2 elements, no matter what those elements are, just to populate this list (say with zeros) takes O(n^2) time, since n*(n-1)/2 = O(n^2)...

You cannot make it sub-quadric, because as you said - the output is itself quadric - and to create it, you need at least #elements_in_output ops.
However, you could do some "cheating" create your list on the fly:
You can create a class CombinationsGetter that implements Iterable<Pair>, and implement its Iterator<Pair>. This way, you will be able to iterate on the elements on the fly, without creating the list first, which might decrease latency for your application.
Note: It will still be quadric! The time to generate the list on the fly will just be distributed between more operations.
EDIT:
Another solution, which is faster then the naive approach - is multithreading.
Create a few threads, each will get his "slice" of the data - and generate relevant pairs, and create its own partial list.
Later - you can use ArrayList.addAll() to convert those different lists into one.
Note: though complexity is stiil O(n^2), it is likely to be much faster - since the creation of pairs is done in parallel, and ArrayList.addAll() is implemented much more effieciently then the trivial insert one by one elements.
EDIT2:
Your second code is still O(n^2), even though it is a "single loop" - the loop itself will repeat O(n^2) times. Have a look at your variable i. It increases only when j==n, and it decreases j back to i+1 when it does it. So, it will result in n + (n-1) + ... + 1 iterations, and this is sum of arithmetic progression, and gets us back to O(n^2) as expected.
We cannot get better then O(n^2), because we are trying to create O(n^2) distinct Pair objects.