Many times I have to sort large amounts of small lists, arrays. It is quite rare that I need to sort large arrays. Which is the fastest sort algorithm for sorting:
arrays
(array)lists
of size 8-15 elements of these types:
integer
string from 10-40 characters
?
I am listing element types because some algorithms do more compare operations and less swap operations.
I am considering Merge Sort, Quick Sort, Insertion Sort and Shell sort (2^k - 1 increment).
Arrays.sort(..) / Collections.sort(..) will make that decision for you.
For example, the openjdk-7 implementation of Arrays.sort(..) has INSERTION_SORT_THRESHOLD = 47 - it uses insertion sort for those with less than 47 elements.
Unless you can prove that this is a bottleneck then the built in sorts are fine:
Collections.sort(myIntList);
Arrays.sort(myArray);
Actually, there isn't a universal answer. Among other things, the performance of a Java sort algorithm will depend on the relative cost of the compare operation, and (for some algorithms) on the order of the input. In the case of a list, it also depends on the list implementation type.
But #Bozho's advice is sound, as is #Sean Patrick Floyd's comment.
FOLLOWUP
If you believe that the performance difference is going to be significant for your use-case, then you should get hold of some implementations of different algorithms, and test them out using the actual data that your application needs to deal with. (And if you don't have the data yet, it is too soon to start tuning your application, because the sort performance will depend on actual data.)
In short, you'll need to do the benchmarking yourself.
Related
The following quote is from "Comparison with other sort algorithms"
section from Wikipedia Merge Sort page
On typical modern architectures, efficient quicksort implementations
generally outperform mergesort for sorting RAM-based arrays.[citation
needed] On the other hand, merge sort is a stable sort and is more
efficient at handling slow-to-access sequential media.
My questions:
Why does Quicksort outperform Mergesort when the data to be sorted can all fit into memory? If all data needed are cached or in memory wouldn't it be fast for both Quicksort and Mergesort to access?
Why is Mergesort more efficient at handling slow-to-access sequential data (such as from disk in the case where the data to be sorted can't all fit into memory)?
(move from my comments below to here)In an array arr of primitives (data are sequential) of n elements. The pair of elements that has to be read and compared in MergeSort is arr[0] and arr[n/2] (happens in the final merge). Now think the pair of elements that has to be read and compared in QuickSort is arr[1] and arr[n] (happens in the first partition, assume we swap the randomly chosen pivot with the first element). We know data are read in blocks and load into cache, or disk to memory (correct me if I am wrong) then isn't there a better chance for the needed data gets load together in one block when using MergeSort? It just seems to me MergeSort would always have the upperhand because it is likely comparing elements that are closer together. I know this is False (see graph below) because QuickSort is obviously faster...... I know MergeSort is not in place and requires extra memory and that is likely to slow things down. Other than that what pieces am I missing in my analysis?
images are from Princeton CS MergeSort and QuickSort slides
My Motive:
I want to understand these above concepts because they are one of the main reasons of why mergeSort is preferred when sorting LinkedList,or none sequential data and quickSort is preferred when sorting Array, or sequential data. And why mergeSort is used to sort Object in Java and quickSort is used to sort primitive type in java.
update: Java 7 API actually uses TimSort to sort Object, which is a hybrid of MergeSort and InsertionSort. For primitives Dual-Pivot QuickSort. These changes were implemented starting in Java SE 7. This has to do with the stability of the sorting algorithm. Why does Java's Arrays.sort method use two different sorting algorithms for different types?
Edit:
I will appreciate an answer that addresses the following aspects:
I know the two sorting algorithms differ in the number of moves, read, and comparisons. If those are that reasons contribute to the behaviors I see listed in my questions (I suspected it) then a thorough explanation of how the steps and process of the sorting algorithm results it having advantages or disadvantages seeking data from disk or memory will be much appreciated.
Examples are welcome. I learn better with examples.
note: if you are reading #rcgldr's answer. check out our conversation in the chat room it has lots of good explanations and details. https://chat.stackoverflow.com/rooms/161554/discussion-between-rcgldr-and-oliver-koo
The main difference is that merge sort does more moves, but fewer compares than quick sort. Even in the case of sorting an array of native types, quick sort is only around 15% faster, at least when I've tested it on large arrays of pseudo random 64 bit unsigned integers, which should be quick sort's best case, on my system (Intel 3770K 3.5ghz, Windows 7 Pro 64 bit, Visual Studio 2015, sorting 16 million pseudo random 64 bit unsigned integers, 1.32 seconds for quick sort, 1.55 seconds for merge sort, 1.32/1.55 ~= 0.85, so quick sort was about 15% faster than merge sort). My test was with a quick sort that had no checks to avoid worst case O(n^2) time or O(n) space. As checks are added to quick sort to reduce or prevent worst case behavior (like fall back to heap sort if recursion becomes too deep), the speed advantage decreases to less than 10% (which is the difference I get between VS2015's implementation of std::sort (modified quick sort) versus std::stable_sort (modified merge sort).
If sorting "strings", it's more likely that what is being sorted is an array of pointers (or references) to those strings. This is where merge sort is faster, because the moves involve pointers, while the compares involve a level of indirection and comparison of strings.
The main reason for choosing quick sort over merge sort is not speed, but space requirement. Merge sort normally uses a second array the same size as the original. Quick sort and top down merge sort also need log(n) stack frames for recursion, and for quick sort limiting stack space to log(n) stack frames is done by only recursing on the smaller partition, and looping back to handle the larger partition.
In terms of cache issues, most recent processors have 4 or 8 way associative caches. For merge sort, during a merge, the two input runs will end up in 2 of the cache lines, and the one output run in a 3rd cache line. Quick sort scans the data before doing swaps, so the scanned data will be in cache, although in separate lines if the two elements being compared / swapped are located far enough from each other.
For an external sort, some variation of bottom up merge sort is used. This because merge sort merge operations are sequential (the only random access occurs when starting up a new pair of runs), which is fast in the case of hard drives, or in legacy times, tape drives (a minimum of 3 tapes drives is needed). Each read or write can be for very large blocks of data, reducing average access time per element in the case of a hard drive, since a large number of elements are read or written at a time with each I/O.
It should also be noted that most merge sorts in libraries are also some variation of bottom up merge sort. Top down merge sort is mostly a teaching environment implementation.
If sorting an array of native types on a processor with 16 registers, such as an X86 in 64 bit mode, 8 of the registers used as start + end pointers (or references) for 4 runs, then a 4-way merge sort is often about the same or a bit faster than quick sort, assuming a compiler optimizes the pointers or references to be register based. It's a similar trade off, like quick sort, 4-way merge sort does more compares (1.5 x compares), but fewer moves (0.5 x moves) than traditional 2-way merge sort.
It should be noted that these sorts are cpu bound, not memory bound. I made a multi-threaded version of a bottom up merge sort, and in the case of using 4 threads, the sort was 3 times faster. Link to Windows example code using 4 threads:
https://codereview.stackexchange.com/questions/148025/multithreaded-bottom-up-merge-sort
The array in question may hold any integer which equals or is bigger than zero and the numbers are unique. The numbers have to be in ascending order.
The array's size will usually be less than 100.
Most of the array is already sorted. By most I mean on avarage atleast 90% of it.
I've found this implementation of TimSort but it is not for primitive values. Autoboxing would cause a lot of overhead.
Performance is most crucial as the sorting-algorithm will be called many times.
Use Arrays.sort:
int[] array = /* something */;
Arrays.sort(array);
Being only one line, this is (obviously) both extremely simple to use and very readable. That should be your #1 priority when writing code. It's also going to be pretty darn fast, because the writers of the standard library have put a lot of effort into performance, particularly relating to sorting algorithms.
The only situation in which you should not use Arrays.sort is if you've almost entirely finished your system, profiled it carefully, and determined that the part of the code that sorts your array is the bottleneck. Even then, you still might not be able to write your own sorting algorithm that performs noticeably better.
Depends what you mean by "almost sorted". Insertion Sort is a very efficient algorithm if the array is nearly sorted (linear complexity when it is sorted), but the performance can vary depending on whether the outliers are close or far from their final sorted position. For example, [1,2,3,4,6,5,7,8,9] will be slightly faster to sort than [1,3,4,5,6,7,8,9,2].
I have an array that I need to sort the values in increasing order. The possible value inside the array are is between 1-9, there will be a lot of repeating value. (fyi: I'm working on a sudoku solver and trying to solve the puzzle starting with the box with least possibilities using backtracking)
The first idea that comes to my mine is to use Shell Sort.
I did some look up and I found out that the java collection uses "modified mergesort"(in which the merge is omitted if the highest element in the low sublist is less than the lowest element in the high sublist).
So I wish to know if the differences in performance will be noticeable if I implement my own sorting algorithm.
If you only have 9 possible values, you probably want counting sort - the basic idea is:
Create an array of counts of size 9.
Iterate through the array and increment the corresponding index in the count array for each element.
Go through the count array and recreate the original array.
The running time of this would be O(n + 9) = O(n), where-as the running time of the standard API sort will be O(n log n).
So yes, this will most likely be faster than the standard comparison-based sort that the Java API uses, but only a benchmark will tell you for sure (and it could depend on the size of your data).
In general, I'd suggest that you first try using the standard API sort, and see if it's fast enough - it's literally just 1 line of code (except if you have to define a comparison function), compared to quite a few more for creating your own sorting function, and quite a bit of effort has gone into making sure it's as fast as possible, while keeping it generic.
If that's not fast enough, try to find and implement a sort that works well with your data. For example:
Insertion sort works well on data that's already almost sorted (although the running time is pretty terrible if the data is far from sorted).
Distribution sorts are worth considering if you have numeric data.
As noted in the comment, Arrays.parallelSort (from Java 8) is also an option worth considering, since it multi-threads the work (which sort doesn't do, and is certainly quite a bit of effort to do yourself ... efficiently).
I'm looking for a data structure or a combination of various data structures that perform very well on random and sequential access.
I need to map an (integer) id to a (double) value and sort by that value. The values can occur multiple times.
The amount of data can possibly be large.
Insertion or deletion are not critical. Iteration and Get Operations are.
I'm using Java. Currently I have a Guava Multimap, built from a TreeMap and ArrayList for sequential access. For random access I use a HashMap in parallel.
Any suggestions?
When insertion and deletion are not critical, then a sorted array might be your friend. You could search there directly via Arrays.binarySearch and you custom Comparator.
In case you don't know any sane upper bound on the size, you can switch to an ArrayList (or implement you own resizing, but why...).
I guess this could be faster then the TreeMap, which is good when insertion and/or deletion are important, but suffers from bad spatial locality (binary tree with many pointers to follow).
The optimal structure would place all the data in a single array, which is impossible in Java (you'd need C struct for this). You could fake it by placing doubles into longs, this is sure to work and to be fast (Double.doubleToLongBits and back are intrinsics, and the length of both datatypes is 64 bits). This would mean a non-trivial amount of work, especially for sorting (if this is uncommon enough, a conversion in some sortable array and back would do).
In order to get faster search, you could use hashing, e.g., via a HashMap pointing to first element and linking the elements. As you keys are ints, some primitive-capable implementation would help (e.g. trove or fastutils or whatever).
There are countless possibilities, but keeping all your data in sync can be hard.
I can't find the documentation of Java 7, I can only find about the Java 6, which is still quick or merge. Does anyone know how to find the documentation of the method Arrays.sort in Java 7?
Java 7 uses Dual-Pivot Quicksort for primitives and TimSort for objects.
According to the Java 7 API doc for primitives:
Implementation note: The sorting
algorithm is a Dual-Pivot Quicksort by
Vladimir Yaroslavskiy, Jon Bentley,
and Joshua Bloch. This algorithm
offers O(n log(n)) performance on many
data sets that cause other quicksorts
to degrade to quadratic performance,
and is typically faster than
traditional (one-pivot) Quicksort
implementations.
According to the Java 7 API doc for objects:
The implementation was adapted from
Tim Peters's list sort for Python (
TimSort). It uses techiques from Peter
McIlroy's "Optimistic Sorting and
Information Theoretic Complexity", in
Proceedings of the Fourth Annual
ACM-SIAM Symposium on Discrete
Algorithms, pp 467-474, January 1993.
Timsort is a hybrid "merge sort and insertion sort."
Not sure if this is much different from what it was in Java 6, for Arrays.sort JDK6:
a tuned quicksort, adapted from Jon L.
Bentley and M. Douglas McIlroy's
"Engineering a Sort Function",
Software-Practice and Experience, Vol.
23(11) P. 1249-1265 (November 1993)
For Object[] or collections (Collections.sort()) merge sort is used.
Yes! ... and also no.
Summary
At the time of writing (2016), in current Open JDK 0 implementations Tim Sort is generally used for sorting arrays of objects (i.e., Arrays.sort(Object[]) and friends) - but for primitive arrays (the remainder of the Arrays.sort methods) a variety of other methods are used.
For primitives, the heuristics choose among a variety of sorting methods such as quicksort, merge sort, counting sort3. depending on the data being sorted. Most of these decisions are simply made up-front based on the type and size of the array being sorted, but for int and long elements the decision is actually adaptive based on the measured sortedness of the array. So you have adaptation/introspection (the heuristics to pick an algorithm) on top of adaptation/introspection (TimSort or similar merge sort) in many cases!
Details
Tim Sort is used for most sorts of objects, such as Arrays.sort(Object[] a), unless the user has specifically requested the legacy behavior by setting system property java.util.Arrays.useLegacyMergeSort to true.
For primitives, the situation is more complex. At least as of JDK 8 (version 1.8.0_111) a variety of heurstics are used depending on the size of the arrays being sorted, the primitive type and the measured "sortedness" of the array. Here's an overview:
For all primitive types other than bytes1, arrays of less than 47 elements are simply sorted using insertion sort (see DualPivotQuicksort.INSERTION_SORT_THRESHOLD). This threshold is also used when sorting sub-arrays that arise when merge or quicksort are used and the size of the subarray falls below the threshold. So some form of insertion sort will be used in all sorts, and for small arrays it is the only algorithm used.
For primitive types byte, short and char, a counting sort is used for largish arrays. This is a simple sort that takes O(n + range) time, where range is the total number of byte (256) or short/char (65536) values. The sort involves allocating an underlying array of range values, so it is only used when the number of elements to sort is a significant fraction of the total range. In particular, it is used for byte arrays greater than 29 elements (i.e. ~11% of the range), and short/char arrays greater than 3200 elements (~5% of the range).
For byte arrays, one of the two approaches above is always used.
For int and long arrays above the insertion sort threshold and for short/char arrays both above the insertion sort threshold and below the counting sort threshold, one of two algorithms may be used: dual pivot quicksort, or merge sort. Which one is used depends on a measure of the sortedness of the array: the input is divided into runs of ascending or descending elements. If the number of such runs is greater than 66, then the array is considered mostly unsorted and is sorted with dual-pivot quicksort. Otherwise, the array is considered mostly sorted, and mergesort is used (using the already enumerated runs as a starting point).
The idea of finding runs and then using mergesort to sort them is in fact very similar to TimSort, although there are some differences. So at least for some parameters, the JDK is using a run-aware mergesort, but for many other combinations of parameters it is using a different algorithm, and at least 5 distinct algorithms are used in total!
Rationale
The reasoning behind the different sort behavior of Object[] versus primitive is probably at least two-fold:
1) Sorts of Object[] are required to be stable: objects which sort equally will appear in the same order as the input. For primitive arrays, no such concept exists: primitives are fully defined by their value, so there is no distinction between a stable and an unstable sort. This allows primitive sorts to dispense with the need for stable algorithms in favor of speed.
2) Sorts of Object[] need to involve the Object.compare() method, which may be arbitrarily complex and expensive. Even if the compare() method is simple, there will generally be method call overhead unless the entire sort method can be inlined2. So sorts of Object[] will generally be biased towards minimizing total comparisons, even at the cost of some additional algorithmic complexity.
Sorts of primitive arrays, on the other hand, just directly compare primitive values which typically takes on the order of a cycle or two. In this case, the algorithm should be optimized considering both the cost of comparisons and the surrounding algorithm, since the are likely to be of the same magnitude.
0 At least for versions between Java 7 and Java 9, and of course this also includes Oracle's JDK as it is based on Open JDK. It is likely that other implementations use a similar approach, but I haven't checked.
1 For byte arrays, the insertion sort threshold is effectively 29 elements since that's the lower cutoff above which counting sort is used.
2 This seems unlikely, since it is quite large.
3 Counting sort is only used for values with relatively limited range of 16-bits or less: byte, short or char.
Yes, Java 7 will use Timsort for Arrays.sort. Here is the commit:
http://hg.openjdk.java.net/jdk7/jdk7/jdk/rev/bfd7abda8f79