We Keep Coding

Java perfomance issue with Arrays.sort [duplicate]

This question already has answers here:
Why is processing a sorted array *slower* than an unsorted array? (Java's ArrayList.indexOf)
(3 answers)
Closed 9 months ago.
I've been solving one algorithmic problem and found solution, as I thought. But unexpectedly I bumped into a weird problem.
Let's assume i have the following code on java 8/17(replicates on both), intel 11th gen processor:
import java.util.Arrays;
import java.util.concurrent.ThreadLocalRandom;
public class DistanceYandex{
static class Elem implements Comparable<Elem>{
int value;
int index;
long dist;
public Elem(int value, int index){
this.value = value;
this.index = index;
}
#Override
public int compareTo(Elem o){
return Integer.compare(value, o.value);
}
}
public static void main(String[] args){
int n = 300_000;
int k = 3_000;
Elem[] elems = new Elem[n];
for(int i = 0; i < n; i++){
elems[i] = new Elem(ThreadLocalRandom.current().nextInt(), i);
}
solve(n, k, elems);
}
private static void solve(int n, int k, Elem[] elems){
Arrays.sort(elems); // interesting line
long time = System.nanoTime();
for(int i = 0; i < n; i++){
elems[i].dist = findDistForIth(elems, i, k);
}
// i omit output, because it's irrelevant
// Arrays.sort(elems, Comparator.comparingInt(elem -> elem.index));
// System.out.print(elems[0].dist);
// for(int i = 1; i < n; i++){
// System.out.print(" " + elems[i].dist);
// }
System.out.println((System.nanoTime() - time)/1_000_000_000.0);
}
private static long findDistForIth(Elem[] elems, int i, int k){
int midElem = elems[i].value;
int left = i - 1;
int right = i + 1;
long dist = 0;
for(int j = 0; j < k; j++){
if(left < 0){
dist += elems[right++].value - midElem;
}else if(right >= elems.length){
dist += midElem - elems[left--].value;
}else{
int leftAdd = midElem - elems[left].value;
int rightAdd = elems[right].value - midElem;
if(leftAdd < rightAdd){
dist+=leftAdd;
left--;
}else{
dist+=rightAdd;
right++;
}
}
}
return dist;
}
}
Point your eyes at solve function.
Here we have simple solution, that calls function findDistForIth n times and measures time it takes(I don't use JMH, because testing system for my problem uses simple one-time time measures). And before it captures start time, it sorts the array by natural order using built-in Arrays.sort function.
As you could notice, measured time doesn't include the time the array gets sorted. Also function findDistForIth's behaviour does not depend on whether input array is sorted or not(it mostly goes to third else branch). But if I comment out line with Arrays.sort I get significantly faster execution: instead of roughly 7.3 seconds, it takes roughly 1.6 seconds. More that 4 times faster!
I don't understand what's going on.
I thought maybe it is gc that's messing up here, I tried to increase memory I give to jvm to 2gb(-Xmx2048M -Xms2048M). Didn't help.
I tried to pass explicit comparator to Arrays.sort as second argument(Comparator.comparingInt(e -> e.value)) and deimplementing Comparable interface on Elem class. Didn't help.
I launched the profiler(Intellij Profiler)
With Arrays.sort included:
With Arrays.sort excluded:
But it didn't give me much information...
I tried building it directly to .jar and launching via java cmd(before i did it via intellij). It also didn't help.
Do anybody know what's goind on?
This problem also replicates in online compiler: https://onlinegdb.com/MPyNIknB8T

May be you need to sort your data using red black tree sorting algo which implemented in SortedSet, Arrays.sort use mergesort sorting algo which works well for small number of data

Merge Sorting In Place

I am working on a Merge Sort method which sorts elements in-place without allocating extra memory. However, it isn't working as of now and I was wondering if anyone could help me, as I understand it is a relatively simple operation. Thank you in advance.
My merge sort method:
RegisterClass[] runMergeSort()
{
RegisterClass[] mergeSorted = new RegisterClass[capacity];
for (int counter = 0; counter < size; counter++)
{
mergeSorted[counter] = registerArray[counter];
}
runMergeSortHelper(mergeSorted, 0, size - 1);
return mergeSorted;
}
My helper method, which uses recursion:
private void runMergeSortHelper(RegisterClass[] workingArray,
int lowIndex,
int highIndex)
{
int midIndex = (lowIndex + highIndex) / 2;
if (lowIndex < highIndex)
{
runMergeSortHelper(workingArray, lowIndex, midIndex);
runMergeSortHelper(workingArray, midIndex+1, highIndex);
runMerge(workingArray, lowIndex, midIndex, highIndex);
}
}
And finally, my Merge method, which SHOULD be putting everything into order, however, it only does this partially.
private void runMerge(RegisterClass[] workingArray,
int lowIndex,
int midIndex,
int highIndex)
{
int counterJay = midIndex;
int counterAye = lowIndex;
int counterKay = lowIndex - 1;
while (counterAye < midIndex && counterJay <= highIndex)
{
counterKay++;
if (workingArray[counterAye].compareTo(workingArray[counterJay]) <= -1)
{
counterAye++;
}
else
{
swapValues(workingArray, counterAye, counterJay);
counterJay++;
counterAye++;
}
}
while (counterAye < midIndex)
{
counterKay++;
swapValues(workingArray, counterAye, counterKay);
counterAye++;
}
while (counterJay <= highIndex)
{
counterKay++;
swapValues(workingArray, counterJay, counterKay);
counterJay++;
}
}
Any advice at all would much be appreciated. I've looked online but nothing seems to help. Please do not refer me to a solution which is NOT an in-place solution.

Swapping isn't going to work with the logic used by the merge function. When a swap occurs, the element that is swapped from the left side to the right side is now out of order, and will be less than (or equal to) all of the remaining elements on the left side.
Whenever an element on the right side is found to be less than an element on the left side, a right rotate of that part of the array is needed to put the right side element into place.
Without resorting to a more complicated implementation, a small optimization can be made by scanning the right side for k = number of leading elements less than the current element on the left side, then do a rotate right by k elements. For random data, this won't help much, but for reverse sorted data, it would help quite a bit.

How to improve a recursive sorting algorithm that is already O(n)?

I have this recursive sorting algorithm I'm using for an assignment, and my teacher said that there's an easy way to improve the running time of my algorithm... But I can't figure out what it is, at all. Unless I'm mistaken, the complexity for the algorithm is O(n)? I'm not sure since we didn't learn how to calculate the complexity of recursive methods in class. Here's the code:
public static void MyAlgorithm(int[] A, int n){
boolean done = true;
int j = 0;
while (j <= n - 2){
if (A[j] > A[j + 1]) {
swap(A,j,j+1);
done= false;
}
j++;
}
j = n - 1;
while (j >= 1){
if (A[j] < A[j - 1]) {
swap(A,j-1,j);
done=false;
}
j--;
}
if (!done)
MyAlgorithm(A, n);
else
return;
}
The only thing I can think of would be adding a if(done) return; after the first loop but it only saves the program from doing a few other operations. Oh and the swap method is basically just:
public static void swap(int[] arr, int pos1, int pos2){
int temp = arr[pos1];
arr[pos1] = arr[pos2];
arr[pos2] = temp;
}
Thank you in advance.

To start, no sorting algorithm can be performed in O(n) using comparisons. As a general rule, all sorting algorithms take AT LEAST O(n*log(n)) time.
The sort you appear to be using is something akin to the cocktail shaker sort, or bidirectional bubble sort. It runs in O(n^2) time. You should definitely research the methods you use and consider why you use them, and also learn how to properly classify things in big O notation.
I imagine your teacher means that you should call the sort as MyAlgorithm(a, n-1). Notice how in your first loop it goes through the entire array? This means that the last element will already be sorted when that loop exits. Similarly, you could add a start index and increment it each time. For example, revised code:
public static void MyAlgorithm(int[] A, int start, int n){
boolean done = true;
int j = start;
while (j <= n - 2){
if (A[j] > A[j + 1]) {
swap(A,j,j+1);
done= false;
}
j++;
}
j = n - 1;
while (j >= start+1){
if (A[j] < A[j - 1]) {
swap(A,j-1,j);
done=false;
}
j--;
}
if (!done)
MyAlgorithm(A, start+1, n-1);
else
return;
}
Then you can call this using: MyAlgorithm(my_array, 0, my_array.length)
Keep in mind that this is still not a fantastic sorting algorithm, and if you ever need to sort large amount of data, you should consider using something faster.

Java seems to be executing bare-bones algorithms faster than C++. Why?

Introduction:
Using two identical mergesort algorithms, I tested the execution speed of C++ (using Visual Studios C++ 2010 express) vs Java (using NetBeans 7.0). I conjectured that the C++ execution would be at least slightly faster, but testing revealed that the C++ execution was 4 - 10 times slower than the Java execution. I believe that I have set all the speed optimisations for C++, and I am publishing as a release rather than as a debug. Why is this speed discrepancy occurring?
Code:
Java:
public class PerformanceTest1
{
/**
* Sorts the array using a merge sort algorithm
* #param array The array to be sorted
* #return The sorted array
*/
public static void sort(double[] array)
{
if(array.length > 1)
{
int centre;
double[] left;
double[] right;
int arrayPointer = 0;
int leftPointer = 0;
int rightPointer = 0;
centre = (int)Math.floor((array.length) / 2.0);
left = new double[centre];
right = new double[array.length - centre];
System.arraycopy(array,0,left,0,left.length);
System.arraycopy(array,centre,right,0,right.length);
sort(left);
sort(right);
while((leftPointer < left.length) && (rightPointer < right.length))
{
if(left[leftPointer] <= right[rightPointer])
{
array[arrayPointer] = left[leftPointer];
leftPointer += 1;
}
else
{
array[arrayPointer] = right[rightPointer];
rightPointer += 1;
}
arrayPointer += 1;
}
if(leftPointer < left.length)
{
System.arraycopy(left,leftPointer,array,arrayPointer,array.length - arrayPointer);
}
else if(rightPointer < right.length)
{
System.arraycopy(right,rightPointer,array,arrayPointer,array.length - arrayPointer);
}
}
}
public static void main(String args[])
{
//Number of elements to sort
int arraySize = 1000000;
//Create the variables for timing
double start;
double end;
double duration;
//Build array
double[] data = new double[arraySize];
for(int i = 0;i < data.length;i += 1)
{
data[i] = Math.round(Math.random() * 10000);
}
//Run performance test
start = System.nanoTime();
sort(data);
end = System.nanoTime();
//Output performance results
duration = (end - start) / 1E9;
System.out.println("Duration: " + duration);
}
}
C++:
#include <iostream>
#include <windows.h>
using namespace std;
//Mergesort
void sort1(double *data,int size)
{
if(size > 1)
{
int centre;
double *left;
int leftSize;
double *right;
int rightSize;
int dataPointer = 0;
int leftPointer = 0;
int rightPointer = 0;
centre = (int)floor((size) / 2.0);
leftSize = centre;
left = new double[leftSize];
for(int i = 0;i < leftSize;i += 1)
{
left[i] = data[i];
}
rightSize = size - leftSize;
right = new double[rightSize];
for(int i = leftSize;i < size;i += 1)
{
right[i - leftSize] = data[i];
}
sort1(left,leftSize);
sort1(right,rightSize);
while((leftPointer < leftSize) && (rightPointer < rightSize))
{
if(left[leftPointer] <= right[rightPointer])
{
data[dataPointer] = left[leftPointer];
leftPointer += 1;
}
else
{
data[dataPointer] = right[rightPointer];
rightPointer += 1;
}
dataPointer += 1;
}
if(leftPointer < leftSize)
{
for(int i = dataPointer;i < size;i += 1)
{
data[i] = left[leftPointer++];
}
}
else if(rightPointer < rightSize)
{
for(int i = dataPointer;i < size;i += 1)
{
data[i] = right[rightPointer++];
}
}
delete left;
delete right;
}
}
void main()
{
//Number of elements to sort
int arraySize = 1000000;
//Create the variables for timing
LARGE_INTEGER start; //Starting time
LARGE_INTEGER end; //Ending time
LARGE_INTEGER freq; //Rate of time update
double duration; //end - start
QueryPerformanceFrequency(&freq); //Determinine the frequency of the performance counter (high precision system timer)
//Build array
double *temp2 = new double[arraySize];
QueryPerformanceCounter(&start);
srand((int)start.QuadPart);
for(int i = 0;i < arraySize;i += 1)
{
double randVal = rand() % 10000;
temp2[i] = randVal;
}
//Run performance test
QueryPerformanceCounter(&start);
sort1(temp2,arraySize);
QueryPerformanceCounter(&end);
delete temp2;
//Output performance test results
duration = (double)(end.QuadPart - start.QuadPart) / (double)(freq.QuadPart);
cout << "Duration: " << duration << endl;
//Dramatic pause
system("pause");
}
Observations:
For 10000 elements, the C++ execution takes roughly 4 times the amount of time as the Java execution.
For 100000 elements, the ratio is about 7:1.
For 10000000 elements, the ratio is about 10:1.
For over 10000000, the Java execution completes, but the C++ execution stalls, and I have to manually kill the process.

I think there might be a mistake in the way you ran the program. When you hit F5 in Visual C++ Express, the program is running under debugger and it will be a LOT slower. In other versions of Visual C++ 2010 (e.g. Ultimate that I use), try hitting CTRL+F5 (i.e. Start without Debugging) or try running the executable file itself (in the Express) and you see the difference.
I run your program with only one modification on my machine (added delete[] left; delete[] right; to get rid of memory leak; otherwise it would ran out of memory in 32 bits mode!). I have an i7 950. To be fair, I also passed the same array to the Arrays.sort() in Java and to the std::sort in C++. I used an array size of 10,000,000.
Here are the results (time in seconds):
Java code: 7.13
Java Arrays.sort: 0.93
32 bits
C++ code: 3.57
C++ std::sort 0.81
64 bits
C++ code: 2.77
C++ std::sort 0.76
So the C++ code is much faster and even the standard library, which is highly tuned for in both Java and C++, tends to show slight advantage for C++.
Edit: I just realized in your original test, you run the C++ code in the debug mode. You should switch to the Release mode AND run it outside the debugger (as I explained in my post) to get a fair result.

I don't program C++ professionally (or even unprofessionally:) but I notice that you are allocating a double on the heap (double *temp2 = new double[arraySize];). This is expensive compared to Java initialisation but more importantly, it constitutes a memory leak since you never delete it, this could explain why your C++ implementation stalls, it's basically run out of memory.

To start with did you try using std::sort (or std::stable_sort which is typically mergesort) to get a baseline performance in C++?
I can't comment on the Java code but for the C++ code:
Unlike Java, new in C++ requires manual intervention to free the memory. Every recursion you'll be leaking memory. I would suggest using std::vector instead as it manages all the memory for you AND the iterator, iterator constructor will even do the copy (and possibly optimized better than your for loop`). This is almost certainly the cause of your performance difference.
You use arraycopy in Java but don't use the library facility (std::copy) in C++ although again this wouldn't matter if you used vector.
Nit: Declare and initialize your variable on the same line, at the point you first need them, not all at the top of the function.
If you're allowed to use parts of the standard library, std::merge could replace your merge algorithm.
EDIT: If you really are using say delete left; to cleanup memory that's probably your problem. The correct syntax would be delete [] left; to deallocate an array.

Your version was leaking so much memory that the timing were meaningless.
I am sure the time was spent thrashing the memory allocator.
Rewrite it to use standard C++ objects for memory management std::vector and see what happens.
Personally I would still expect the Java version to win (just). Because the JIT allows machine specific optimizations and while the C++ can do machine specific optimizations generally it will only do generic architecture optimizations (unless you provide the exact architecture flags).
Note: Don't forget to compile with optimizations turned on.
Just cleaning up your C++:
I have not tried to make a good merge sort (just re-written) in a C++ style
void sort1(std::vector<double>& data)
{
if(data.size() > 1)
{
std::size_t centre = data.size() / 2;
std::size_t lftSize = centre;
std::size_t rhtSize = data.size() - lftSize;
// Why are we allocating new arrays here??
// Is the whole point of the merge sort to do it in place?
// I forget bbut I think you need to go look at a knuth book.
//
std::vector<double> lft(data.begin(), data.begin() + lftSize);
std::vector<double> rht(data.begin() + lftSize, data.end());
sort1(lft);
sort1(rht);
std::size_t dataPointer = 0;
std::size_t lftPointer = 0;
std::size_t rhtPointer = 0;
while((lftPointer < lftSize) && (rhtPointer < rhtSize))
{
data[dataPointer++] = (lft[lftPointer] <= rht[rhtPointer])
? lft[lftPointer++]
: rht[rhtPointer++];
}
std::copy(lft.begin() + lftPointer, lft.end(), &data[dataPointer]);
std::copy(rht.begin() + rhtPointer, rht.end(), &data[dataPointer]);
}
}
Thinking about merge sort. I would try this:
I have not tested it, so it may not work correctly. Bu it is an attempt to not keep allocating huge amounts of memory to do the sort. instead it uses a single temp area and copies the result back when the sort is done.
void mergeSort(double* begin, double* end, double* tmp)
{
if (end - begin <= 1)
{ return;
}
std::size_t size = end - begin;
double* middle = begin + (size / 2);
mergeSort(begin, middle, tmp);
mergeSort(middle, end, tmp);
double* lft = begin;
double* rht = middle;
double* dst = tmp;
while((lft < middle) && (rht < end))
{
*dst++ = (*lft < *rht)
? *lft++
: *rht++;
}
std::size_t count = dst - tmp;
memcpy(begin, tmp, sizeof(double) * count);
memcpy(begin + count, lft, sizeof(double) * (middle - lft));
memcpy(begin + count, rht, sizeof(double) * (end - rht));
}
void sort2(std::vector<double>& data)
{
double* left = &data[0];
double* right = &data[data.size()];
std::vector<double> tmp(data.size());
mergeSort(left,right, &tmp[0]);
}

A couple of things.
Java is highly optimized and after the code has executed once the JIT compiler then executes the code as native.
Your System.arraycopy in Java is going to execute much faster than simply copying each element one at a time. try replacing this copy with a memcpy and you will see that it is much faster.
EDIT:
Look at this post: C++ performance vs. Java/C#

It is hard to tell from just looking at your code, but I would hazard a guess that the reason is in the handling recursion rather than actual computations. Try using some sorting algorithm that relies on the iteration instead of the recursion and share the results of the performance comparison.

I don't know why Java is so much faster here.
I compared it with the built in Arrays.sort() and it was 4x faster again. (It doesn't create any objects).
Usually if there is a test where Java is much faster its because Java is much better at removing code which doesn't do anything.
Perhaps you could use memcpy rather than a loop at the end.

Try to make a global vector as a buffer, and try not to allocate a lot of memory.
This will run faster than your code, because if uses some tricks(uses only one buffer and the memory is allocated when the program starts, so the memory will not be fragmented):
#include <cstdio>
#define N 500001
int a[N];
int x[N];
int n;
void merge (int a[], int l, int r)
{
int m = (l + r) / 2;
int i, j, k = l - 1;
for (i = l, j = m + 1; i <= m && j <= r;)
if (a[i] < a[j])
x[++k] = a[i++];
else
x[++k] = a[j++];
for (; i <= m; ++i)
x[++k] = a[i];
for (; j <= r; ++j)
x[++k] = a[j];
for (i = l; i <= r; ++i)
a[i] = x[i];
}
void mergeSort (int a[], int l, int r)
{
if (l >= r)
return;
int m = (l + r) / 2;
mergeSort (a, l, m);
mergeSort (a, m + 1, r);
merge (a, l, r);
}
int main ()
{
int i;
freopen ("algsort.in", "r", stdin);
freopen ("algsort.out", "w", stdout);
scanf ("%d\n", &n);
for (i = 1; i <= n; ++i)
scanf ("%d ", &a[i]);
mergeSort (a, 1, n);
for (i = 1; i <= n; ++i)
printf ("%d ", a[i]);
return 0;
}

QuickSort vs MergeSort, what am I doing wrong?

I am trying to implement several sorting algorithms in Java, to compare the performances. From what I've read, I was expecting quickSort to be faster than mergeSort, but on my code it is not, so I assume there must be a problem with my quickSort algorithm:
public class quickSortExample{
public static void main(String[] args){
Random gen = new Random();
int n = 1000000;
int max = 1500000;
ArrayList<Integer> d = new ArrayList<Integer>();
for(int i = 0; i < n; i++){
d.add(gen.nextInt(max));
}
ArrayList<Integer> r;
long start, end;
start = System.currentTimeMillis();
r = quickSort(d);
end = System.currentTimeMillis();
System.out.println("QuickSort:");
System.out.println("Time: " + (end-start));
//System.out.println(display(d));
//System.out.println(display(r));
}
public static ArrayList<Integer> quickSort(ArrayList<Integer> data){
if(data.size() > 1){
int pivotIndex = getPivotIndex(data);
int pivot = data.get(pivotIndex);
data.remove(pivotIndex);
ArrayList<Integer> smallers = new ArrayList<Integer>();
ArrayList<Integer> largers = new ArrayList<Integer>();
for(int i = 0; i < data.size(); i++){
if(data.get(i) <= pivot){
smallers.add(data.get(i));
}else{
largers.add(data.get(i));
}
}
smallers = quickSort(smallers);
largers = quickSort(largers);
return concat(smallers, pivot, largers);
}else{
return data;
}
}
public static int getPivotIndex(ArrayList<Integer> d){
return (int)Math.floor(d.size()/2.0);
}
public static ArrayList<Integer> concat(ArrayList<Integer> s, int p, ArrayList<Integer> l){
ArrayList<Integer> arr = new ArrayList<Integer>(s);
arr.add(p);
arr.addAll(l);
return arr;
}
public static String display(ArrayList<Integer> data){
String s = "[";
for(int i=0; i < data.size(); i++){
s += data.get(i) + ", ";
}
return (s+"]");
}
}
Results (on 1 million integer between 0 and 1500000):
mergeSort (implemented with arrayList too): 1.3sec (average) (0.7sec with int[] instead)
quickSort: 3sec (average)
Is it just the choice of my pivot which is bad, or are there some flaws in the algo too.
Also, is there a faster way to code it with int[] instead of ArrayList()? (How do you declare the size of the array for largers/smallers arrays?)
PS: I now it is possible to implement it in an inplace manner so it uses less memory, but this is not the point of this.
EDIT 1: I earned 1 sec by changing the concat method.
Thanks!

PS: I now it is possible to implement it in an inplace manner so it uses less memory, but this is not the point of this.
It's not just to use less memory. All that extra work you do in the "concat" routine instead of doing a proper in-place QuickSort is almost certainly what's costing so much. If you can use extra space anyway, you should always code up a merge sort because it'll tend to do fewer comparisons than a QuickSort will.
Think about it: in "concat()" you inevitably have to make another pass over the sub-lists, doing more comparisons. If you did the interchange in-place, all in a single array, then once you've made the decision to interchange two places, you don't make the decision again.

I think the major problem with your quicksort, like you say, is that it's not done in place.
The two main culprits are smallers and largers. The default size for an ArrayList is 10. In the initial call to quickSort a good pivot will mean that smallers and largers grow to 500,000. Since the ArrayList only doubles in size when it reaches capacity, it will have to be resized at around 19 times.
Since you are make a new smaller and larger with each level of recursion your going to be performing approximately 2*(19+18+...+2+1) resizes. That's around 400 resizes the ArrayList objects have to perform before they are even concatenated. The concatenation process will probably perform a similar number of resizes.
All in all, this is a lot of extra work.
Oops, just noticed data.remove(pivotIndex). The chosen pivot index (middle of the array) is also going to be causing additional memory operations (even though middle is usual a better choice than beginning or end or the array). That is arraylist will copy the entire block of memory to the 'right' of the pivot one step to the left in the backing array.
A quick note on the chosen pivot, since the integers you are sorting are evenly distributed between n and 0 (if Random lives up to its name), you can use this to choose good pivots. That is, the first level of quick sort should choose max*0.5 as its pivot. The second level with smallers should choose max*0.25 and the second level with largers should choose max*0.75 (and so on).

I think, that your algo is quite inefficient because you're using intermediate arrays = more memory + more time for allocation/copy. Here is the code in C++ but the idea is the same: you have to swap the items, and not copy them to another arrays
template<class T> void quickSortR(T* a, long N) {
long i = 0, j = N;
T temp, p;
p = a[ N/2 ];
do {
while ( a[i] < p ) i++;
while ( a[j] > p ) j--;
if (i <= j) {
temp = a[i]; a[i] = a[j]; a[j] = temp;
i++; j--;
}
} while ( i<=j );
if ( j > 0 ) quickSortR(a, j);
if ( N > i ) quickSortR(a+i, N-i);
}

Fundamentals of OOP and data structures in Java By Richard Wiener, Lewis J. Pinson lists quicksort as following, which may or may not be faster (I suspect it is) than your implementation...
public static void quickSort (Comparable[] data, int low, int high) {
int partitionIndex;
if (high - low > 0) {
partitionIndex = partition(data, low, high);
quickSort(data, low, partitionIndex - 1);
quickSort(data, partitionIndex + 1, high);
}
}
private static int partition (Comparable[] data, int low, int high) {
int k, j;
Comparable temp, p;
p = data[low]; // Partition element
// Find partition index(j).
k = low;
j = high + 1;
do {
k++;
} while (data[k].compareTo(p) <= 0 && k < high);
do {
j--;
} while (data[j].compareTo(p) > 0);
while (k < j) {
temp = data[k];
data[k] = data[j];
data[j] = temp;
do {
k++;
} while (data[k].compareTo(p) <= 0);
do {
j--;
} while (data[j].compareTo(p) > 0);
}
// Move partition element(p) to partition index(j).
if (low != j) {
temp = data[low];
data[low] = data[j];
data[j] = temp;
}
return j; // Partition index
}

I agree that the reason is unnecessary copying. Some more notes follow.
The choice of pivot index is bad, but it's not an issue here, because your numbers are random.
(int)Math.floor(d.size()/2.0) is equivalent to d.size()/2.
data.remove(pivotIndex); is unnecessary copying of n/2 elements. Instead, you should check in the following loop whether i == pivotIndex and skip this element. (Well, what you really need to do is inplace sort, but I'm just suggesting straightforward improvements.)
Putting all elements that are equal to pivot in the same ('smaller') part is a bad idea. Imagine what happens when all elements of the array are equal. (Again, not an issue in this case.)
for(i = 0; i < s.size(); i++){
arr.add(s.get(i));
}
is equivalent to arr.addAll(s). And of course, unnecessary copying here again. You could just add all elements from the right part to the left one instead of creating new list.
(How do you declare the size of the array for largers/smallers arrays?)
I'm not sure if I got you right, but do you want array.length?
So, I think that even without implementing in-place sort you can significantly improve performance.

Technically, Mergesort has a better time-behavior ( Θ(nlogn) worst and average cases ) than Quicksort ( Θ(n^2) worst case, Θ(nlogn) average case). So it is quite possible to find inputs for which Mergesort outperforms Quicksort. Depending on how you pick your pivots, you can make the worst-case rare. But for a simple version of Quicksort, the "worst case" will be sorted (or nearly sorted) data, which can be a rather common input.
Here's what Wikipedia says about the two:
On typical modern architectures,
efficient quicksort implementations
generally outperform mergesort for
sorting RAM-based arrays. On the other
hand, merge sort is a stable sort,
parallelizes better, and is more
efficient at handling slow-to-access
sequential media.[citation needed]
Merge sort is often the best choice
for sorting a linked list: in this
situation it is relatively easy to
implement a merge sort in such a way
that it requires only Θ(1) extra
space, and the slow random-access
performance of a linked list makes
some other algorithms (such as
quicksort) perform poorly, and others
(such as heapsort) completely
impossible.