I used the following code to test the performance between Array/ArrayList/LinkedList
import java.util.ArrayList;
import java.util.LinkedList;
public class Main3 {
public static void main(String[] args) throws Exception{
int n = 20000000;
long bt = 0, et = 0;
int[] a0 = new int[n];
ArrayList<Integer> a1 = new ArrayList<>(n);
LinkedList<Integer> a2 = new LinkedList<>();
Integer[] a3 = new Integer[n];
bt = System.currentTimeMillis();
for(int i=0; i<n; i++){
a0[i] = i;
}
et = System.currentTimeMillis();
System.out.println("===== loop0 time =======" + (et - bt));
bt = System.currentTimeMillis();
for(int i=0; i<n; i++){
a1.add(i);
}
et = System.currentTimeMillis();
System.out.println("===== loop1 time =======" + (et - bt));
bt = System.currentTimeMillis();
for(int i=0; i<n; i++){
a2.add(i);
}
et = System.currentTimeMillis();
System.out.println("===== loop2 time =======" + (et - bt));
bt = System.currentTimeMillis();
for(int i=0; i<n; i++){
a3[i] = i;
}
et = System.currentTimeMillis();
System.out.println("===== loop3 time =======" + (et - bt));
}
}
The result is
===== loop0 time =======11
===== loop1 time =======6776
===== loop2 time =======17305
===== loop3 time =======56
Why the ArralyList/LinkedList is so slower than array ?
How could I improve the performance.
env:
Java: jdk1.8.0_231
Thanks
There are potential inaccuracies in your benchmark, but the overall ranking of the results is probably correct. You may get faster results for all of the benchmarks if you "warm-up" the code before taking timings to allow the JIT compiler to generate native code and optimise it. Some benchmark results may be closer or even equal.
Iterating over an int array is going to be much faster than iterating over a List of Integer objects. A LinkedList is going to be slowest of all. These statements assume the optimiser doesn't make radical changes.
Let's look at why:
An int array (int[]) is a contiguous area of memory containing your 4 byte ints arranged end-to-end. The loop to iterate over this and set the elements just has to work its way through the block of memory setting each 4 bytes in turn. In principle an index check is required, but in practice the optimiser can realise this isn't necessary and remove it. The JIT compiler is well able to optimise this kind of thing based on native CPU instructions.
An ArrayList of Integer objects contains an array of references which point to individual Integer objects (or are null). Each Integer object will have to be allocated separately (although Java can re-use Integers of small numbers). There is an overhead to allocate new objects and in addition the reference may be 8 bytes instead of 4. Also, if the list size is not specified (though it is in your case) the internal array may need to be reallocated. There is an overhead due to calling the add method instead of assigning to the array directly (the optimizer may remove this though).
Your array of Integer benchmark is similar to the array list but doesn't have the overhead of the list add method call (which has to track the list size). Probably your benchmark overstates the difference between this array and the array list though.
A LinkedList is the worst case. Linked lists are optimised for inserting in the middle. They have references to point to the next item in the list and nodes to hold those references in addition to the Integer object that needs allocating. This is a big memory overhead that also requires some initialisation and you would not use a linked list unless you were expecting to insert a lot of elements into the middle of the list.
Related
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 4 years ago.
Improve this question
I did some research and wrote the following article: http://www.heavyweightsoftware.com/blog/linkedlist-vs-arraylist/ and wanted to post a question here.
class ListPerformanceSpec extends Specification {
def "Throwaway"() {
given: "A Linked List"
List<Integer> list
List<Integer> results = new LinkedList<>()
when: "Adding numbers"
Random random = new Random()
//test each list 100 times
for (int ix = 0; ix < 100; ++ix) {
list = new LinkedList<>()
LocalDateTime start = LocalDateTime.now()
for (int jx = 0; jx < 100000; ++jx) {
list.add(random.nextInt())
}
LocalDateTime end = LocalDateTime.now()
long diff = start.until(end, ChronoUnit.MILLIS)
results.add(diff)
}
then: "Should be equal"
true
}
def "Linked list"() {
given: "A Linked List"
List<Integer> list
List<Integer> results = new LinkedList<>()
when: "Adding numbers"
Random random = new Random()
//test each list 100 times
for (int ix = 0; ix < 100; ++ix) {
list = new LinkedList<>()
LocalDateTime start = LocalDateTime.now()
for (int jx = 0; jx < 100000; ++jx) {
list.add(random.nextInt())
}
long total = 0
for (int jx = 0; jx < 10000; ++jx) {
for (Integer num : list) {
total += num
}
total = 0
}
LocalDateTime end = LocalDateTime.now()
long diff = start.until(end, ChronoUnit.MILLIS)
results.add(diff)
}
then: "Should be equal"
System.out.println("Linked list:" + results.toString())
true
}
def "Array list"() {
given: "A Linked List"
List<Integer> list
List<Integer> results = new LinkedList<>()
when: "Adding numbers"
Random random = new Random()
//test each list 100 times
for (int ix = 0; ix < 100; ++ix) {
list = new ArrayList<>()
LocalDateTime start = LocalDateTime.now()
for (int jx = 0; jx < 100000; ++jx) {
list.add(random.nextInt())
}
long total = 0
for (int jx = 0; jx < 10000; ++jx) {
for (Integer num : list) {
total += num
}
total = 0
}
LocalDateTime end = LocalDateTime.now()
long diff = start.until(end, ChronoUnit.MILLIS)
results.add(diff)
}
then: "Should be equal"
System.out.println("Array list:" + results.toString())
true
}
}
Why does ArrayList outperform LinkedList by 28% for sequential access when LinkedList should be faster?
My question is different from When to use LinkedList over ArrayList? because I'm not asking when to choose it, but why it's faster.
Array-based lists, as Java ArrayList, use much less memory for the same data amount than link-based lists (LinkedList), and this memory is organized sequentially. This essentially decreases CPU cache trashing with side data. As soon as access to RAM requires 10-20 times more delay than L1/L2 cache access, this is causing sufficient time difference.
You can read more about these cache issues in books like this one, or similar resources.
OTOH, link-based lists outperform array-based ones in operation like insering to middle of list or deleting there.
For a solution that have both memory economy (so, fast iterating) and fast inserting/deleting, one should look at combined approaches, as in-memory B⁺-trees, or array of array lists with proportionally increased sizes.
From LinkedList source code:
/**
* Appends the specified element to the end of this list.
*
* <p>This method is equivalent to {#link #addLast}.
*
* #param e element to be appended to this list
* #return {#code true} (as specified by {#link Collection#add})
*/
public boolean add(E e) {
linkLast(e);
return true;
}
/**
* Links e as last element.
*/
void linkLast(E e) {
final Node<E> l = last;
final Node<E> newNode = new Node<>(l, e, null);
last = newNode;
if (l == null)
first = newNode;
else
l.next = newNode;
size++;
modCount++;
}
From ArrayList source code:
/**
* Appends the specified element to the end of this list.
*
* #param e element to be appended to this list
* #return <tt>true</tt> (as specified by {#link Collection#add})
*/
public boolean add(E e) {
ensureCapacityInternal(size + 1); // Increments modCount!!
elementData[size++] = e;
return true;
}
private void ensureExplicitCapacity(int minCapacity) {
modCount++;
// overflow-conscious code
if (minCapacity - elementData.length > 0)
grow(minCapacity);
}
So linked list has to create new node for each element added, while array list does not. ArrayList does not reallocate/resize for each new element, so most of time array list simply set object in array and increment size, while linked list does much more work.
You also commented:
When I wrote a linked list in college, I allocated blocks at a time and then farmed them out.
I do not think this would work in Java. You cannot do pointer tricks in Java, so you would have to allocate a lot of small arrays, or create empty nodes ahead. In both cases overhead would probably be a bit higher.
Why does ArrayList outperform LinkedList by 28% for sequential access when LinkedList should be faster?
You're assuming that, but don't provide anything to back it up. But it's not really a great surprise. An ArrayList has an array as the underlying data store. Accessing this sequentially is extremely fast, because you know exactly where every element is going to be. The only slowdown comes when the array grows beyond a certain size and needs to be expanded, but that can be optimised.
The real answer would probably be: check the Java source code, and compare the implementations of ArrayList and LinkedList.
One explanation is that your base assumption (that multiplication is slower than memory fetches) is questionable.
Based on this document, a AMD Bulldozer takes 1 clock cycles to perform a 64 bit integer multiply instruction (register x register) with 6 cycles of latency1. By contrast, a memory to register load takes 1 clock cycle with 4 cycles of latency. But that assumes that you get a cache hit for the memory fetch. If you get a cache miss, you need to add a number of cycles. (20 clock cycles for an L2 cache miss, according to this source.)
Now that is just one architecture, and others will vary. And we also need to consider other issues, like constraints on the number of multiplications that can be overlapped, and how well the compiler can organize the instructions to get them minimize instruction dependencies. But the fact remains that for a typical modern pipelined chip architecture, the CPU can execute integer multiplies as fast as it can execute memory to register moves, and much faster if there are more cache misses in the memory fetches.
Your benchmark is using lists with 100,000 Integer elements. When you look at the amount of memory involved, and the relative locality of the heap nodes that represent the lists and the elements, the linked list case will use significantly more memory, and have correspondingly worse memory locality. That will lead to more cache misses per cycle of the inner loop, and worse performance.
Your benchmark results are not surprising2 to me.
The other thing to note is that if you use Java LinkedList, a separate heap node is used to represent the list nodes. You can implement your own linked lists more efficiently if your element class has its own next field that can be used to chain the elements. However, brings its own limitations; e.g. an element can only be in one list at a time.
Finally, as #maaartinus points out, a full IMUL is not required in the case of a Java ArrayList. When reading or writing the ArrayList's array, the indexing multiplication will be either x 4 or x 8 and that can be performed by a MOV with one of the standard addressing modes; e.g.
MOV EAX, [EDX + EBX*4 + 8]
This multiplication can be done (at the hardware level) by shifting with much less latency than 64 bit IMUL.
1 - In this context, the latency is the number of cycles delay before the result of the instruction is available ... to the next instruction that depends on it. The trick is to order the instructions so that other work is done during the delay.
2 - If anything, I am surprised that LinkedList appears to be doing so well. Maybe calling Random.nextInt() and autoboxing the result is dominating the loop times?
In my exam's first question : I am working on a small task where I am required to store around 500Million+ elements in an Array.
However, I am running into a heap space problem. Could you please help me with this to best optimal storage algorithm ?
I found "BitSet" but I dont know how to use it.
Step 1 - Create 3 long[] arrays with very large length (Least 100M+)
Step 2 - Init values should be randomly generated, not sorted, may contain duplicates
Step 3 - Merge them after init with randomly (3 long[] arrays)
Step 4 - Duplicate items should be removed in output
I wrote a few things :
package exam1;
import java.time.Duration;
import java.time.Instant;
import java.util.HashSet;
import java.util.Iterator;
import java.util.Random;
/**
*
* #author Furkan
*/
//VM OPTIONS -> -Xincgc -Xmx4g -Xms4g
public final class Exam1 {
private static final int LENGTH = 100000000;
private volatile long[] m_testArr1 = null;
private volatile long[] m_testArr2 = null;
private volatile long[] m_testArr3 = null;
private volatile long[] m_merged = null;
private Random m_r = new Random(System.currentTimeMillis());
public static void main(String[] args) {
Exam1 exam = new Exam1();
Instant start1 = Instant.now();
System.out.println("Fill Started");
exam.Fill();
Instant end1 = Instant.now();
System.out.println("Fill Ended : " + Duration.between(start1, end1));
Instant start2 = Instant.now();
System.out.println("Merge Started");
exam.Merge();
Instant end2 = Instant.now();
System.out.println("Merge Ended : " + Duration.between(start1, end1));
Instant start3 = Instant.now();
System.out.println("DupRemove Started");
exam.DupRemove();
Instant end3 = Instant.now();
System.out.println("DupRemove Ended : " + Duration.between(start1, end1));
}
private void Fill(){
this.m_testArr1 = new long[Exam1.LENGTH];
this.m_testArr2 = new long[Exam1.LENGTH];
this.m_testArr3 = new long[Exam1.LENGTH];
for (int i = 0; i < Exam1.LENGTH; i++) {
this.m_testArr1[i] = this.m_r.nextLong();
this.m_testArr2[i] = this.m_r.nextLong();
this.m_testArr3[i] = this.m_r.nextLong();
}
}
private void Merge(){
this.m_merged = this.TryMerge(this.m_testArr1, this.m_testArr2, this.m_testArr3);
}
private void DupRemove(){
this.m_merged = this.RemoveDuplicates(this.m_merged);
}
public long[] TryMerge(long[] arr1, long[] arr2, long[] arr3){
int aLen = arr1.length;
int bLen = arr2.length;
int cLen = arr3.length;
int len = aLen + bLen + cLen;
//TODO: Use BitSize for RAM optimize. IDK how to use...
//OutOfMemory Exception on this line.
long[] mergedArr = new long[len];
this.m_merged = new long[len];
//long[] mergedArr = (long[]) Array.newInstance(long.class, aLen+bLen+cLen);
System.arraycopy(arr1, 0, mergedArr, 0, aLen);
System.arraycopy(arr2, 0, mergedArr, aLen, bLen);
System.arraycopy(arr3, 0, mergedArr, (aLen + bLen), cLen);
return mergedArr;
}
//!!!NOT WORKING!!!
private long[] RemoveDuplicates(long[] arr){
HashSet<Long> set = new HashSet<Long>();
final int len = arr.length;
for(int i = 0; i < len; i++){
set.add(arr[i]);
}
long[] clean = new long[set.size()];
int i = 0;
for (Iterator<Long> it = set.iterator(); it.hasNext();) {
clean[i++] = it.next();
}
return clean;
}
}
UPDATE
Original Question ;
-Implement an efficient methot to merge 3 sets of very large (Length: 100M+) long[] arrays.
-Input data randomly generated, not sorted, may contain duplicateds
-Duplicate items should be removed in output.
(8 GB RAM i have)
Run Args: -Xincgc -Xmx4g -Xms4g
Exception : Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at test.
Since you have limited space, and assuming you're allowed to modify the 3 random arrays, I'd suggest the following.
For each of the 3 arrays:
Sort the array, e.g. using Arrays.sort().
Eliminate duplicates by compacting non-repeating numbers to the beginning.
E.g. if you have {1,2,2,3,3}, you compact to {1,2,3,?,?} with length 3, where ? means value don't matter.
(optional) Move to array of correct size, and discard original array, to free up space for result array(s).
Create result array of size len1 + len2 + len3.
Merge the 3 arrays to the result, eliminating duplicates between the arrays.
E.g. if you have {1,3,5}, {1,2,3}, you end up with {1,2,3,5,?,?} with length 4.
If needed, copy result to new array of correct size.
If low on memory, release 3 original arrays before doing this to free up space.
Use a Bloom filter to identify possible duplicates, then use a hash set to weed out the false positives from the set of possible duplicates, i.e.
foreach source array element, add it to the Bloom filter; if the element is (possibly) contained in the bloom filter, then add it to a hash set, else add it to the merged array. When all source arrays are processed, check each element of the merged array to see if it is in the hash set, removing duplicates from the hash set. Finally, add all remaining elements of the hash set to the merged array.
Guava has a bloom filter data structure that you can use.
If you don't have enough memory to store all the data you need to change something by analysing the business requirements and the real world situation.
Maybe you should use some built in collection framework as others suggested.
Or if it is not allowed (for whatever reason) you should save the data somewhere else than memory. E.g.
sort the arrays
watch the three array with three moving indexes (i, j, k)
always pick the smallest of arr1[i], arr2[j], arr3[k]
ignore if it is a duplicate and move on
write to a file if it is a new value
and increment the corresponding index
do it until the end of each array
Now you have the sorted duplicate free merged array in a file which you can read if necessary after dropping the originals.
I have this:
long hnds[] = new long[133784560]; // 133 million
Then I quickly fill the array (couple of ms) and then I somehow want to know the number of unique (i.e. distinct) values. Now, I don't even need this realtime, I just need to try out a couple of variations and see how many unique values each gives.
I tried e.g. this:
import org.apache.commons.lang3.ArrayUtils;
....
HashSet<Long> length = new HashSet<Long>(Arrays.asList(ArrayUtils.toObject(hnds)));
System.out.println("size: " + length.size());
and after waiting for half an hour it gives a heap space error (I have Xmx4000m).
I also tried initializing Long[] hnds instead of long[] hnds, but then the initial filling of the array takes forever. Or for example use a Set from the beginning when adding the values, but also then it takes forever. Is there any way to count the distinct values of a long[] array without waiting forever? I'd write it to a file if I have to, just some way.
My best suggestion would be to use a library like fastutil (http://fastutil.di.unimi.it/) and then use the custom unboxed hash set:
import it.unimi.dsi.fastutil.longs.LongOpenHashSet;
System.out.println(new LongOpenHashSet(hnds).size());
(Also, by the way, if you can accept approximate answers, there are much more efficient algorithms you can try; see e.g. this paper for details.)
Just sort it and count.
int sz = 133784560;
Random randy = new Random();
long[] longs = new long[sz];
for(int i = 0; i < sz; i++) { longs[i] = randy.nextInt(10000000); }
Arrays.sort(longs);
long lastSeen = longs[0];
long count = 0;
for(int i = 1; i < sz; i++) {
if(longs[i] != lastSeen) count++;
lastSeen = longs[i];
}
Takes about 15 seconds on my laptop.
if I have a for loop like...
for (int i = 0; i < myArray.length; i++) { ... }
...does myArray.lengthget evaluated every iteration? So would something like...
int len = myArray.length;
for (int i = 0; i < len; i++) { ... }
... be a small performance increase?
regardless myArray.length is just a field so there is nothing to evaluate
Java array has length as public final int so it gets initialized once and when you refer to it there is no code execution like a method call
The public final field length, which contains the number of components of the array. length may be positive or zero.
The first form will probably incur some performance penalty, since evaluating it will require, before the iflt, an aload, an arraylength and an iload; whereas the second is only two iloads.
#ajp rightly mentions that myArray may change; so it is highly unlikely that the compiler will optimize the first form into the second for you (unless, maybe, myArray is final).
However, the JIT, when it kicks in, is probably smart enough so that, if myArray doesn't change, it will turn the first form into the second.
Just in case, anyway, use the second form (this is what I always do, but that's just out of habit). Note that you can always javap the generated class file to see the generated byte code and compare.
By the way, Wikipedia has a very handy page listing all of a JVM's bytecodes. As you may see, quite a lot of them are dedicated to arrays!
Yes, the termination expression gets evaluated every time. So you're right that storing the length once could be a small performance increase. But more importantly, it changes the logic, which could make a difference if myArray gets reassigned.
for (int i = 0; i < myArray.length; i++) {
if (something-something-something) {
myArray = appendToMyArray(myArray, value); // sets myArray to a new, larger array
}
}
Now it makes a big difference whether you store the array length in a variable first.
You wouldn't normally see code like this with an array. But with an arrayList or other collection, whose size could increase (or decrease) in the body of the loop, it makes a big difference whether you compute the size once or every time. This idiom shows up in algorithms where you keep a "To-Do list". For example, here's a partial algorithm to find everyone who's connected directly or indirectly to some person:
ArrayList<Person> listToCheck = new ArrayList<>(KevinBacon);
for (int i = 0; i < listToCheck.size(); i++) {
List<Person> connections = allConnections(listToCheck.get(i));
for (Person p : connections) {
if ([p has not already been checked]) {
listToCheck.add(p); // increases listToCheck.size()!!!
}
}
}
Not really. Both cases are comparing the value at two memory addresses with every iteration, except you are doing unnecessary assigning when you use a len variable. The performance difference is probably very small, and the first line is more readable, so I would use the first way as it is more readable. If you want to be even more readable and efficient, use a for-each loop if you are just going to do a linear iteration through your array. For-each loops look work like this:
int [] myArray = {1,2,3};
for(int i:myArray){
System.out.print(i);
}
will print:
1
2
3
as i is set to each element of the array. The for each loop can be used for many objects, and is a nice feature to learn.
Here is a guide explaining it.
https://www.udemy.com/blog/for-each-loop-java/
I have an array of 6 elements. Is it more efficient (time-wise) to set those elements to null, or to create a new array? I'm going to be using this array hundreds of times.
I would use this method.
System.arraycopy
It is native and pretty efficient.
I would keep an array of nulls and copy it into my array using this method, each time when I want to reset my array. So my advise would be not to create a new array each time, but also not to loop (using Java code) and set the elements to null yourself. Just use this native method which Java provides.
import java.util.Arrays;
public class Test056 {
public static void main(String[] args) {
String[] arrNull = new String[10000];
String[] arrString = new String[10000];
long t1 = System.nanoTime();
for (int i=0; i<10000; i++){
System.arraycopy(arrNull, 0, arrString, 0, arrNull.length);
}
long t2 = System.nanoTime();
System.out.println(t2 - t1);
long t3 = System.nanoTime();
for (int i=0; i<10000; i++){
Arrays.fill(arrString, null);
}
long t4 = System.nanoTime();
System.out.println(t4 - t3);
}
}
Mu - You are not asking the right question.
Do not worry about the efficiency of such trivial operations until it is a known performance bottleneck in your application.
You say you will be nulling/creating 6 references 100s of times. So, you will be creating/nulling/looping < 6000 references. Which is trivial in modern programming.
There are likely much better places where you should be spending your development time.
Creating a new array should be more efficient time-wise because it would only allocate empty references.
Setting the elements to null would imply walking through the entire array to set references to null (which is default behavior on array creation) which is more time consuming (even if for an array of 6 elements it's totally negligible).
EDIT : time-wise is bold because memory-wise is may not be your best option. Since you'll be creating new references, if you instantiate new object, make sure the object of your previous array are garbage collected properly (again, with only 6 elements it must be gigantic objects to see any bad performance impact).
The following bit of code demonstrates that it's faster to assign each value to null.
Note that this is valid for arrays with 6 elements. Eventually it may be faster to create a new array.
Object[] array = new Object[6];
long start = System.nanoTime();
for(int i = 0; i < Integer.MAX_VALUE; i++){
array[0] = null;
array[1] = null;
array[2] = null;
array[3] = null;
array[4] = null;
array[5] = null;
}
System.out.println("elapsed nanoseconds: " + (System.nanoTime() - start));
int length = array.length;
start = System.nanoTime();
for(int i = 0; i < Integer.MAX_VALUE; i++){
array = new Object[6];
}
System.out.println("elapsed nanoseconds: " + (System.nanoTime() - start));
length = array.length;
and the output:
elapsed nanoseconds: 264095957
elapsed nanoseconds: 17885568039
The best and secure way to create an array is the next one:
List<Object> myList = new ArrayList<Object>(0);
So can can regenerate by this way the array each time you want. You don't need to set to null. By this way, the array is destroyed and regenerated in memory.
You can read more info in Oracle Java documentation: