We Keep Coding

Set vs List when need both unique elements and access by index

I need to keep a unique list of elements seen and I also need to pick random one from them from time to time. There are two simple ways for me to do this.
Keep elements seen in a Set - that gives me uniqueness of elements. When there is a need to pick random one, do the following:
elementsSeen.toArray()[random.nextInt(elementsSeen.size())]
Keep elements seen in a List - this way no need to convert to array as there is the get() function for when I need to ask for a random one. But here I would need to do this when adding.
if (elementsSeen.indexOf(element)==-1) {elementsSeen.add(element);}
So my question is which way would be more efficient? Is converting to array more consuming or is indexOf worse? What if attempting to add an element is done 10 or 100 or 1000 times more often?
I am interested in how to combine functionality of a list (access by index) with that of a set (unique adding) in the most performance effective way.

If using more memory is not a problem then you can get the best of both by using both list and set inside a wrapper:
public class MyContainer<T> {
private final Set<T> set = new HashSet<>();
private final List<T> list = new ArrayList<>();
public void add(T e) {
if (set.add(e)) {
list.add(e);
}
}
public T getRandomElement() {
return list.get(ThreadLocalRandom.current().nextInt(list.size()));
}
// other methods as needed ...
}

HashSet and TreeSet both extend AbstractCollection, which includes the toArray() implementation as shown below:
public Object[] toArray() {
// Estimate size of array; be prepared to see more or fewer elements
Object[] r = new Object[size()];
Iterator<E> it = iterator();
for (int i = 0; i < r.length; i++) {
if (! it.hasNext()) // fewer elements than expected
return Arrays.copyOf(r, i);
r[i] = it.next();
}
return it.hasNext() ? finishToArray(r, it) : r;
}
As you can see, its responsible for allocating the space for an array, as well as creating an Iterator object for copying. So, for a Set, adding is O(1), but retrieving a random element will be O(N) because of the element copy operation.
A List, on the other hand, allows you quick access to a specific index in the backing array, but doesn't guarantee uniqueness. You would have to re-implement the add, remove and associated methods to guarantee uniqueness on insert. Adding a unique element will be O(N), but retrieval will be O(1).
So, it really depends on which area is your potential high usage point. Are the add/remove methods going to be heavily used, with random access used sparingly? Or is this going to be a container for which retrieval is most important, since few elements will be added or removed over the lifetime of the program?
If the former, I'd suggest using the Set with toArray(). If the latter, it may be beneficial for you to implement a unique List to take advantage to the fast retrieval. The significant downside is add contains many edge cases for which the standard Java library takes great care to work with in an efficient manner. Will your implementation be up to the same standards?

Write some test code and put in some realistic values for your use case. Neither of the methods are so complex that it's not worth the effort, if performance is a real issue for you.
I tried that quickly, based on the exact two methods you described, and it appears that the Set implementation will be quicker if you are adding considerably more than you are retrieving, due to the slowness of the indexOf method. But I really recommend that you do the tests yourself - you're the only person who knows what the details are likely to be.
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Random;
import java.util.Set;
public class SetVsListTest<E> {
private static Random random = new Random();
private Set<E> elementSet;
private List<E> elementList;
public SetVsListTest() {
elementSet = new HashSet<>();
elementList = new ArrayList<>();
}
private void listAdd(E element) {
if (elementList.indexOf(element) == -1) {
elementList.add(element);
}
}
private void setAdd(E element) {
elementSet.add(element);
}
private E listGetRandom() {
return elementList.get(random.nextInt(elementList.size()));
}
#SuppressWarnings("unchecked")
private E setGetRandom() {
return (E) elementSet.toArray()[random.nextInt(elementSet.size())];
}
public static void main(String[] args) {
SetVsListTest<Integer> test;
List<Integer> testData = new ArrayList<>();
int testDataSize = 100_000;
int[] addToRetrieveRatios = new int[] { 10, 100, 1000, 10000 };
for (int i = 0; i < testDataSize; i++) {
/*
* Add 1/5 of the total possible number of elements so that we will
* have (on average) 5 duplicates of each number. Adjust this to
* whatever is most realistic
*/
testData.add(random.nextInt(testDataSize / 5));
}
for (int addToRetrieveRatio : addToRetrieveRatios) {
/*
* Test the list method
*/
test = new SetVsListTest<>();
long t1 = System.nanoTime();
for(int i=0;i<testDataSize; i++) {
// Use == 1 here because we don't want to get from an empty collection
if(i%addToRetrieveRatio == 1) {
test.listGetRandom();
} else {
test.listAdd(testData.get(i));
}
}
long t2 = System.nanoTime();
System.out.println(((t2-t1)/1000000L)+" ms for list method with add/retrieve ratio "+addToRetrieveRatio);
/*
* Test the set method
*/
test = new SetVsListTest<>();
t1 = System.nanoTime();
for(int i=0;i<testDataSize; i++) {
// Use == 1 here because we don't want to get from an empty collection
if(i%addToRetrieveRatio == 1) {
test.setGetRandom();
} else {
test.setAdd(testData.get(i));
}
}
t2 = System.nanoTime();
System.out.println(((t2-t1)/1000000L)+" ms for set method with add/retrieve ratio "+addToRetrieveRatio);
}
}
}
Output on my machine was:
819 ms for list method with add/retrieve ratio 10
1204 ms for set method with add/retrieve ratio 10
1547 ms for list method with add/retrieve ratio 100
133 ms for set method with add/retrieve ratio 100
1571 ms for list method with add/retrieve ratio 1000
23 ms for set method with add/retrieve ratio 1000
1542 ms for list method with add/retrieve ratio 10000
5 ms for set method with add/retrieve ratio 10000

You could extend HashSet and track the changes to it, maintaining a current array of all entries.
Here I keep a copy of the array and adjust it every time the set changes. For a more robust (but more costly) solution you could use toArray in your pick method.
class PickableSet<T> extends HashSet<T> {
private T[] asArray = (T[]) this.toArray();
private void dirty() {
asArray = (T[]) this.toArray();
}
public T pick(int which) {
return asArray[which];
}
#Override
public boolean add(T t) {
boolean added = super.add(t);
dirty();
return added;
}
#Override
public boolean remove(Object o) {
boolean removed = super.remove(o);
dirty();
return removed;
}
}
Note that this will not recognise changes to the set if removed by an Iterator - you will need to handle that some other way.

So my question is which way would be more efficient?
Quite a difficult question to answer depending on what one does more, insert or select at random?
We need to look at the Big O for each of the operations. In this case (best cases):
Set: Insert O(1)
Set: toArray O(n) (I'd assume)
Array: Access O(1)
vs
List: Contains O(n)
List: Insert O(1)
List: Access O(1)
So:
Set: Insert: O(1), Access O(n)
List: Insert: O(n), Access O(1)
So in the best case they are much of a muchness with Set winning if you insert more than you select, and List if the reverse is true.
Now the evil answer - Select one (the one that best represents the problem (so Set IMO)), wrap it well and run with it. If it is too slow then deal with it later, and when you do deal with it, look at the problem space. Does your data change often? No, cache the array.

It depends what you value more.
List implementations in Java normally makes use of an array or a linked list. That means inserting and searching for an index is fast, but searching for a specific element will require looping thought the list and comparing each element until the element is found.
Set implementations in Java mainly makes use of an array, the hashCode method and the equals method. So a set is more taxing when you want to insert, but trumps list when it comes to looking for an element. As a set doesn't guarantee the order of the elements in the structure, you will not be able to get an element by index. You can use an ordered set, but this brings with it latency on the insert due to the sort.
If you are going to be working with indexes directly, then you may have to use a List because the order that element will be placed into Set.toArray() changes as you add elements to the Set.
Hope this helps :)

memory and time efficient way to sort random incoming data

I need to sort data coming from different lists of random values (values can be repeated) into a list of unique values in a memory and time efficient way (there are hundreds of lists that can have up to thousands of records each). Right now I have 2 methods
Method 1- Sort as data comes in:
public List<ClassB> ListSorter1(List<ClassA> listA){
List<ClassB> data = new ArrayList<>();
for (ClassA a : listA) {
int idx = Collections.binarySearch(data, a.getValue());
if (idx < 0) {
int ip = -(idx + 1);
data.add(ip, a.getValue());
}
}
}
Method 2 - get all the unique data and then sort:
public List<ClassB> ListSorter2 (List<ClassA> listA){
List<ClassB> data = new ArrayList<>();
for (ClassA a : listA) {
if (!data.contains(a.getValue())) {
data.add(a.getValue());
}
}
Collections.sort(data);
}
The problem I'm having is that method 2 performs better (about 20% faster than method 1 and roughly the same memory usage) when <ClassB> is simple data (Integer), but as soon as I change to a more complex class, the time required to sort the list skyrockets, taking up to 10 times more than method 1 (and still about the same memory usage), both using the same comparator function.
Why this difference in performance?
Is there a more efficient way to do this?

First of all it is strange that Method 1 is 20% slower than Method 2, but I assume that it is tested on a very small collection.
Cause for a big slowdown in Method 2 is for two reasons:
When you iterating data is not sorted, so
contains method has to go through whole list in order to find element - which is O(n). contains has O(n) complexity no meter if data is sorted, since it iterates over the whole collection.
So, for Method 2 it is O(n^2) complexity
For method 1, you are managing ordered list, and you are using binarySearch which is O(ln(n)).
So, method 1 has a complexity of O(n*ln(n))

Find the highest N numbers in an infinite list

I was asked this question in a recent Java interview.
Given a List containing millions of items, maintain a list of the highest n items. Sorting the list in descending order then taking the first n items is definitely not efficient due to the list size.
Below is what I did, I'd appreciate if anyone could provide a more efficient or elegant solution as I believe this could also be solved using a PriorityQueue:
public TreeSet<Integer> findTopNNumbersInLargeList(final List<Integer> largeNumbersList,
final int highestValCount) {
TreeSet<Integer> highestNNumbers = new TreeSet<Integer>();
for (int number : largeNumbersList) {
if (highestNNumbers.size() < highestValCount) {
highestNNumbers.add(number);
} else {
for (int i : highestNNumbers) {
if (i < number) {
highestNNumbers.remove(i);
highestNNumbers.add(number);
break;
}
}
}
}
return highestNNumbers;
}

The for loop at the bottom is unnecessary, because you can tell right away if the number should be kept or not.
TreeSet lets you find the smallest element in O(log N)*. Compare that smallest element to number. If the number is greater, add it to the set, and remove the smallest element. Otherwise, keep walking to the next element of largeNumbersList.
The worst case is when the original list is sorted in ascending order, because you would have to replace an element in the TreeSet at each step. In this case the algorithm would take O(K log N), where K is the number of items in the original list, an improvement of logNK over the solution of sorting the array.
Note: If your list consists of Integers, you could use a linear sorting algorithm that is not based on comparisons to get the overall asymptotic complexity to O(K). This does not mean that the linear solution would be necessarily faster than the original for any fixed number of elements.
* You can maintain the value of the smallest element as you go to make it O(1).

You don't need nested loops, just keep inserting and remove the smallest number when the set is too large:
public Set<Integer> findTopNNumbersInLargeList(final List<Integer> largeNumbersList,
final int highestValCount) {
TreeSet<Integer> highestNNumbers = new TreeSet<Integer>();
for (int number : largeNumbersList) {
highestNNumbers.add(number);
if (highestNNumbers.size() > highestValCount) {
highestNNumbers.pollFirst();
}
}
return highestNNumbers;
}
The same code should work with a PriorityQueue, too. The runtime should be O(n log highestValCount) in any case.
P.S. As pointed out in the other answer, you can optimize this some more (at the cost of readability) by keeping track of the lowest number, avoiding unnecessary inserts.

It's possible to support amortized O(1) processing of new elements and O(n) querying of the current top elements as follows:
Maintain a buffer of size 2n, and whenever you see a new element, add it to the buffer. When the buffer gets full, use quick select or another linear median finding algorithm to select the current top n elements, and discard the rest. This is an O(n) operation, but you only need to perform it every n elements, which balances out to O(1) amortized time.
This is the algorithm Guava uses for Ordering.leastOf, which extracts the top n elements from an Iterator or Iterable. It is fast enough in practice to be quite competitive with a PriorityQueue based approach, and it is much more resistant to worst case input.

I would start by saying that your question, as stated, is impossible. There is no way to find the highest n items in a List without fully traversing it. And there is no way to fully traverse an infinite List.
That said, the text of your question differs from the title. There is a massive difference between very large and infinite. Please bear that in mind.
To answer the feasible question, I would begin by implementing a buffer class to encapsulate the behaviour of keeping the top N, lets call it TopNBuffer:
class TopNBuffer<T extends Comparable<T>> {
private final NavigableSet<T> backingSet = new TreeSet<>();
private final int limit;
public TopNBuffer(int limit) {
this.limit = limit;
}
public void add(final T t) {
if (backingSet.add(t) && backingSet.size() > limit) {
backingSet.pollFirst();
}
}
public SortedSet<T> highest() {
return Collections.unmodifiableSortedSet(backingSet);
}
}
All we do here is to, on add, if the number is not unique, and adding the number makes the Set exceeds its limit, then we simply remove the lowest element from the Set.
The method highest gives an unmodifiable view of the current highest elements. So, in Java 8 syntax, all you need to do is:
final TopNBuffer<Integer> topN = new TopNBuffer<>(n);
largeNumbersList.foreach(topN::add);
final Set<Integer> highestN = topN.highest();
I think in an interview environment, its not enough to simply whack lots of code into a method. Demonstrating an understanding of OO programming and separation of concerns is also important.

Performance (runtime) of pulling arbitrary element from HashSet

I am using HashSets in an implementation to have fast adding, removing and element testing (amortized constant time).
However, I'd also like a method to obtain an arbitraty element from that set. The only way I am aware of is
Object arbitraryElement = set.iterator.next();
My question is - how fast (asymptotically speaking) is this? Does this work in (not amortized) constant time in the size of the set, or does the iterator().next() method do some operations that are slower? I ask because I seem to lose a log-factor in my implementation as experiments show, and this is one of the few lines affected.
Thank you very much!

HashSet.iterator().next() linearly scans the table to find the next contained item.
For the default load factor of .75, you would have three full slots for every empty one.
There is, of course, no guarantee what the distribution of the objects in the backing array will be & the set will never actually be that full so scans will take longer.
I think you'd get amortized constant time.
Edit: The iterator does not create a deep copy of anything in the set. It only references the array in the HashSet. Your example creates a few objects, but nothing more & no big copies.

I wouldn't expect this to be a logarithmic factor, on average, but it might be slow in some rare cases. If you care about this, use LinkedHashSet, which will guarantee constant time.

I would maintain an ArrayList of your keys, and when you need a random object, just generate an index, grab the key, and pull it out of the set. O(1) baby...

Getting the first element out of a HashSet using the iterator is pretty fast: I think it's amortised O(1) in most cases. This assumes the HashSet is reasonably well-populated for it's given capacity - if the capacity is very large compared to the number of elements then it will be more like O(capacity/n) which is the average number of buckets the iterator needs to scan before finding a value.
Even scanning an entire HashSet with an iterator is only O(n+capacity) which is effectively O(n) if your capacity is appropriately scaled. So it's still not particularly expensive (unless your HashSet is very large)
If you want better than that , you'll need a different data structure.
If you really need the fast access of arbitrary elements by index then I'd personally just put the objects in an ArrayList which will give you very fast O(1) access by index. You can then generate the index as a random number if you want to select an arbitrary element with equal probability.
Alternatively, if you want to get an arbitrary element but don't care about indexed access then a LinkedHashSet may be a good alternative.

This is from the JDK 7 JavaDoc for HashSet:
Iterating over this set requires time proportional to the sum of the HashSet instance's size (the number of elements) plus the "capacity" of the backing HashMap instance (the number of buckets). Thus, it's very important not to set the initial capacity too high (or the load factor too low) if iteration performance is important.
I looked at the JDK 7 implementation of HashSet and LinkedHashSet. For the former, the next operation is a linked-list traversal within a has bucket, and between buckets an array traversal, where the size of the array is given by capacity(). The latter is strictly a linked list traversal.

If you need an arbitrary element in the probabilistic sense, you could use the following approach.
class MySet<A> {
ArrayList<A> contents = new ArrayList();
HashMap<A,Integer> indices = new HashMap<A,Integer>();
Random R = new Random();
//selects random element in constant O(1) time
A randomKey() {
return contents.get(R.nextInt(contents.size()));
}
//adds new element in constant O(1) time
void add(A a) {
indices.put(a,contents.size());
contents.add(a);
}
//removes element in constant O(1) time
void remove(A a) {
int index = indices.get(a);
contents.set(index,contents.get(contents.size()-1));
contents.remove(contents.size()-1);
indices.set(contents.get(contents.size()-1),index);
indices.remove(a);
}
//all other operations (contains(), ...) are those from indices.keySet()
}

If you are repeatedly choosing an arbitrary set element using an iterator and often removing that element, this can lead to a situation where the internal representation becomes unbalanced and finding the first element degrades to linear time complexity.
This is actually a pretty common occurrence when implementing algorithms involving graph traversal.
Use a LinkedHashSet to avoid this problem.
Demonstration:
import java.util.HashSet;
import java.util.Iterator;
import java.util.LinkedHashSet;
import java.util.Random;
import java.util.Set;
import java.util.function.Supplier;
import java.util.stream.Collectors;
public class SetPeek {
private static final Random rng = new Random();
private static <T> T peek(final Iterable<T> i) {
return i.iterator().next();
}
private static long testPeek(Set<Integer> items) {
final long t0 = System.currentTimeMillis();
for (int i = 0; i < 100000; i++) {
peek(items);
}
final long t1 = System.currentTimeMillis();
return t1 - t0;
}
private static <S extends Set<Integer>> S createSet(Supplier<S> factory) {
final S set = new Random().ints(100000).boxed()
.collect(Collectors.toCollection(factory));
// Remove first half of elements according to internal iteration
// order. With the default load factor of 0.75 this will not trigger
// a rebalancing.
final Iterator<Integer> it = set.iterator();
for (int k = 0; k < 50000; k++) {
it.next();
it.remove();
}
return set;
}
public static void main(String[] args) {
final long hs = testPeek(createSet(HashSet::new));
System.err.println("HashSet: " + hs + " ms");
final long lhs = testPeek(createSet(LinkedHashSet::new));
System.err.println("LinkedHashSet: " + lhs + " ms");
}
}
Results:
HashSet: 6893 ms
LinkedHashSet: 8 ms

Count the occurrences of items in ArrayList

I have a java.util.ArrayList<Item> and an Item object.
Now, I want to obtain the number of times the Item is stored in the arraylist.
I know that I can do arrayList.contains() check but it returns true, irrespective of whether it contains one or more Items.
Q1. How can I find the number of time the Item is stored in the list?
Q2. Also, If the list contains more than one Item, then how can I determine the index of other Items because arrayList.indexOf(item) returns the index of only first Item every time?

You can use Collections class:
public static int frequency(Collection<?> c, Object o)
Returns the number of elements in the specified collection equal to the specified object. More formally, returns the number of elements e in the collection such that (o == null ? e == null : o.equals(e)).
If you need to count occurencies of a long list many times I suggest you to use an HashMap to store the counters and update them while you insert new items to the list. This would avoid calculating any kind of counters.. but of course you won't have indices.
HashMap<Item, Integer> counters = new HashMap<Item, Integer>(5000);
ArrayList<Item> items = new ArrayList<Item>(5000);
void insert(Item newEl)
{
if (counters.contains(newEl))
counters.put(newEl, counters.get(newEl)+1);
else
counters.put(newEl, 1);
items.add(newEl);
}
A final hint: you can use other collections framework (like Apache Collections) and use a Bag datastructure that is described as
Defines a collection that counts the number of times an object appears in the collection.
So exactly what you need..

This is easy to do by hand.
public int countNumberEqual(ArrayList<Item> itemList, Item itemToCheck) {
int count = 0;
for (Item i : itemList) {
if (i.equals(itemToCheck)) {
count++;
}
}
return count;
}
Keep in mind that if you don't override equals in your Item class, this method will use object identity (as this is the implementation of Object.equals()).
Edit: Regarding your second question (please try to limit posts to one question apiece), you can do this by hand as well.
public List<Integer> indices(ArrayList<Item> items, Item itemToCheck) {
ArrayList<Integer> ret = new ArrayList<Integer>();
for (int i = 0; i < items.size(); i++) {
if (items.get(i).equals(itemToCheck)) {
ret.add(i);
}
}
return ret;
}

As the other respondents have already said, if you're firmly committed to storing your items in an unordered ArrayList, then counting items will take O(n) time, where n is the number of items in the list. Here at SO, we give advice but we don't do magic!
As I just hinted, if the list gets searched a lot more than it's modified, it might make sense to keep it sorted. If your list is sorted then you can find your item in O(log n) time, which is a lot quicker; and if you have a hashcode implementation that goes well with your equals, all the identical items will be right next to each other.
Another possibility would be to create and maintain two data structures in parallel. You could use a HashMap containing your items as keys and their count as values. You'd be obligated to update this second structure any time your list changes, but item count lookups would be o(1).

I could be wrong, but it seems to me like the data structure you actually want might be a Multiset (from google-collections/guava) rather than a List. It allows multiples, unlike Set, but doesn't actually care about the order. Given that, it has a int count(Object element) method that does exactly what you want. And since it isn't a list and has implementations backed by a HashMap, getting the count is considerably more efficient.

Thanks for your all nice suggestion. But this below code is really very useful as we dont have any search method with List that can give number of occurance.
void insert(Item newEl)
{
if (counters.contains(newEl))
counters.put(newEl, counters.get(newEl)+1);
else
counters.put(newEl, 1);
items.add(newEl);
}
Thanks to Jack. Good posting.
Thanks,
Binod Suman
http://binodsuman.blogspot.com

I know this is an old post, but since I did not see a hash map solution, I decided to add a pseudo code on hash-map for anyone that needs it in the future. Assuming arraylist and Float data types.
Map<Float,Float> hm = new HashMap<>();
for(float k : Arralistentry) {
Float j = hm.get(k);
hm.put(k,(j==null ? 1 : j+1));
}
for(Map.Entry<Float, Float> value : hm.entrySet()) {
System.out.println("\n" +value.getKey()+" occurs : "+value.getValue()+" times");
}