We Keep Coding

Large Map freezes my program

I have large Map where I store some objects. The Map is large: it has around 200k objects. When I try to run some methods, that require to read map values, the program freezes. When I debug it, it seems that my IDE is 'collecting data' (picture). It has never completed the task.
I have 16GB RAM.
What can I do to speed this up?

I get performance issues around 61 million elements.
import java.util.*;
public class BreakingMaps{
public static void main(String[] args){
int count = Integer.MAX_VALUE>>5;
System.out.println(count + " objects tested");
HashMap<Long, String> set = new HashMap<>(count);
for(long i = 0; i<count; i++){
Long l = i;
set.put(l, l.toString());
}
Random r = new Random();
for(int i = 0; i<1000; i++){
long k = r.nextInt()%count;
k = k<0?-k:k;
System.out.println(set.get(k));
}
}
}
I run the program with java -Xms12G -Xmx13G BreakingMaps
I suspect your problem is not the map, but circumstances surrounding the map. If I write the same program, but use a class with hashcode colisions then the program cannot handle 200K elements.
static class Key{
final long l;
public Key(long l){
this.l = l;
}
#Override
public int hashCode(){
return 1;
}
#Override
public boolean equals(Object o){
if(o!=null && o instanceof Key){
return ((Key)o).l==l;
}
return false;
}
}

Look at this - as the solution you can increase the heap size for your app:
java -Xmx6g myprogram.
But it's not very good. I'd suggest you to try to rework your data processing approach. Maybe you can apply some filtering before fetching the data to decrease the data size or implement some calculation on database level.

What is the most efficient way to count the intersections between two sets (Java)?

Question Background
I am comparing two (at a time, actually many) text files, and I want to determine how similar they are. To do so, I have created small, overlapping groups of text from each file. I now want to determine the number of those groups from one file which are also from the other file.
I would prefer to use only Java 8 with no external libraries.
Attempts
These are my two fastest methods. The first contains a bunch of logic which allows it to stop if meeting the threshold is not possible with the remaining elements (this saves a bit of time in total, but of course executing the extra logic also takes time). The second is slower. It does not have those optimizations, actually determines the intersection rather than merely counting it, and uses a stream, which is quite new to me.
I have an integer threshold and dblThreshold (the same value cast to a double), which are the minimum percentage of the smaller file which must be shared to be of interest. Also, from my limited testing, it seems that writing all the logic for either set being larger is faster than calling the method again with reversed arguments.
public int numberShared(Set<String> sOne, Set<String> sTwo) {
int numFound = 0;
if (sOne.size() > sTwo.size()) {
int smallSize = sTwo.size();
int left = smallSize;
for (String item: sTwo) {
if (numFound < threshold && ((double)numFound + left < (dblThreshold) * smallSize)) {
break;
}
if (sOne.contains(item)) {
numFound++;
}
left--;
}
} else {
int smallSize = sOne.size();
int left = smallSize;
for (String item: sOne) {
if (numFound < threshold && ((double)numFound + left < (dblThreshold) * smallSize)) {
break;
}
if (sTwo.contains(item)) {
numFound++;
}
left--;
}
}
return numFound;
}
Second method:
public int numberShared(Set<String> sOne, Set<String> sTwo) {
if (sOne.size() < sTwo.size()) {
long numFound = sOne.parallelStream()
.filter(segment -> sTwo.contains(segment))
.collect(Collectors.counting());
return (int)numFound;
} else {
long numFound = sTwo.parallelStream()
.filter(segment -> sOne.contains(segment))
.collect(Collectors.counting());
return (int)numFound;
}
}
Any suggestions for improving upon these methods, or novel ideas and approaches to the problem are much appreciated!
Edit: I just realized that the first part of my threshold check (which seeks to eliminate, in some cases, the need for the second check with doubles) is incorrect. I will revise it as soon as possible.

If I understand you correctly, you have already determined which methods are fastest, but aren't sure how to implement your threshold-check when using Java 8 streams. Here's one way you could do that - though please note that it's hard for me to do much testing without having proper data and knowing what thresholds you're interested in, so take this simplified test case with a grain of salt (and adjust as necessary).
public class Sets {
private static final int NOT_ENOUGH_MATCHES = -1;
private static final String[] arrayOne = { "1", "2", "4", "9" };
private static final String[] arrayTwo = { "2", "3", "5", "7", "9" };
private static final Set<String> setOne = new HashSet<>();
private static final Set<String> setTwo = new HashSet<>();
public static void main(String[] ignoredArguments) {
setOne.addAll(Arrays.asList(arrayOne));
setTwo.addAll(Arrays.asList(arrayTwo));
boolean isFirstSmaller = setOne.size() < setTwo.size();
System.out.println("Number shared: " + (isFirstSmaller ?
numberShared(setOne, setTwo) : numberShared(setTwo, setOne)));
}
private static long numberShared(Set<String> smallerSet, Set<String> largerSet) {
SimpleBag bag = new SimpleBag(3, 0.5d, largerSet, smallerSet.size());
try {
smallerSet.forEach(eachItem -> bag.add(eachItem));
return bag.duplicateCount;
} catch (IllegalStateException exception) {
return NOT_ENOUGH_MATCHES;
}
}
public static class SimpleBag {
private Map<String, Boolean> items;
private int threshold;
private double fraction;
protected int duplicateCount = 0;
private int smallerSize;
private int numberLeft;
public SimpleBag(int aThreshold, double aFraction, Set<String> someStrings,
int otherSetSize) {
threshold = aThreshold;
fraction = aFraction;
items = new HashMap<>();
someStrings.forEach(eachString -> items.put(eachString, false));
smallerSize = otherSetSize;
numberLeft = otherSetSize;
}
public void add(String aString) {
Boolean value = items.get(aString);
boolean alreadyExists = value != null;
if (alreadyExists) {
duplicateCount++;
}
items.put(aString, alreadyExists);
numberLeft--;
if (cannotMeetThreshold()) {
throw new IllegalStateException("Can't meet threshold; stopping at "
+ duplicateCount + " duplicates");
}
}
public boolean cannotMeetThreshold() {
return duplicateCount < threshold
&& (duplicateCount + numberLeft < fraction * smallerSize);
}
}
}
So I've made a simplified "Bag-like" implementation that starts with the contents of the larger set mapped as keys to false values (since we know there's only one of each). Then we iterate over the smaller set, adding each item to the bag, and, if it's a duplicate, switching the value to true and keeping track of the duplicate count (I initially did a .count() at the end of .stream().allMatch(), but this'll suffice for your special case). After adding each item, we check whether we can't meet the threshold, in which case we throw an exception (arguably not the prettiest way to exit the .forEach(), but in this case it is an illegal state of sorts). Finally, we return the duplicate count, or -1 if we encountered the exception. In my little test, change 0.5d to 0.51d to see the difference.

What's the fastest way to initialize a large list of integers?

I need to pre-populate a List with a large number of integer values.
Is there are faster way to do this other than iteration?
Current Code:
class VlanManager {
Queue<Integer> queue = Lists.newLinkedList();
public VlanManager(){
for (int i = 1; i < 4094; i++) {
queue.add(i);
}
}
This code is in the constructor of a class that is created pretty frequently so I'd like this to be as efficient (read:performance not lines of code) as possible

4094 isnt to many items to loop but if it is getting called very frequently you might look at doing something with a static variable.
private static Integer[] theList;
static {
theList = new Integer[4094];
for (int i = 1; i < 4094; i++) {
theList[i-1] = i;
}
}
then make that list a List
Queue<Integer> intQue = new LinkedList(Arrays.asList(theList));
There is a danger of using this method if you have a list of mutable objects. Heres an example of what can happen. Integers are immutable so this doesnt actually apply to your question as it stands
class MyMutableObject {
public int theValue;
}
class Test {
private static MyMutableObject[] theList;
static {
theList = new MyMutableObject[4094];
for (int i = 1; i <= 4094; i++) {
theList[i-1] = new MyMutableObject();
theList[i-1].theValue = i;
}
}
public static void main(String [] args) {
Queue<MyMutableObject> que = new LinkedList(Arrays.asList(theList));
System.out.println(que.peek().theValue); // 1
// your actually modifing the same object as the one in your static list
que.peek().theValue = -100;
Queue<MyMutableObject> que2 = new LinkedList(Arrays.asList(theList));
System.out.println(que2.peek().theValue); // -100
}
}
#Bohemian Has some good points on using a static List instead of an array, while the performance gains are very small they are none the less performance gains. Also because the 'array' is actually only ever being used as a List not an array it should be declared as such.
private static List<Integer> theList;
static {
theList = new ArrayList(4094);
for (Integer i = 0; i < 4094; i++) {
theList.add(i+1);
}
}

The fastest way would be to create a reference list (initialized using an instance block - neatly wrapping it all up in one statement):
private static final List<Integer> LIST = new ArrayList<Integer>(4094) {{
for (int i = 1; i < 4094; i++)
LIST.add(i);
}};
Then in your constructor, initialize the queue using the copy constructor:
Queue<Integer> queue;
public VlanManager(){
queue = new LinkedList<Integer>(LIST);
}
You will not write a faster implementation than what's in the JDK.

I realize this question has already been answered. But I think one important answer is missing: The fastest way to initialize a LinkedList with the values 0..4093 is .. DON'T DO IT AT ALL. Especially if speed is an issue.
What you basically are doing is creating a structure consisting of 4093 Node elements each consiting of two pointers to prev/next element and one pointer to an Integer object. Each of this Nodes must be created (and free). In addition nearly each contained Integer must be created (and freed). 'Nearly' because Java uses a cache for Integer but normally (you can change this with system properties) in the range of -127..127.
This is a lot to do in order to get a simple list of integer and if used intensively gives the GC a lot to do afterwards.
That being said there are numerous possible ways of doing this in a more efficient way. But they depend on what your concrete usage pattern is. Just to name a few:
Use an Array: boolean [] inUse' and set the taken vlan-id totrue` if it's taken
Even better use a BitSet instead of the array
Don't store which vlan is free, but which vlan is taken. I think they tend to be free and so there are much more free as there are taken ones. (this means much less to keep track of).
If you insist on using a LinkedList don't initialize it with your class but have it already initialized. This depends on how much of them you would need. You could keep a pool of them. Or perhaps your codes allows reusage of old lists. (yes, you could sort them after usage.)
Surely there are more...
All of this methods require you to build your own 'Queue' interface. But perhaps this has not to be as rich as Java's. And it really isn't that difficult. If you really use this intensively you could reach perfomance improvement factor 10x-1000x++.
A possible implementation using BitSet with an instantiation cost of nearly nothing could be:
import java.util.BitSet;
import org.testng.annotations.Test;
public class BitSetQueue {
// Represents the values 0..size-1
private final BitSet bitset;
private final int size;
private int current = 0;
private int taken = 0;
public BitSetQueue( int size ){
this.bitset = new BitSet( size );
this.size = size;
this.current = size-1;
}
public int poll(){
// prevent endless loop
if( taken == size ) return -1;
// seek for next free value.
// can be changed according to policy
while( true ){
current = (current+1)%size;
if( ! bitset.get( current ) ){
bitset.set( current );
taken++;
return current;
}
}
}
public boolean free( int num ){
if( bitset.get( num ) ){
bitset.clear( num );
taken--;
return true;
}
return false;
}
#Test
public static void usage(){
BitSetQueue q = new BitSetQueue( 4094 );
for( int i = 0; i < 4094; i++ ){
assertEquals( q.poll(), i );
}
assertEquals( q.poll(), -1 ); // No more available
assertTrue( q.free( 20 ) );
assertTrue( q.free( 51 ) );
assertEquals( q.poll(), 20 );
assertEquals( q.poll(), 51 );
}
}

Having trouble understanding how to maintain state using classes

I'm new to using OOP, I typically just put all my code in a single class and use methods. But I want to maintain state information and think classes are the best fit but I'm having trouble wrapping my head around it.
Say I have a list of items and I want to stop when the total sum of all previous items in the list equals X(in this case 10 so it takes item 1 + 2, then 2+3.etc..until it hits the threshold 10), I can use a method to calculate it but it involves me doing the entire process all over again when all I really need to do is increment by the last item and then see if my data exceeds the threshold. Here's my code so far but I know its not good because although it works its really just using the class as an independent method and recalculating on every loop. My goal is to,using this structure, reduce loops if not necessary to check thresholds.
Any suggestions?
Code:
public class LearningClassesCounter {
public static void main(String[] args) {
int[] list = new int[]{1,2,3,4,5,6,7,8,9,10};
int[] data_list = new int[list.length];
for (int current_location = 0; current_location<list.length;current_location++) {
//can only put commands in here. Nothing above.
Counter checker = new Counter(data_list);
System.out.println(checker.check_data(current_location));
for (int i =0; i<100; i++){
if (checker.check_data(current_location) == false) {
break;
}
data_list[current_location] = (list[current_location]+1); //this is just a random function, it could be any math function I just put it in here to show that some work is being done.
}
}
//its done now lets print the results
for (Integer item : data_list) {
System.out.println(item);
}
}
}
class Counter {
private int[] data_list;
private int total_so_far;
// create a new counter with the given parameters
public Counter(int[] data_list) {
this.data_list = data_list;
this.total_so_far = 0;
}
public boolean check_data(int current_location) {
// TODO Auto-generated method stub
int total_so_far = 0;
//System.out.println(total_so_far);
for (int item : data_list) {
total_so_far = item + total_so_far;
if (total_so_far >= 10) {
break;
}
}
if (total_so_far>=10) {
return false;
} else {
return true;
}
}
}
I don't need anyone to fix my code or anything(I want to do it myself, the code is just to give an idea of what I'm doing). I'm more interested in the flaw in my logic and maybe a way for me to better think about designing classes so I can apply them to my own situations better.

So the solution is that you do not update the data_list directly. Instead have a setter method in the Counter class that takes the index and value to update. It updates the value in the array and also updates a count value.
Something like this:
class Counter{
private final int[] list;
private count = 0;
private final maxCount = 10;
public Counter(int[] list){
this.list = list;
}
public boolean updateValueAndCheckPastMax(int index, int value){
list[index] = value;
count += value;
return count >= maxCount;
}
}

You are way over thinking this, and a counter class is not really necessary in this case.
I'm also interested as to why you'd be doing this line:
data_list[current_location] = (list[current_location]+1);
Do you want your data_list to be the same as list, but each value is incremented by 1?
If you are merely trying to return a sub-array of the values that are < 10, i would suggest just doing this in a for loop, and using an int as a counter.

Java Mutable BigInteger Class

I am doing calculations with BigIntegers that uses a loop that calls multiply() about 100 billion times, and the new object creation from the BigInteger is making it very slow. I was hoping somebody had written or found a MutableBigInteger class. I found the MutableBigInteger in the java.math package, but it is private and when I copy the code into a new class, many errors come up, most of which I don't know how to fix.
What implementations exist of a Java class like MutableBigInteger that allows modifying the value in place?

Is their any particular reason you cannot use reflection to gain access to the class?
I was able to do so without any problems, here is the code:
public static void main(String[] args) throws Exception {
Constructor<?> constructor = Class.forName("java.math.MutableBigInteger").getDeclaredConstructor(int.class);
constructor.setAccessible(true);
Object x = constructor.newInstance(new Integer(17));
Object y = constructor.newInstance(new Integer(19));
Constructor<?> constructor2 = Class.forName("java.math.MutableBigInteger").getDeclaredConstructor(x.getClass());
constructor2.setAccessible(true);
Object z = constructor.newInstance(new Integer(0));
Object w = constructor.newInstance(new Integer(0));
Method m = x.getClass().getDeclaredMethod("multiply", new Class[] { x.getClass(), x.getClass()});
Method m2 = x.getClass().getDeclaredMethod("mul", new Class[] { int.class, x.getClass()});
m.setAccessible(true);
m2.setAccessible(true);
// Slightly faster than BigInteger
for (int i = 0; i < 200000; i++) {
m.invoke(x, y, z);
w = z;
z = x;
x = w;
}
// Significantly faster than BigInteger and the above loop
for (int i = 0; i < 200000; i++) {
m2.invoke(x, 19, x);
}
BigInteger n17 = new BigInteger("17");
BigInteger n19 = new BigInteger("19");
BigInteger bigX = n17;
// Slowest
for (int i = 0; i < 200000; i++) {
bigX = bigX.multiply(n19);
}
}
Edit:
I decided to play around with a bit more, it does appear that java.math.MutableBigInteger doesn't behave exactly as you would expect.
It operates differently when you multiply and it will throw a nice exception when it has to increase the size of the internal array when assigning to itself. Something I guess is fairly expected. Instead I have to swap around the objects so that it is always placing the result into a different MutableBigInteger. After a couple thousand calculations the overhead from reflection becomes negligible. MutableBigInteger does end up pulling ahead and offers increasingly better performance as the number of operations increases. If you use the 'mul' function with an integer primitive as the value to multiply with, the MutableBigInteger runs almost 10 times faster than using BigInteger. I guess it really boils down to what value you need to multiply with. Either way if you ran this calculation "100 billion times" using reflection with MutableBigInteger, it would run faster than BigInteger because there would be "less" memory allocation and it would cache the reflective operations, removing overhead from reflection.

JScience has a class call LargeInteger, which is also immutable, but which they claim has significantly improved perfomance compared to BigInteger.
http://jscience.org/
APFloat's Apint might be worth checking out too. http://www.apfloat.org/apfloat_java/

I copied MutableBigInteger, then commented out some methods' bodies I dind't need, adding a nice
throw new UnsupportedOperationException("...");
when invoked.
here is how it looks.
In Revisions you can see what's changed from the original java.math.MutableBigInteger.
I also added some convenience methods,
public void init(long val) {};
public MutableBigInteger(long val) {};
// To save previous value before modifying.
public void addAndBackup(MutableBigInteger addend) {}
// To restore previous value after modifying.
public void restoreBackup() {}
Here is how I used it:
private BigInteger traverseToFactor(BigInteger offset, BigInteger toFactorize, boolean forward) {
MutableBigInteger mbiOffset = new MutableBigInteger(offset);
MutableBigInteger mbiToFactorize = new MutableBigInteger(toFactorize);
MutableBigInteger blockSize = new MutableBigInteger(list.size);
if (! MutableBigInteger.ZERO.equals(mbiOffset.remainder(blockSize))) {
throw new ArithmeticException("Offset not multiple of blockSize");
}
LongBigArrayBigList pattern = (LongBigArrayBigList) list.getPattern();
while (true) {
MutableBigInteger divisor = new MutableBigInteger(mbiOffset);
for (long i = 0; i < pattern.size64(); i++) {
long testOperand = pattern.getLong(i);
MutableBigInteger.UNSAFE_AUX_VALUE.init(testOperand);
divisor.addAndBackup(MutableBigInteger.UNSAFE_AUX_VALUE);
if (MutableBigInteger.ZERO.equals(mbiToFactorize.remainder(divisor))) {
return divisor.toBigInteger();
}
divisor.restoreBackup();
}
if (forward) {
mbiOffset.add(blockSize);
} else {
mbiOffset.subtract(blockSize);
}
System.out.println(mbiOffset);
}
}