We Keep Coding

Java 8 hashmap high memory usage

I use a hashmap to store a QTable for an implementation of a reinforcement learning algorithm. My hashmap should store 15000000 entries. When I ran my algorithm I saw that the memory used by the process is over 1000000K. When I calculated the memory, I would expect it to use not more than 530000K. I tried to write an example and I got the same high memory usage:
public static void main(String[] args) {
HashMap map = new HashMap<>(16_000_000, 1);
for(int i = 0; i < 15_000_000; i++){
map.put(i, i);
}
}
My memory calulation:
Each entryset is 32 bytes
Capacity is 15000000
HashMap Instance uses: 32 * SIZE + 4 * CAPACITY
memory = (15000000 * 32 + 15000000 * 4) / 1024 = 527343.75K
Where I'm wrong in my memory calculations?

Well, in the best case, we assume a word size of 32 bits/4 bytes (with CompressedOops and CompressedClassesPointers). Then, a map entry consists of two words JVM overhead (klass pointer and mark word), key, value, hashcode and next pointer, making 6 words total, in other words, 24 bytes. So having 15,000,000 entry instances will consume 360 MB.
Additionally, there’s the array holding the entries. The HashMap uses capacities that are a power of two, so for 15,000,000 entries, the array size is at least 16,777,216, consuming 64 MiB.
Then, you have 30,000,000 Integer instances. The problem is that map.put(i, i) performs two boxing operations and while the JVM is encouraged to reuse objects when boxing, it is not required to do so and reusing won’t happen in your simple program that is likely to complete before the optimizer ever interferes.
To be precise, the first 128 Integer instances are reused, because for values in the -128 … +127 range, sharing is mandatory, but the implementation does this by initializing the entire cache on the first use, so for the first 128 iterations, it doesn’t create two instances, but the cache consists of 256 instances, which is twice that number, so we end up again with 30,000,000 Integer instances total. An Integer instance consist of at least the two JVM specific words and the actual int value, which would make 12 bytes, but due to the default alignment, the actually consumed memory will be 16 bytes, dividable by eight.
So the 30,000,000 created Integer instances consume 480 MB.
This makes a total of 360 MB + 64 MiB + 480 MB, which is more than 900 MB, making a heap size of 1 GB entirely plausible.
But that’s what profiling tools are for. After running your program, I got
Note that this tool only reports the used size of the objects, i.e. the 12 bytes for an Integer object without considering the padding that you will notice when looking at the total memory allocated by the JVM.

I sort of had the same requirement as you.. so decided to throw my thoughts here.
1) There is a great tool for that: jol.
2) Arrays are objects too, and every object in java has two additional headers: mark and klass, usually 4 and 8 bytes in size (this can be tweaked via compressed pointers, but not going to go into details).
3) Is is important to note about the load factor here of the map (because it influences the resize of the internal array). Here is an example:
HashMap<Integer, Integer> map = new HashMap<>(16, 1);
for (int i = 0; i < 13; ++i) {
map.put(i, i);
}
System.out.println(GraphLayout.parseInstance(map).toFootprint());
HashMap<Integer, Integer> map2 = new HashMap<>(16);
for (int i = 0; i < 13; ++i) {
map2.put(i, i);
}
System.out.println(GraphLayout.parseInstance(map2).toFootprint());
Output of this is different(only the relevant lines):
1 80 80 [Ljava.util.HashMap$Node; // first case
1 144 144 [Ljava.util.HashMap$Node; // second case
See how the size is bigger for the second case because the backing array is twice as big (32 entries). You can only put 12 entries in a 16 size array, because the default load factor is 0.75: 16 * 0.75 = 12.
Why 144? The math here is easy: an array is an object, thus: 8+4 bytes for headers. Plus 32 * 4 for references = 140 bytes. Due to memory alignment of 8 bytes, there are 4 bytes for padding resulting in a total 144 bytes.
4) entries are stored inside either a Node or a TreeNode inside the map (Node is 32 bytes and TreeNode is 56 bytes). As you use ONLY integers, you will have only Nodes, as there should be no hash collisions. There might be collisions, but this does not yet mean that a certain array entry will be converted to a TreeNode, there is a threshold for that. We can easily prove that there will be Nodes only:
public static void main(String[] args) {
Map<Integer, List<Integer>> map = IntStream.range(0, 15_000_000).boxed()
.collect(Collectors.groupingBy(WillThereBeTreeNodes::hash)); // WillThereBeTreeNodes - current class name
System.out.println(map.size());
}
private static int hash(Integer key) {
int h = 0;
return (h = key.hashCode()) ^ h >>> 16;
}
The result of this will be 15_000_000, there was no merging, thus no hash-collisions.
5) When you create Integer objects there is pool for them (ranging from -127 to 128 - this can be tweaked as well, but let's not for simplicity).
6) an Integer is an object, thus it has 12 bytes header and 4 bytes for the actual int value.
With this in mind, let's try and see the output for 15_000_000 entries (since you are using a load factor of one, there is no need to create the internal capacity of 16_000_000). It will take a lot of time, so be patient. I also gave it a
-Xmx12G and -Xms12G
HashMap<Integer, Integer> map = new HashMap<>(15_000_000, 1);
for (int i = 0; i < 15_000_000; ++i) {
map.put(i, i);
}
System.out.println(GraphLayout.parseInstance(map).toFootprint());
Here is what jol said:
java.util.HashMap#9629756d footprint:
COUNT AVG SUM DESCRIPTION
1 67108880 67108880 [Ljava.util.HashMap$Node;
29999872 16 479997952 java.lang.Integer
1 48 48 java.util.HashMap
15000000 32 480000000 java.util.HashMap$Node
44999874 1027106880 (total)
Let's start from bottom.
total size of the hashmap footprint is: 1027106880 bytes or 1 027 MB.
Node instance is the wrapper class where each entry resides. it has a size of 32 bytes; there are 15 million entries, thus the line:
15000000 32 480000000 java.util.HashMap$Node
Why 32 bytes? It stores the hashcode(4 bytes), key reference (4 bytes), value reference (4 bytes), next Node reference (4 bytes), 12 bytes header, 4 bytes padding, resulting in 32 bytes total.
1 48 48 java.util.HashMap
A single hashmap instance - 48 bytes for it's internals.
If you really want to know why 48 bytes:
System.out.println(ClassLayout.parseClass(HashMap.class).toPrintable());
java.util.HashMap object internals:
OFFSET SIZE TYPE DESCRIPTION VALUE
0 12 (object header) N/A
12 4 Set AbstractMap.keySet N/A
16 4 Collection AbstractMap.values N/A
20 4 int HashMap.size N/A
24 4 int HashMap.modCount N/A
28 4 int HashMap.threshold N/A
32 4 float HashMap.loadFactor N/A
36 4 Node[] HashMap.table N/A
40 4 Set HashMap.entrySet N/A
44 4 (loss due to the next object alignment)
Instance size: 48 bytes
Space losses: 0 bytes internal + 4 bytes external = 4 bytes total
Next the Integer instances:
29999872 16 479997952 java.lang.Integer
30 million integer objects (minus 128 that are cached in the pool)
1 67108880 67108880 [Ljava.util.HashMap$Node;
we have 15_000_000 entries, but the internal array of a HashMap is a power of two size, that's 16,777,216 references of 4 bytes each.
16_777_216 * 4 = 67_108_864 + 12 bytes header + 4 padding = 67108880

Why maximum integer number multiplication gives 1 as result

I think much explanation is not required, why below calculation gives result as 1?
int a = 2147483647;
int b = 2147483647;
int c = a * b;
long d = a * b;
double e = a * b;
System.out.println(c); //1
System.out.println(d); //1
System.out.println(e); //1.0

The binary representation of the integer number 2147483647 is as following:
01111111 11111111 11111111 11111111
Multiplying this with itself results in the number 4611686014132420609 whose binary representation is:
00111111 11111111 11111111 11111111 00000000 00000000 00000000 00000001
This is too large for the int type which has only 32 bits. The multiplication of a * b is done as an integer multiplication only, regardless of the variable's type to which the result is assigned (which might do a widening conversion, but only after the multiplication).
So, the result simply cuts off all bits that do not fit into the 32 bits, leaving only the following result:
00000000 00000000 00000000 00000001
And this simply is the value 1.
If you want keep the information, you must do the multiplication with the long type which has 64 bits:
long a = 2147483647;
long b = 2147483647;
long mult = a * b;
System.out.println((int) mult); // 1
System.out.println(mult); // 4611686014132420609
System.out.println((double) mult); // 4.6116860141324206E18
If you need more bits for a calculation you might consider BigInteger (for integral numbers) or BigDecimal (for decmial numbers).

2147483647 * 2147483647 = 4611686014132420609
Which in Hexa = ‭3FFFFFFF 00000001‬, after truncation only 00000001 remains which represents 1.

First of all, the reason that the three attempts all give the same answer is that they are all performing 32 bit multiplications and the multiplication overflows, resulting in "loss of information". The overflow / loss of information happens before the value of the RHS1 expression is assigned to the variable on the LHS.
In the 2nd and 3rd case you could cause the expression to be evaluated using 64 bit or floating point:
int c = a * b;
long d = ((long) a) * b;
double e = ((double) a) * b;
and you would not get overflow.
As to why you get overflow in the 32 bit case, that is simple. The result is larger than 32 bits. The other answers explain why the answer is 1.
Just for fun, here is an informal proof.
Assume that we are talking about a modular number system with numbers in the range 2N-1 to 2N-1 - 1. In such a number system, X * 2N maps to zero ... for all integers X.
If we multiply the max value by itself we get
(2N-1 - 1) * (2N-1 - 1)
-> 22N-2 - 2 * 2N-1 + 1
-> 22N-2 - 2N + 1
Now map that into the original range:
22N-2 maps to 0
2N maps 0
1 maps to 1
0 + 0 + 0 -> 1
1 - LHS == left hand side, RHS == right hand side.

What you are seeing is simply the result of integer overflow, which follows this rule:
Integer.MAX_VALUE + 1 = Integer.MIN_VALUE
One way to see what is happening is to contrive Java's int type as ranging from -7 to 7, with the same rule still applying. Let's see what happens when we multiply 7*7:
7 + 7 = 14 -> -2 (7 x 2)
-2 + 7 = 5 (7 x 3)
5 + 7 = 12 -> -4 (7 x 4)
-4 + 7 = 3 (7 x 5)
3 + 7 = 10 -> -6 (7 x 6)
-6 + 7 = 1 (7 x 7, one is leftover)
The same thing is happening in your code, with 2147483647 overflowing according to:
2147483647 + 1 = -2147483648

Because:
2147483647 * 2147483647 = 4611686014132420609
Integer capacity = 4294967295
4611686014132420609 % 4294967295 = 1

integer has only 3 byte memory allocation as well as double has 8 byte memory allocation your multiplication is so large than its give only starting which has (0000 0001)2 = 1

Java Code Integer Output

I am trying to write a code which raises 2 to
the powers of 0, 1, 2. 3. 4, .... n
For example for 31 it would reproduce the following results:
1 2 4 8 16 32 64 128 256 512 1024 2048 4096 8192
16384 32768 65536 131072 262144 524288 1048576 2097152
4194304 8388608 16777216 33554432 67108864 134217728
268435456 536870912 1073741824 2147483648
I have written the following code:
public class Powersof2 {
public static void printPowersOf2(int n) {
if (n >= 0 && n <= 64) {
for (int i = 0 ; i <= n ; i++) {
System.out.print( Math.pow(2, i) + " ");
}
} else {
System.out.println("You have input an invalid value");
return;
}
}
public static void main(String[] args) {
printPowersOf2(31);
System.out.println();
}
}
However, it produces the following result instead:
1.0 2.0 4.0 8.0 16.0 32.0 64.0 128.0 256.0 512.0
1024.0 2048.0 4096.0 8192.0 16384.0 32768.0 65536.0
131072.0 262144.0 524288.0 1048576.0 2097152.0 4194304.0
8388608.0 1.6777216E7 3.3554432E7 6.7108864E7 1.34217728E8
2.68435456E8 5.36870912E8 1.073741824E9 2.147483648E9
How can I fix that?
Also I have another question
When I input larger values for n such as 62 the values start to be the same. For example 62 gives:
1 2 4 8 16 32 64 128 256 512 1024 2048 4096
8192 16384 32768 65536 131072 262144 524288 1048576
2097152 4194304 8388608 16777216 33554432 67108864 134217728
268435456 536870912 1073741824 2147483647 2147483647
2147483647 2147483647 2147483647 2147483647 2147483647
2147483647 2147483647 2147483647 2147483647 2147483647
2147483647 2147483647 2147483647 2147483647 2147483647
2147483647 2147483647 2147483647 2147483647 2147483647
2147483647 2147483647 2147483647 2147483647 2147483647
2147483647 2147483647 2147483647 2147483647 2147483647
how can I fix this issue?

Since Math.pow returns a double while you are looking for an integer, consider using a simple bit trick for raising 2 to power n like this:
1L << n
The reason behind this trick is the same as that behind multiplying by powers of ten in the decimal system by writing additional zeros after the number.
This works for ns between 0 and 63, and produce a long:
for (int i = 0 ; i <= n ; i++) {
System.out.print( (1L << i) + " ");
}

Math.pow() always returns doubles. You need to cast it to an int. System.out.print((int)Math.pow(2, i)+ " ");

Because public static double pow(double a, double b) return a double. And Java double is 64 bit IEE754 number with precision, so you see the decimal points in the end.
You can cast it to int as (int)Math.pow(2, i)but be aware that for 2^31 onwards you will not get expected results because int range is from -2^31 to +2^31.
** First Solution:**
So, to get your expected result to get power up to 63, cast to long because Java long has range from -2^63 to +2^63
for (int i = 0; i <= 64; i++) {
System.out.println((long)Math.pow(2, i));
}
You query about result as so many 2147483647 :
This is called as Narrowing, when a bigger container value is casted to smaller container value.
Below excerpt from Java Language Specification - $5.1.3 Narrowing Primitive Conversion, for rules governing primitive narrowing.
So, after power of 31, all the higher bits (32, 33....) are discarded, and hence you always result of 31 bits i.e. 2^31.
In you case, as per JLS, your "integral type T" was int, so all bits are discarded but the n lowest order bits (which means 31, because you are narrowing to int)
A narrowing conversion of a signed integer to an integral type T
simply discards all but the n lowest order bits, where n is the number
of bits used to represent type T. In addition to a possible loss of
information about the magnitude of the numeric value, this may cause
the sign of the resulting value to differ from the sign of the input
value.
A comprehensive solution: (as per discussion going around)
Primitive data types has range limitation, so there are BigInteger, BigDecimal which provides arbitrarily long values and precision. So, you can use BigInteger to get accurate result.
Notice that if you Java's long then for 2^63 you get result as 9223372036854775807 while as per Power of two Wiki result should be 9223372036854775808 which you get when you use BigInteger
Bottom line: Use BigInteger to get desired range of power of 2.
Hope this covers all your concern and give you scalable solution.
for (int i = 0; i <= 64; i++) {
//System.out.println((long)Math.pow(2, i));
System.out.println(BigInteger.valueOf(2).pow(i)); //Gives exact result as per Wiki information
}

These are the powers of 2 through 2^31, just in the default output format of a double. With lower values, it will append .0, and with higher values, it's in pseudo-scientific notation. E.g. 1.6777216E7 means 1.677216 x 107, or 1,677,216. The double is returned from Math.pow.
You may find an integer type data type better formatted for your purpose. Use long, because int isn't quite big enough to store 2^31.
long product = 1;
for (int i = 0 ; i <= n ; i++)
{
System.out.print( product + " ");
product *= 2; // Equivalent: product <<= 1;
}

i could have used below code part, instead math.pow() , i am always scared of type conversion
int num=1;
for(int i=0;i<n;i++){
System.out.println(num);
num=num*2;
}
but as suggested it can not go beyond 31 . it is limit of int data type. try BigInteger if you want to print more

With Java 8 you can simply use the following stream:
IntStream.range(0, 32)
.map(value -> 1 << value)
.forEach(value -> System.out.println("value = " + value));
Also, the same is applicable to LongStream.

Why both (byte) 400000 and (byte) -400000 result -128

In Java, why both (byte) 400000 and (byte) -400000 have result -128?
Actually, I followed the calculation method from https://stackoverflow.com/a/9085666/1037217
For case: 400000
Binary: 1100001101010000000
Trim to 8 digits: 10000000
Since the left most digit is 1, so -1 from it: 01111111
Then invert it: 10000000
Result: -128
For case: -400000
Binary: -1100001101010000000
Trim to 8 digits: 10000000
Since the left most digit is 1, so -1 from it: 01111111
Then invert it: 10000000
Result: 128
The same method works on
(short) 40000 = -25536
(short) -40000 = 25536

Casting an int to byte will preserve the int number's last 8 bits (the last byte).
400000 = 0x61a80
-400000 = 0xfff9e580
Both of your numbers have the same last 8 bits: 0x80 which is -1 in 2's complement.
For example:
System.out.println((byte)0x23403); // Prints 3 (the last 8 bits: 0x03 = 3)
System.out.println((byte)0x23483); // Prints -125 (last 8 bits: 0x83 = -125)
// (in 2's complement: 0x83 = -(128-3) = -125)

Because byte has the range -128 to 127. Both of your values overflow and are then subject to a narrowing conversion. To quote JLS Example 5.1.3-2. Narrowing Primitive Conversions that lose information,
// An int value too big for byte changes sign and magnitude:

As you say:
For case: 400000 Binary: 1100001101010000000 Trim to 8 digits: 10000000 Since the left most digit is 1, so -1 from it: 01111111 Then invert it: 10000000 Result: -128
For case: -400000 Binary: -1100001101010000000 Trim to 8 digits: 10000000 Since the left most digit is 1, so -1 from it: 01111111 Then invert it: 10000000 Result: 128
In both cases, the bit pattern you get is 10000000. That equates to -128 both times. A byte cannot represent the value 128; it is out of range.
However, your procedure is not quite right. You can't just put a negative sign there and then "trim to 8 digits". A negative sign is not a valid state for a bit. You should probably study the 2s complement representation of integers.

Why Integer.toBinaryString returns 32 bits if the argument is negative?

I was just messing around the Integer.toBinaryString(int) method.
When I pass a positive number say 7, it outputs 111 but when I pass negative 7, it outputs 11111111111111111111111111111001. I understand that Java uses 2's complement to represent negative numbers, but why 32 bits (I also know that an int is 32 bits long but doesn't fit in the answer)?

Ok so I did some digging...
I wrote up a little program probably close to what you did.
public class IntTest {
public static void main(String[] args){
int a = 7;
int b = -7;
System.out.println(Integer.toBinaryString(a));
System.out.println(Integer.toBinaryString(b));
}
}
My output:
111
11111111111111111111111111111001
So 111 is the same if it had 29 "0"s in front of it. That is just wasted space and time.
If we follow the instructions for twos compliment from this guy here you can see that what we must do is flip the bits ( zeros become ones and ones become zeros ) then we add 1 to the result.
So 0000 0000 0000 0000 0000 0000 0000 0111 becomes 1111 1111 1111 1111 1111 1111 1111 1001
The ones can not be thrown out because they are significant in the twos compliment representation. This is why you have 32 bits in the second case.
Hope this helps! -- Code on!!!

Because Java ints are signed 32-bit. If you use a negative number the first bit must be 1.
System.out.println(Integer.toBinaryString(0));
System.out.println(Integer.toBinaryString(Integer.MAX_VALUE)); // 31 bits
System.out.println(Integer.toBinaryString(Integer.MAX_VALUE - 1)); // 31 bits
System.out.println(Integer.toBinaryString(Integer.MAX_VALUE + 1)); // 32 bits
System.out.println(Integer.SIZE);
Output is
0
1111111111111111111111111111111
1111111111111111111111111111110
10000000000000000000000000000000
32
Note that Integer.MAX_VALUE + 1 is Integer.MIN_VALUE (and it has an extra bit).

It outputs the smallest number it can, stripping leading zeroes. In the case of a negative number, the first bit of the 32 bits is a sign bit (i.e. -1 is 1, 30 zeros, and another 1). So, since it has to output the sign bit (it's significant), it outputs all 32 bits.
Here's a cool semi-relevant example of using the sign bit and the unsigned shift operator :). If you do:
int x = {positive_value};
int y = {other_positive_value};
int avg = (x + y) >>> 1;
The x and y integers can both use the first 31 bits since the 32nd bit is the sign. This way, if they overflow, they overflow into the sign bit and make the value negative. The >>> is an unsigned shift operator which shifts the value back one bit to the right which is effectively a divide by two and floor operation, which gives a proper average.
If you, on the other hand, had done:
int x = {value};
int y = {other_value};
int avg = (x + y) / 2;
And you had gotten an overflow, you would end up with the wrong result as you'd be dividing a negative value by 2.