I have large Map where I store some objects. The Map is large: it has around 200k objects. When I try to run some methods, that require to read map values, the program freezes. When I debug it, it seems that my IDE is 'collecting data' (picture). It has never completed the task.
I have 16GB RAM.
What can I do to speed this up?
I get performance issues around 61 million elements.
import java.util.*;
public class BreakingMaps{
public static void main(String[] args){
int count = Integer.MAX_VALUE>>5;
System.out.println(count + " objects tested");
HashMap<Long, String> set = new HashMap<>(count);
for(long i = 0; i<count; i++){
Long l = i;
set.put(l, l.toString());
}
Random r = new Random();
for(int i = 0; i<1000; i++){
long k = r.nextInt()%count;
k = k<0?-k:k;
System.out.println(set.get(k));
}
}
}
I run the program with java -Xms12G -Xmx13G BreakingMaps
I suspect your problem is not the map, but circumstances surrounding the map. If I write the same program, but use a class with hashcode colisions then the program cannot handle 200K elements.
static class Key{
final long l;
public Key(long l){
this.l = l;
}
#Override
public int hashCode(){
return 1;
}
#Override
public boolean equals(Object o){
if(o!=null && o instanceof Key){
return ((Key)o).l==l;
}
return false;
}
}
Look at this - as the solution you can increase the heap size for your app:
java -Xmx6g myprogram.
But it's not very good. I'd suggest you to try to rework your data processing approach. Maybe you can apply some filtering before fetching the data to decrease the data size or implement some calculation on database level.
I am working on a project for school and things are going well until i tried to perform a merge sort on my ArrayList.
It will run but then it errors out. The first error of many is Exception in thread "main" java.lang.StackOverflowError.
I have looked over the code and cant find out why the error is occurring.
It does give me a location ( line 74:first_half = mergeSort(first_half); ) but i don't see the issue.
public static void main(String[] args) throws IOException {
// URL url = new
// URL("https://www.cs.uoregon.edu/Classes/15F/cis212/assignments/phonebook.txt");
FileReader fileReader = new FileReader("TestSort.txt");
BufferedReader bufferReader = new BufferedReader(fileReader);
String entry = bufferReader.readLine();
// Scanner s = new Scanner(url.openStream());
// int count = 0;
while (entry != null) {
// String person = s.nextLine();
String phoneNum = entry.substring(0, 7);
String name = entry.substring(9);
PhonebookEntry newentry = new PhonebookEntry(name, phoneNum);
phoneBook.add(newentry);
entry = bufferReader.readLine();
}
// ********************Selection
// Sort*************************************
ArrayList<PhonebookEntry> sortList = new ArrayList<PhonebookEntry>(phoneBook);
for (int min = 0; min < sortList.size(); min++) {
for (int i = min; i < sortList.size(); i++) {
int res = sortList.get(min).getName().compareTo(sortList.get(i).getName());
if (res > 0) {
PhonebookEntry temp = sortList.get(i);
sortList.set(i, sortList.get(min));
sortList.set(min, temp);
}
}
}
for (PhonebookEntry sortentry : sortList) {
System.out.println(sortentry);
}
System.out.println(mergeSort(mergeSortList));
}
// *****************************merge sort******************************************
static int mergecounter = 0;
static ArrayList<PhonebookEntry> mergeSortList = new ArrayList<PhonebookEntry>(appMain.phoneBook);
public static ArrayList<PhonebookEntry> mergeSort(ArrayList<PhonebookEntry> mergeSortLists) {
if (mergeSortLists.size() == 1) {
return mergeSortLists;
}
int firstHalf = mergeSortLists.size() % 2 == 0 ? mergeSortLists.size() / 2 : mergeSortLists.size() / 2 + 1;
ArrayList<PhonebookEntry> first_half = new ArrayList<PhonebookEntry>(mergeSortLists.subList(0, firstHalf));
ArrayList<PhonebookEntry> mergeSortHalf2 = new ArrayList<PhonebookEntry>(
mergeSortLists.subList(first_half.size(), mergeSortLists.size()));
System.out.println(++mergecounter);
first_half = mergeSort(first_half);
mergeSortHalf2 = mergeSort(mergeSortHalf2);
return merge(first_half, mergeSortHalf2);
}
public static ArrayList<PhonebookEntry> merge(ArrayList<PhonebookEntry> first_half,
ArrayList<PhonebookEntry> mergeSortHalf2) {
ArrayList<PhonebookEntry> returnMerge = new ArrayList<PhonebookEntry>();
while (first_half.size() > 0 && mergeSortHalf2.size() > 0) {
if (first_half.get(0).getName().compareTo(mergeSortHalf2.get(0).getName()) > 0) {
returnMerge.add(mergeSortHalf2.get(0));
mergeSortHalf2.remove(0);
}
else {
returnMerge.add(first_half.get(0));
first_half.remove(first_half.get(0));
}
}
while (first_half.size() > 0) {
returnMerge.add(first_half.get(0));
first_half.remove(first_half.get(0));
}
while (mergeSortHalf2.size() > 0) {
returnMerge.add(mergeSortHalf2.get(0));
mergeSortHalf2.remove(mergeSortHalf2.get(0));
}
return returnMerge;
}
}
My opinion there is no error in code.
How so sure?
I ran you code in my environment and its executed without any error.
With the text file i found at https://www.cs.uoregon.edu/Classes/15F/cis212/assignments/phonebook.txt As input
and done a simple implementation for PhonebookEntry
Then why is this error?
First off all try to understand the error, I mean why StackOverflowError occur. As there are lots of I am not going to explain this
But please read the top answer of this two thread and i am sure you will know why this happen.
Thread 1: What is a StackOverflowError?
Thread 2: What actually causes a Stack Overflow error?
If you read those I hope you understand the summury is You Ran Out Of Memory.
Then why I didnt got that error: Possible reason is
In my environment I configured the jvm to run with a higher memory 1024m to 1556m (as eclipse parameter)
Now lets analyze your case with solution:
Input: you have big input here ( 50,000 )
To check you code try to shorten the input and test.
You have executed two algorithm in a sigle method over this big Input:
When a method execute all its varibles stay in the memory untill it complete its execution.
so when you are calling merge sort all previouly user vairables and others stay in the memory which can contribute to this situation
Now if you use separated method and call them from the main method like write an method for selection sort, all its used varible will go out of scope
and possibly be free (if GC collect them) after the selection sort is over.
So write two separated method for reading input file and selection sort.
And Please Please close() those FileReader and BufferedReader.
Get out of those static mehtod . Make them non static create and object of the class and call them from main method
So its all about code optimization
And also you can just increase the memory for jvm and test by doing like this java -Xmx1556m -Xms1024m when ruining the app in command line
BTW, Thanks for asking this this question its gives me something to think about
Here is the function i wrote. it take a Stringbuffer text then assign v[0]=text[0] , then starts from text[1] >>>text[n-1] the comparing. The vector v should contain the characters. I don't know where is the problem. Can you help me?
public void setdirectory(StringBuffer text)
{
String temp;
boolean t;
v.add(0,String.valueOf(text.charAt(0))); //A[0]=first letter in text.
for(int i=1;i<text.length();++i)
{
temp=String.valueOf(text.charAt(i));
try{
for(int j=0;j<v.capacity();++j)
{
if(!temp.equals(v.elementAt(j)))
{
v.add(i,temp);
}
v.trimToSize();
}
// System.out.println(v.capacity());
}catch(ArrayIndexOutOfBoundsException e)
{
System.out.println("usage error");
}
}
}
If you're using Java 8+, then it might be simpler to use the new Stream API...
String str = "aabbc";
StringBuilder sb = new StringBuilder(str.length());
str.chars().distinct().forEach(c -> sb.append((char)c));
System.out.println(sb.toString());
Which prints
abc
I'd write a function to get unique characters, and assuming you need to preserve the insertion order, I'd use a LinkedHashSet<Character> and I'd prefer StringBuilder over StringBuffer. Something like
static String getUniqueCharacters(String text) {
Set<Character> set = new LinkedHashSet<>();
for (char ch : text.toCharArray()) {
set.add(ch);
}
StringBuilder sb = new StringBuilder();
for (char ch : set) {
sb.append(ch);
}
return sb.toString();
}
An alternative Java 8 solution is:
String str = "aabbc";
String str2 = str.chars().distinct().mapToObj(j->""+(char)j).collect(Collectors.joining());
System.out.println(str2);
Behind the scenes, this is similar to other answers here as IntStream::distinct is implemented using a LinkedHashSet<Integer>, and joining uses a StringBuilder.
You need to keep track of where you are adding your value in the vector. Also the number of objects in a vector is size(), not capacity() (look up the API for both; capacity() shows the current number of 'spaces' filled and available to fill before the vector needs to expand, it doesn't show how much of it has actually been filled).
Doh, and the third reason your code would not have worked: you were adding the character every time it found a non-matching one in the vector (over-writing itself each time so you would have only seen the last addition)
public void setdirectory(StringBuffer text) {
String temp;
boolean t;
int addAt = 0;
v.add(addAt,String.valueOf(text.charAt(0))); //A[0]=first letter in text.
for(int i=1;i<text.length();++i) {
temp=String.valueOf(text.charAt(i));
try {
boolean found = false
for(int j=0;j<v.size();++j) {
if(temp.equals(v.elementAt(j))) {
found = true;
break;
}
}
if (!found) {
addAt++;
v.add(addAt,temp);
}
} catch(ArrayIndexOutOfBoundsException e) {
System.out.println("usage error");
}
}
}
And although this would fix your code as it stands (which will be an important exercise for a beginner programmer), there are other ways of doing this that you should explore.
more updates
As is explained in the selected answer, the problem is in JVM's garbage collection algorithm.
JVM uses card marking algorithm to keep track of modified references in object fields. For each reference assignment to a field, it marks an associated bit in the card to be true -- this causes a false-sharing hence blocks scaling. The details are well described in this article: https://blogs.oracle.com/dave/entry/false_sharing_induced_by_card
The option -XX:+UseCondCardMark (in Java 1.7u40 and up) mitigates the problem, and makes it scale almost perfectly.
updates
I found out (hinted from Park Eung-ju) that assigning an object into a field variable makes the difference. If I remove the assignment, it scales perfectly.
I think probably it has something to do with Java memory model -- such as, an object reference must point to a valid address before it gets visible, but I am not completely sure. Both double and Object reference (likely) have 8 bytes size on 64 bit machine, so it seems to me that assigning a double value and an Object reference should be the same in terms of synchronization.
Anyone has a reasonable explanation?
Here I have a weird Java multi-threading scalability problem.
My code simply iterates over an array (using the visitor pattern) to compute simple floating-point operations and assign the result to another array. There is no data dependency, nor synchronization, so it should scale linearly (2x faster with 2 threads, 4x faster with 4 threads).
When primitive (double) array is used, it scales very well. When object type (e.g. String) array is used, it doesn't scale at all (even though the value of the String array is not used at all...)
Here's the entire source code:
import java.util.ArrayList;
import java.util.Arrays;
import java.util.concurrent.CyclicBarrier;
class Table1 {
public static final int SIZE1=200000000;
public static final boolean OBJ_PARAM;
static {
String type=System.getProperty("arg.type");
if ("double".equalsIgnoreCase(type)) {
System.out.println("Using primitive (double) type arg");
OBJ_PARAM = false;
} else {
System.out.println("Using object type arg");
OBJ_PARAM = true;
}
}
byte[] filled;
int[] ivals;
String[] strs;
Table1(int size) {
filled = new byte[size];
ivals = new int[size];
strs = new String[size];
Arrays.fill(filled, (byte)1);
Arrays.fill(ivals, 42);
Arrays.fill(strs, "Strs");
}
public boolean iterate_range(int from, int to, MyVisitor v) {
for (int i=from; i<to; i++) {
if (filled[i]==1) {
// XXX: Here we are passing double or String argument
if (OBJ_PARAM) v.visit_obj(i, strs[i]);
else v.visit(i, ivals[i]);
}
}
return true;
}
}
class HeadTable {
byte[] filled;
double[] dvals;
boolean isEmpty;
HeadTable(int size) {
filled = new byte[size];
dvals = new double[size];
Arrays.fill(filled, (byte)0);
isEmpty = true;
}
public boolean contains(int i, double d) {
if (filled[i]==0) return false;
if (dvals[i]==d) return true;
return false;
}
public boolean contains(int i) {
if (filled[i]==0) return false;
return true;
}
public double groupby(int i) {
assert filled[i]==1;
return dvals[i];
}
public boolean insert(int i, double d) {
if (filled[i]==1 && contains(i,d)) return false;
if (isEmpty) isEmpty=false;
filled[i]=1;
dvals[i] = d;
return true;
}
public boolean update(int i, double d) {
assert filled[i]==1;
dvals[i]=d;
return true;
}
}
class MyVisitor {
public static final int NUM=128;
int[] range = new int[2];
Table1 table1;
HeadTable head;
double diff=0;
int i;
int iv;
String sv;
MyVisitor(Table1 _table1, HeadTable _head, int id) {
table1 = _table1;
head = _head;
int elems=Table1.SIZE1/NUM;
range[0] = elems*id;
range[1] = elems*(id+1);
}
public void run() {
table1.iterate_range(range[0], range[1], this);
}
//YYY 1: with double argument, this function is called
public boolean visit(int _i, int _v) {
i = _i;
iv = _v;
insertDiff();
return true;
}
//YYY 2: with String argument, this function is called
public boolean visit_obj(int _i, Object _v) {
i = _i;
iv = 42;
sv = (String)_v;
insertDiff();
return true;
}
public boolean insertDiff() {
if (!head.contains(i)) {
head.insert(i, diff);
return true;
}
double old = head.groupby(i);
double newval=Math.min(old, diff);
head.update(i, newval);
head.insert(i, diff);
return true;
}
}
public class ParTest1 {
public static int THREAD_NUM=4;
public static void main(String[] args) throws Exception {
if (args.length>0) {
THREAD_NUM = Integer.parseInt(args[0]);
System.out.println("Setting THREAD_NUM:"+THREAD_NUM);
}
Table1 table1 = new Table1(Table1.SIZE1);
HeadTable head = new HeadTable(Table1.SIZE1);
MyVisitor[] visitors = new MyVisitor[MyVisitor.NUM];
for (int i=0; i<visitors.length; i++) {
visitors[i] = new MyVisitor(table1, head, i);
}
int taskPerThread = visitors.length / THREAD_NUM;
MyThread[] threads = new MyThread[THREAD_NUM];
CyclicBarrier barrier = new CyclicBarrier(THREAD_NUM+1);
for (int i=0; i<THREAD_NUM; i++) {
threads[i] = new MyThread(barrier);
for (int j=taskPerThread*i; j<taskPerThread*(i+1); j++) {
if (j>=visitors.length) break;
threads[i].addVisitors(visitors[j]);
}
}
Runtime r=Runtime.getRuntime();
System.out.println("Force running gc");
r.gc(); // running GC here (excluding GC effect)
System.out.println("Running gc done");
// not measuring 1st run (excluding JIT compilation effect)
for (int i=0; i<THREAD_NUM; i++) {
threads[i].start();
}
barrier.await();
for (int i=0; i<10; i++) {
MyThread.start = true;
long s=System.currentTimeMillis();
barrier.await();
long e=System.currentTimeMillis();
System.out.println("Iter "+i+" Exec time:"+(e-s)/1000.0+"s");
}
}
}
class MyThread extends Thread {
static volatile boolean start=true;
static int tid=0;
int id=0;
ArrayList<MyVisitor> tasks;
CyclicBarrier barrier;
public MyThread(CyclicBarrier _barrier) {
super("MyThread"+(tid++));
barrier = _barrier;
id=tid;
tasks = new ArrayList(256);
}
void addVisitors(MyVisitor v) {
tasks.add(v);
}
public void run() {
while (true) {
while (!start) { ; }
for (int i=0; i<tasks.size(); i++) {
MyVisitor v=tasks.get(i);
v.run();
}
start = false;
try { barrier.await();}
catch (InterruptedException e) { break; }
catch (Exception e) { throw new RuntimeException(e); }
}
}
}
The Java code can be compiled with no dependency, and you can run it with the following command:
java -Darg.type=double -server ParTest1 2
You pass the number of worker threads as an argument (the above uses 2 threads).
After setting up the arrays (that is excluded from the measured time), it does a same operation for 10 times, printing out the execution time at each iteration.
With the above option, it uses double array, and it scales very well with 1,2,4 threads (i.e. the execution time reduces to 1/2, and 1/4), but
java -Darg.type=Object -server ParTest1 2
With this option, it uses Object (String) array, and it doesn't scale at all!
I measured the GC time, but it was insignificant (and I also forced running GC before measuring times). I have tested with Java 6 (updates 43) and Java 7 (updates 51), but it's the same.
The code has comments with XXX and YYY describing the difference when arg.type=double or arg.type=Object option is used.
Can you figure out what is going on with the String-type argument passing here?
HotSpot VM generates following assemblies for reference type putfield bytecode.
mov ref, OFFSET_OF_THE_FIELD(this) <- this puts the new value for field.
mov this, REGISTER_A
shr 0x9, REGISTER_A
movabs OFFSET_X, REGISTER_B
mov %r12b, (REGISTER_A, REGISTER_B, 1)
putfield operation is completed in 1 instruction.
but there are more instructions following.
They are "Card Marking" instructions. (http://www.ibm.com/developerworks/library/j-jtp11253/)
Writing reference field to every objects in a card (512 bytes), will store a value in a same memory address.
And I guess, store to same memory address from multiple cores mess up with cache and pipelines.
just add
byte[] garbage = new byte[600];
to MyVisitor definition.
then every MyVisitor instances will be spaced enough not to share card marking bit, you will see the program scales.
This is not a complete answer but may provide a hint for you.
I have changed your code
Table1(int size) {
filled = new byte[size];
ivals = new int[size];
strs = new String[size];
Arrays.fill(filled, (byte)1);
Arrays.fill(ivals, 42);
Arrays.fill(strs, "Strs");
}
to
Table1(int size) {
filled = new byte[size];
ivals = new int[size];
strs = new String[size];
Arrays.fill(filled, (byte)1);
Arrays.fill(ivals, 42);
Arrays.fill(strs, new String("Strs"));
}
after this change, the running time with 4 threads with object type array reduced.
According to http://docs.oracle.com/javase/specs/jls/se7/html/jls-17.html#jls-17.7
For the purposes of the Java programming language memory model, a single write to a non-volatile long or double value is treated as two separate writes: one to each 32-bit half. This can result in a situation where a thread sees the first 32 bits of a 64-bit value from one write, and the second 32 bits from another write.
Writes and reads of volatile long and double values are always atomic.
Writes to and reads of references are always atomic, regardless of whether they are implemented as 32-bit or 64-bit values.
Assigning references are always atomic,
and double is not atomic except when it is defined as volatile.
The problem is sv can be seen by other threads and its assignment is atomic.
Therefore, wrapping visitor's member variables (i, iv, sv) using ThreadLocal will solve the problem.
"sv = (String)_v;" makes the difference. I also confirmed that the type casting is not the factor. Just accessing _v can't make the difference. Assigning some value to sv field makes the difference. But I can't explain why.
The user enters an expression. Suppose user entered the following as input:
new y java.util.ArrayList int:5
i have successfully tokenized the string and stored it into different locations of my String array. next thing i want to do is that i should check whats on the index and do things according as mentioned in the above input for reflection. Am stuck how to do it. here is my code so far
public static void handling_input()
{
System.out.println("Welcome To Java Command Prompt: ");
aLine = null;
try
{
System.out.println("Enter The Command Line Expression: ") ;
keyboard = new BufferedReader(new InputStreamReader(System.in));
aLine = keyboard.readLine();
st = new StringTokenizer(aLine);
dt = new StringTokenizer(aLine);
}
catch (IOException ex)
{
System.out.println("Error reading input!");
}
}
public static void storing_tokens()
{
int counter =0;
while(st.hasMoreTokens())
{
counter++;
st.nextToken();
}
int i=0;
expression_keeper= new String[counter];
do
{
expression_keeper[i] = dt.nextToken().toString();
i++;
}while(dt.hasMoreTokens());
}
public static void token_classification()
{
for(int i=0; i<expression_keeper.length; i++)
{
if(expression_keeper[0].equalsIgnoreCase("new"))
{
}
else
if(expression_keeper[0].equalsIgnoreCase("call"))
{
}
else
if(expression_keeper[0].equalsIgnoreCase("print"))
{
}
else
{
System.out.println("Invalid Script!");
}
}
}
}
Inside this if condition:
if(expression_keeper[0].equalsIgnoreCase("new"))
{
}
i want to create the specified class,its object and assign values to the modifiers mentioned!
It is unclear to me what your input string tokens really mean. Is "java.util.ArrayList" the type for "y" and should it have an initial size of 5 units? Or should the first element be an integer of 5?
In the past I have found writing my own syntax tokenizer and parser a complicated thing to do. Even in simple cases I have often found that using something like JavaCC was easier in the long run.
By specifying your syntax formally you give much but structure to your code and it's debuggability. And then as said elsewhere use introspection to do the creation. The packages to do this are in java.lang.reflect.