We Keep Coding

What exactly is akka.dispatch.Dispatcher$$anon$1?

I was analyzing a Java heap dump when one of our application servers ran out of memory. I was using Eclipse Memory Analyzer. It reported the following.
One instance of "akka.dispatch.Dispatcher$$anon$1" loaded by
"sun.misc.Launcher$AppClassLoader # 0xc5602128" occupies 675,632,768
(73.50%) bytes.
What do the $$anon and $1 in the class name mean?
What exactly is the object that's taking 73.5% of heap space? What could be the cause?

First anonymous class. Looking at source code I guess it is the Mailbox: https://github.com/akka/akka/blob/master/akka-actor/src/main/scala/akka/dispatch/Dispatcher.scala#L89 (Scala objects created with traits are compiled to anonymous classes)
The tool you are using should be able to tell which objects are directly hold/referenced by Mailbox and are actually using memory. Probably actors are simply not processing messages fast enough.

How to analyze MAT with eclipse

My web application is running in apache tomcat.
The classloader/component org.apache.catalina.loader.WebappClassLoader # 0x7a199fae8 occupies 1,70,86,32,104 (88.08%) bytes.
The memory is accumulated in one instance of java.util.concurrent.ConcurrentHashMap$Segment[] loaded by <system class loader>.
I got this problem while analyzing Heapdump. How to analyze it further ?

You provide very little information so I only can provide very little advice… ;-)
First you need to find out who is using the largest objects (the HashMap in your case). Try to look at the contents of the HashMap so you may find out what it is used for. You should also try to look at where this objects are referenced.
Than you can try to limit its size. Depending on whether it is used by a framework you use or by your own code this can be easy (e.g. configuration change for a frameworks cache), medium (e.g. you need to refactor your own code) or difficult (e.g. it is deeply buried in a library you have no control over).
Often the culprit is not the one you expect: Only because an object instance (in your case the HashMap) accumulates a lot of memory does not mean the "owner" of this object is the root cause of the problem. You might well have to look some levels above or below in the object tree or even in a completely different location. In most cases it is crucial that you know your application very well.
Update: You can try to inspect the contents of a HashMap by right clicking it and selecting Java Collections, Hash Entries. For general objects you can use List objects, with incoming references (to list all objects that reference the selected object) or with outgoing references (to list all object that are referenced by the selected object).
Memory analysis is not an easy task and can require a lot of time, at least if you are not used to it…
If you need further assistance you need to provide more details about your application, the frameworks you use and how the heap looks like in MAT.

Out of memory while processing large number of objects in HashMap/Hashtable/ArrayList

I am working on a project, in which need to load thousands of object's data in HashMap/Hashtable/ArrayList. No issue with small application but it goes out of memory in large application.
Please suggest how to handle this situation?

Why does it need to be in memory? I would say use a random access file.

I am wondering about your requirement here, why is it important to load thousands of objects at the same time? Can you provide more details about it? Perhaps your implementation can be reworked so that you don't need that many objects loaded in memory.

don't read all the data into memory at once, or expand the memory available prior to execution.
You can not increase heap size programmatically.

Either you have to increase the memory or u have to check through the code whether there is some point where application is creating many objects(probably in loops) . If it is there nullify that objects(make sure it will not affect your application flow). Another option try to use a lighter object (say A bean can be made lighter to a string object if you can properly override the toString() method. This will also increase the performance of your application)

You could use a cache system, like ehcache for example. That would give you some control over the "memory" used. There's other cache implementation, ehcache might not suit your needs.

What should i do to get classes via import statement from external jars in Domino Designer?

I work with Domino server and develop java agents. Usually i put external jar files into script library, but periodically get OutOfMemory. I extracted jar's from script library to jvm/lib/ext. I can get classes via ClassLoader, but can't via import statement.
What should i do to get it works via import statement?

Close Lotus Notes client and Designer.
Place the JARs to your local jvm/lib/ext.
Open Lotus Notes Client and Designer.
Now you can import the classes.

You can actually add the jar's to the Java Agent explicitly. Be aware that a Java Agent has no package name by default, and you'll need to quote the correct package name when quoting the package in the import statement. Importing jar's into agents is described here, and for version 8.5 here and here
Also, I suspect that you're not explicitly cleaning up you're java objects. The connection between the JVM and Domino memory heap is "weak". So you have to do you're own garbage collection on Domino objects in order to keep the server's memory clean. It's still black magic to me, but my understanding is that even though agents are supposed to "contain" the session and then release memory at termination, Domino objects not properly recycled can bleed Domino's heap memory. Below are my simple tips for keeping your memory clean:
1/ Keep the session object in a wrapper object. Create a class that has the Domino session object within it, (I called it SessionWrapper). Then declare the SessionWrapper object within the NotesMain method only, don't declare it at the class level. This SessionWrapper class will need to have it's own recycle method that calls Session.recycle(), and you'll need to call it at the end of NotesMain. The reason for this is explained in point #2. If you don't call any other methods within NotesMain, then you don't really need this wrapper. See point #4 about recycling the session.
2/ Pass the SessionWrapper to all methods where you require Domino access. Basically you're passing the session around in this wrapper class as a parameter. This is because we don't want to declare the session object at the class level. This is not ideal, but it will prevent memory leaks caused from keeping the session class at the class level.
3/ Agressive recycling. This has never made much sense to me until I saw this example loop
ViewEntryCollection vec = view.getAllEntries();
ViewEntry ve = vec.getFirstEntry();
While (ve!=null) {
ViewEntry veNext = vec.getNextEntry(ve);
// do stuff
ve.recycle;
ve = null;
ve = veNext;
}
See how the "ve" object is getting recycled and veNext is getting the next object. Basically you need to recycle every object once you're finished working with it, the "getNext" methods doesn't actually recycle the object in the Domino heap memory, if you don't recycle it, it's orphaned and Domino won't clean it up, and the server will eventually run out of memory for Domino objects. Note that I am not recycling veNext. I don't need to because I assign ve => veNext. "ve" will have a link to the same Domino object and I call recycle ve at the end of the loop. From this, you can see that many java objects can point to the same Domino instance object in the back end. If I tried to call veNext.recycle after ve.recycle, I would get an "Object has been removed or recycle" error.
4/ Recylce session at the end of the NotesMain. Make sure you call SessionWrapper.recycle() at the end of NotesMain to ensure you're releasing that bit of memory back to the server.
5/ Ensure that you're providing enough memory for the JVM on the server. See this technote. Also be aware of "HTTPJVMMaxHeapSizeSet=1" which is a strange setting to ensure memory settings "stick".. More about it here.
For many years IBM hasn't provided the internal cleanup tasks of objects that is so deperately needed to eliminate this overhead on developers. You're then required to explicitly deallocate memory which is also a performance hit to, but it's a necessary compromise for stability. These points are by no means exhaustive, but I found that stability with agents and servlets greatly improved when observing these rules.

how can I get the History of an object or trace an Object

I have a requirement, where support in my application a lot of processing is happening, at some point of time an exception occrured, due to an object. Now I would like to know the whole history of that object. I mean whatever happened with that object over the period of time since the application has started.
Is this peeping into this history of Object possible thru anyway using JMX or anything else ?
Thanks

In one word: No
With a few more words:
The JVM does not keep any history on any object past its current state, except for very little information related to garbage collection and perhaps some method call metrics needed for the HotSpot optimizer. Doing otherwise would imply a huge processing and memory overhead. There is also the question of granularity; do you log field changes only? Every method call? Every CPU instruction during a method call? The JVM simply takes the easy way out and does none of the above.
You have to isolate the class and/or specific instance of that object and log any operation that you need on your own. You will probably have to do that manually - I have yet to find a bytecode instrumentation library that would allow me to insert logging code at runtime...
Alternatively, you might be able to use an instrumenting profiler, but be prepared for a huge performance drop when doing that.

That's not possible with standard Java (or any other programming language I'm aware of). You should add sufficient logging to your application, which will allow you to get some idea of what's happened. Also, learn to use your IDE's debugger if you don't already know how.

I generally agree with #thkala and #artbristol (+1 for both).
But you have a requirement and have no choice: you need a solution.
I'd recommend you to try to wrap your objects with dynamic proxies that perform auditing, i.e. write all changes that happen to object.
You can probably use AspectJ for this. The aspect will note what method was called and what are the parameters that were sent. You can also use other, lower level tools, e.g. Javasist or CgLib.

Answer is No.JVM doesn't mainatain the history of object's state.Maximum what you can do you can keep track of states of your object that could be some where in-memory and when you get exception you can serialize that in-memory object and then i think you can do analysis.