Java Native Access - GetExtendedTcpTable: Bounds exceeds available space - java

I have implemented the GetExtendedTcpTable() with JNA, but when I use the function, I receive an error:
java.lang.IndexOutOfBoundsException: Bounds exceeds available space : size=28, offset=52
The definition of my function is as follows:
public interface IPHlpAPIExtended extends IPHlpAPI {
IPHlpAPIExtended INSTANCE = Native.load("IPHlpAPI", IPHlpAPIExtended.class, W32APIOptions.DEFAULT_OPTIONS);
int GetExtendedTcpTable( MIB_TCPTABLE_OWNER_PID pTcpTable, IntByReference pdwSize, boolean bOrder, int ulAf, int TableClass, int Reserved );
#Structure.FieldOrder( { "dwNumEntries", "table" } )
public static class MIB_TCPTABLE_OWNER_PID extends Structure {
public WinDef.DWORD dwNumEntries;
public MIB_TCPROW_OWNER_PID[] table = new MIB_TCPROW_OWNER_PID[]{ new MIB_TCPROW_OWNER_PID() };
public MIB_TCPTABLE_OWNER_PID() {
}
public MIB_TCPTABLE_OWNER_PID( Pointer pointer ) {
super( pointer );
this.read();
}
#Override
public void read() {
super.read();
if ( dwNumEntries.intValue() > 0 ) {
table = ( MIB_TCPROW_OWNER_PID[] ) table[0].toArray( dwNumEntries.intValue() );
} else {
table = new MIB_TCPROW_OWNER_PID[]{ new MIB_TCPROW_OWNER_PID() };
}
}
}
}
And:
IPHlpAPIExtended.MIB_TCPTABLE_OWNER_PID pTcpTable = new IPHlpAPIExtended.MIB_TCPTABLE_OWNER_PID();
final IntByReference pdwSize = new IntByReference( 0 );
try {
if ( IPHlpAPIExtended.INSTANCE.GetExtendedTcpTable( pTcpTable, pdwSize, true, AF_INET,
IPHlpAPIExtended.TCP_TABLE_CLASS.TCP_TABLE_OWNER_PID_ALL.ordinal(), 0 ) == ERROR_INSUFFICIENT_BUFFER ) {
IPHlpAPIExtended.MIB_TCPTABLE_OWNER_PID newPTcpTable = new IPHlpAPIExtended.MIB_TCPTABLE_OWNER_PID( pTcpTable.getPointer() );
assertThat(
IPHlpAPIExtended.INSTANCE.GetExtendedTcpTable( newPTcpTable, pdwSize, true, AF_INET,
IPHlpAPIExtended.TCP_TABLE_CLASS.TCP_TABLE_OWNER_PID_ALL.ordinal(), 0 ) ).isEqualTo( NO_ERROR );
for ( int i = 0; i < newPTcpTable.dwNumEntries.intValue(); i++ ) {
final IPHlpAPIExtended.MIB_TCPROW_OWNER_PID row = newPTcpTable.table[i];
System.out.println( row.dwOwningPid.intValue() );
}
} else {
// TODO getLasError()
Assertions.fail( "GetExtendedTcpTable terminate with errors" );
}
} catch ( Throwable t ) {
t.printStackTrace();
}
The dwNumEntries field is correctly populated. But the row:
table = ( MIB_TCPROW_OWNER_PID[] ) table[0].toArray( dwNumEntries.intValue() );
Raises the exception.
EDIT:
This is the mapping for MIB_TCPROW_OWNER_PID :
#Structure.FieldOrder( { "dwState", "dwLocalAddr", "dwLocalPort", "dwRemoteAddr", "dwRemotePort", "dwOwningPid" } )
public static class MIB_TCPROW_OWNER_PID extends Structure {
public WinDef.DWORD dwState;
public WinDef.DWORD dwLocalAddr;
public WinDef.DWORD dwLocalPort;
public WinDef.DWORD dwRemoteAddr;
public WinDef.DWORD dwRemotePort;
public WinDef.DWORD dwOwningPid;
}
EDIT
My solution has been changed to the signature of the function.
int GetExtendedTcpTable( Memory pTcpTable, IntByReference pdwSize, boolean bOrder, int ulAf, int TableClass, int Reserved );
Now pTcpTable it's a more generic Memory parameter, and the code is now the following:
final IPHlpAPIExtended.MIB_TCPTABLE_OWNER_PID pTcpTable = new IPHlpAPIExtended.MIB_TCPTABLE_OWNER_PID();
final IntByReference pdwSize = new IntByReference( 0 );
final Memory mTcpTable = new Memory( pTcpTable.size() );
if ( IPHlpAPIExtended.INSTANCE.GetExtendedTcpTable( mTcpTable, pdwSize, true, AF_INET,
IPHlpAPIExtended.TCP_TABLE_CLASS.TCP_TABLE_OWNER_PID_ALL.ordinal(), 0 ) == ERROR_INSUFFICIENT_BUFFER ) {
final Memory newMTcpTable = new Memory( pdwSize.getValue() );
assertThat( IPHlpAPIExtended.INSTANCE.GetExtendedTcpTable( newMTcpTable, pdwSize, true, AF_INET,
IPHlpAPIExtended.TCP_TABLE_CLASS.TCP_TABLE_OWNER_PID_ALL.ordinal(), 0 ) ).isEqualTo( NO_ERROR );
final IPHlpAPIExtended.MIB_TCPTABLE_OWNER_PID newPTcpTable = new IPHlpAPIExtended.MIB_TCPTABLE_OWNER_PID( newMTcpTable );
System.out.println( "Table Size: " + newPTcpTable.dwNumEntries.intValue() );
for (int i = 0; i < newPTcpTable.dwNumEntries.intValue(); i++) {
final IPHlpAPIExtended.MIB_TCPROW_OWNER_PID item = newPTcpTable.table[i];
System.out.println( "PID: " + item.dwOwningPid.longValue());
}
}

The problem is that you're using Structure#toArray in a way it was not designed. It's intended to work on memory already alloated for the full size of the array (or not allocated at all, in which case it allocates).
But you are using it in a case where, in your mapping of the MIB_TCPTABLE_OWNER_PID structure, you have defined the second element as an array, with a fixed size of 1, and thus allocated memory for it.
public MIB_TCPROW_OWNER_PID[] table = new MIB_TCPROW_OWNER_PID[]{ new MIB_TCPROW_OWNER_PID() };
This isn't wrong in itself, and you have to declare something there, although there's no need to initialize the element with a Java object. Just assign new MIB_TCPROW_OWNER_PID[1].
JNA allocates memory for the size of a structure immediately on assignment, so the result of the way you assigned it is that the table element already is mapped to native memory with a size of exactly 28 (4 bytes for the DWORD and 4*6 bytes for the single MIB_TCPROW_OWNER_PID element.)
This initial allocation of the single element to table works fine, but when you attempt to reassign that variable using Structure#toArray it fails the memory bounds check because the smaller memory has been allocated, and toArray() checks if memory has been allocated before creating new memory.
It may be possible to reassign the variable defining your own array in plain Java, without using Jna's Structure#toArray() and its bounds checks. This code should work in some circumstances if the native side allocates the memory; however, if you are required to allocate memory yourself, it won't work.
#Override
public void read() {
readField("dwNumEntries");
table = new MIB_TCPROW_OWNER_PID[dwNumEntries.intValue()];
super.read();
}
Another approach is to not bother redefining the array (table) but instead map table as a Pointer or a StructureByReference (a more strongly typed pointer). Then instead of the read() override, you would access the native memory pointed to, and create the array on the fly. You could use a getter function like:
public MIB_TCPROW_OWNER_PID[] getTable() {
Pointer[] array = table.getPointerArray(0, dwNumEntries.intValue());
MIB_TCPROW_OWNER_PID[] rows = new MIB_TCPROW_OWNER_PID[array.length];
for (int i=0; i < rows.length; i++) {
rows[i] = new MIB_TCPROW_OWNER_PID(array[i]);
}
return rows;
}
Finally as you have discovered, if you allocate the memory yourself using a Memory object, and return that, you can then pass the pre-allocated memory to the constructor and your code should work.

Related

Solution collection java using gurobi

I am struggling as newby with collecting solutions in java. The problem entails a mixed-integer linear problem in which i have to collect multiple solutions. I use gurobi for the optimalisation and we should get multiple solutions for the decision variables. I am using the Solutioncollection that is given below for this. In the solution class the solution is retrieved and makes it possible to set an objective value. My question is how do i get different values for the decision variables. Till now I have the following code in the optimizer and the solution collection below:
CLSPSolutionCollection collection = new CLSPSolutionCollection ();
CLSPSolution sol = new CLSPSolution (K,T);
model.optimize();
// 6. determine production quantities, setups and overtime ----------------------------------------------------------
if (model.get(GRB.IntAttr.Status) == GRB.Status.OPTIMAL) {
int s = model.get(GRB.IntAttr.SolCount);
model.set(GRB.IntParam.SolutionNumber, s);
sol.setObjectiveValue((int) model.get(GRB.DoubleAttr.ObjNVal)); // produce result
for (int k = 0; k < K; k++) {
for (int t = 0; t < T; t++) {
sol.setProductionQuantity(k, t, q[k][t].get(GRB.DoubleAttr.Xn)); // set decision variable q
sol.setSetupDecision(k, t, gamma[k][t].get(GRB.DoubleAttr.Xn)); // set decision variable Q
sol.setOvertimeDecision(t, beta[t].get(GRB.DoubleAttr.Xn)); // set decision variable Q
//collection.add();
}}
}
// if
// clean up ---------------------------------------------------------------------------------------------------------
model.dispose();
env.dispose();
} catch ( GRBException e){
e.printStackTrace();
} // catcher in the rye
return collection;
// COLLECTION CLASS
public class CLSPSolutionCollection {
private ArrayList<CLSPSolution> solutions;
public CLSPSolutionCollection ( ) {
// TODO
} // constructor
//TODO ..
public void add ( CLSPSolution solution ) {
solutions.add(solution);
} // add
public int size ( ) {
return solutions.size();
} // size
public CLSPSolution get ( int index ) {
return solutions.get(index);
} // get
} // CLSPSolutionCollection

What's the fastest way to initialize a large list of integers?

I need to pre-populate a List with a large number of integer values.
Is there are faster way to do this other than iteration?
Current Code:
class VlanManager {
Queue<Integer> queue = Lists.newLinkedList();
public VlanManager(){
for (int i = 1; i < 4094; i++) {
queue.add(i);
}
}
This code is in the constructor of a class that is created pretty frequently so I'd like this to be as efficient (read:performance not lines of code) as possible
4094 isnt to many items to loop but if it is getting called very frequently you might look at doing something with a static variable.
private static Integer[] theList;
static {
theList = new Integer[4094];
for (int i = 1; i < 4094; i++) {
theList[i-1] = i;
}
}
then make that list a List
Queue<Integer> intQue = new LinkedList(Arrays.asList(theList));
There is a danger of using this method if you have a list of mutable objects. Heres an example of what can happen. Integers are immutable so this doesnt actually apply to your question as it stands
class MyMutableObject {
public int theValue;
}
class Test {
private static MyMutableObject[] theList;
static {
theList = new MyMutableObject[4094];
for (int i = 1; i <= 4094; i++) {
theList[i-1] = new MyMutableObject();
theList[i-1].theValue = i;
}
}
public static void main(String [] args) {
Queue<MyMutableObject> que = new LinkedList(Arrays.asList(theList));
System.out.println(que.peek().theValue); // 1
// your actually modifing the same object as the one in your static list
que.peek().theValue = -100;
Queue<MyMutableObject> que2 = new LinkedList(Arrays.asList(theList));
System.out.println(que2.peek().theValue); // -100
}
}
#Bohemian Has some good points on using a static List instead of an array, while the performance gains are very small they are none the less performance gains. Also because the 'array' is actually only ever being used as a List not an array it should be declared as such.
private static List<Integer> theList;
static {
theList = new ArrayList(4094);
for (Integer i = 0; i < 4094; i++) {
theList.add(i+1);
}
}
The fastest way would be to create a reference list (initialized using an instance block - neatly wrapping it all up in one statement):
private static final List<Integer> LIST = new ArrayList<Integer>(4094) {{
for (int i = 1; i < 4094; i++)
LIST.add(i);
}};
Then in your constructor, initialize the queue using the copy constructor:
Queue<Integer> queue;
public VlanManager(){
queue = new LinkedList<Integer>(LIST);
}
You will not write a faster implementation than what's in the JDK.
I realize this question has already been answered. But I think one important answer is missing: The fastest way to initialize a LinkedList with the values 0..4093 is .. DON'T DO IT AT ALL. Especially if speed is an issue.
What you basically are doing is creating a structure consisting of 4093 Node elements each consiting of two pointers to prev/next element and one pointer to an Integer object. Each of this Nodes must be created (and free). In addition nearly each contained Integer must be created (and freed). 'Nearly' because Java uses a cache for Integer but normally (you can change this with system properties) in the range of -127..127.
This is a lot to do in order to get a simple list of integer and if used intensively gives the GC a lot to do afterwards.
That being said there are numerous possible ways of doing this in a more efficient way. But they depend on what your concrete usage pattern is. Just to name a few:
Use an Array: boolean [] inUse' and set the taken vlan-id totrue` if it's taken
Even better use a BitSet instead of the array
Don't store which vlan is free, but which vlan is taken. I think they tend to be free and so there are much more free as there are taken ones. (this means much less to keep track of).
If you insist on using a LinkedList don't initialize it with your class but have it already initialized. This depends on how much of them you would need. You could keep a pool of them. Or perhaps your codes allows reusage of old lists. (yes, you could sort them after usage.)
Surely there are more...
All of this methods require you to build your own 'Queue' interface. But perhaps this has not to be as rich as Java's. And it really isn't that difficult. If you really use this intensively you could reach perfomance improvement factor 10x-1000x++.
A possible implementation using BitSet with an instantiation cost of nearly nothing could be:
import java.util.BitSet;
import org.testng.annotations.Test;
public class BitSetQueue {
// Represents the values 0..size-1
private final BitSet bitset;
private final int size;
private int current = 0;
private int taken = 0;
public BitSetQueue( int size ){
this.bitset = new BitSet( size );
this.size = size;
this.current = size-1;
}
public int poll(){
// prevent endless loop
if( taken == size ) return -1;
// seek for next free value.
// can be changed according to policy
while( true ){
current = (current+1)%size;
if( ! bitset.get( current ) ){
bitset.set( current );
taken++;
return current;
}
}
}
public boolean free( int num ){
if( bitset.get( num ) ){
bitset.clear( num );
taken--;
return true;
}
return false;
}
#Test
public static void usage(){
BitSetQueue q = new BitSetQueue( 4094 );
for( int i = 0; i < 4094; i++ ){
assertEquals( q.poll(), i );
}
assertEquals( q.poll(), -1 ); // No more available
assertTrue( q.free( 20 ) );
assertTrue( q.free( 51 ) );
assertEquals( q.poll(), 20 );
assertEquals( q.poll(), 51 );
}
}

Accessing a private variable from another class

NOTE: I have looked at the other posts, but I'm still quite lost.
This is the code for a private variable that I have in one class:
private int readFile( String fileName)
{
try
{
File f = new File( fileName );
Scanner input = new Scanner( f );
while( input.hasNextLine( ) )
{
String s = input.nextLine( );
String[ ] sArr = s.split( " " );
String animal = sArr[ 0 ];
double cost = Double.parseDouble(sArr [ 1 ] );
boolean penNeeded = Boolean.parseBoolean( sArr[ 2 ] );
boolean available = Boolean.parseBoolean( sArr[ 3 ] );
Pet p = new Pet( animal, cost, penNeeded, available );
if (count < animalList.length )
{
animalList[count] = p;
count++;
}
}
input.close( );
}
catch( Exception e )
{
System.out.println("Error reading the file:");
System.out.println( e );
e.printStackTrace( );
}
return count;
}
I need to access it in this piece of code located in another class:
static public void processTransaction( String fileName, PettingZoo pz )
{
try
{
// variable should be accessed here
}
catch( Exception e )
{
System.out.println("Error reading the file:");
System.out.println( e );
e.printStackTrace( );
}
}
How can I do this? I think that I need to use a modifier of some sort, but I don't know how which one or how to implement it.
You cannot directly access private variables from another class. That's the whole point of declaring it private. What you need to do is use setter and getter methods in class A, then call the get method from class B.
If you want access to a private variable, you can use getter and setter methods.
Example:
private int variable = 5; //<--- your private variable of class A
// a public method (into the same class A)
// that allows the sharing of your private variable
public int getVariable() {
return variable;
}
Now you can call the method getVariable() from an other class (B) and take the value of the private variable (of class A).
As per your comment you can access the private int readFile(String fileName) method by changing the modified of the method. Change the modifier of the method to public or protected. Also since the accessing method is static you need to change the method as static.
So change it as
public static int readFile( String fileName)
{
}
In the processTransaction method invoke it as,
ClassName.readFile("file_name.extn");

How to pass an array of structures using JNA

I referenced another stackoverflow article "How do I map a pointer to an array of structures in JNA" to come up with the following code to enumerate Windows service dependencies.
Structure and function declarations:
static class SERVICE_STATUS extends Structure {
public int dwServiceType;
public int dwCurrentState;
public int dwControlsAccepted;
public int dwWin32ExitCode;
public int dwServiceSpecificExitCode;
public int dwCheckPoint;
public int dwWaitHint;
public SERVICE_STATUS(){}
}
static class ENUM_SERVICE_STATUS extends Structure {
public ENUM_SERVICE_STATUS(){ }
public WString lpServiceName;
public WString lpDisplayName;
SERVICE_STATUS serviceStatus;
}
boolean EnumDependentServicesW(Pointer hService, int serviceState, ENUM_SERVICE_STATUS serviceStatuses, int size, IntByReference bytesNeeded, IntByReference servicesReturned);
If there is only one service dependency, the following code works:
IntByReference bytesNeeded = new IntByReference();
IntByReference numberOfServices = new IntByReference();
Advapi32.ENUM_SERVICE_STATUS serviceStatus = new Advapi32.ENUM_SERVICE_STATUS();
Advapi32.ENUM_SERVICE_STATUS[] serviceStatuses = (Advapi32.ENUM_SERVICE_STATUS[]) serviceStatus.toArray(1);
if (!advapi32.EnumDependentServicesW(serviceHandle, Advapi32.SERVICE_ACTIVE, null, 0, bytesNeeded, numberOfServices)) {
if (advapi32.EnumDependentServicesW (serviceHandle, Advapi32.SERVICE_ACTIVE, serviceStatuses[0], bytesNeeded.getValue(), bytesNeeded, numberOfServices)) {
for(int i = numberOfServices.getValue() - 1; i >= 0; i--){
logger.debug("Service Name: " + serviceStatuses[i].lpServiceName.toString());
}
}
If there are 2 service dependencies, I get a NullPointerException for lpServiceName in the logger.debug call:
IntByReference bytesNeeded = new IntByReference();
IntByReference numberOfServices = new IntByReference();
Advapi32.ENUM_SERVICE_STATUS serviceStatus = new Advapi32.ENUM_SERVICE_STATUS();
Advapi32.ENUM_SERVICE_STATUS[] serviceStatuses = (Advapi32.ENUM_SERVICE_STATUS[]) serviceStatus.toArray(2);
if (!advapi32.EnumDependentServicesW(serviceHandle, Advapi32.SERVICE_ACTIVE, null, 0, bytesNeeded, numberOfServices)) {
if (advapi32.EnumDependentServicesW (serviceHandle, Advapi32.SERVICE_ACTIVE, serviceStatuses[0], bytesNeeded.getValue(), bytesNeeded, numberOfServices)) {
for(int i = numberOfServices.getValue() - 1; i >= 0; i--){
logger.debug("Service Name: " + serviceStatuses[i].lpServiceName.toString());
}
}
The numberOfServices value for the code above is 2, as expected. I'm trying to pass the structure array instead of a pointer because I want JNA to do the memory synching. How should I be passing/using the array of structures?
According to the docs for EnumDependentServices,
lpServices [out, optional]
A pointer to an array of
ENUM_SERVICE_STATUS structures that receives the name and service
status information for each dependent service in the database. The
buffer must be large enough to hold the structures, plus the strings
to which their members point.
You're pretty much ignoring the required buffer size as reported by bytesNeeded. You should use the bytesNeeded value to create a Memory instance of the requisite size, then use that Memory instance to create a new ENUM_SERVICE_STATUS instance, instead of creating the structure independently of the required buffer size.

How can I measure thread stack depth?

I have a 32-bit Java service with scalability problems: with high user count we run out of memory because of excessive thread count. In the long term, I plan to switch to 64-bit and to reduce the threads-per-user ratio. In the short term, I'd like to reduce the stack size (-Xss, -XX:ThreadStackSize) to get some more headroom. But this is risky because if I make it too small, I'm going to get StackOverflowErrors.
How can I measure the average and maximum stack size for my application to guide my decision for an optimal -Xss value? I'm interested in two possible approaches:
Measuring a running JVM during integration testing. What profiling tools will report max stack depth?
Static analysis of the application looking for deep call hierarchies. Reflection in dependency injection makes it unlikely that this would work.
Update: I know the long-term right way to fix this problem. Please focus on the question I've asked: how do I measure stack depth?
Update 2: I got a nice answer on a related question specifically about JProfiler: Can JProfiler measure stack depth? (I posted the separate question as per JProfiler's community support recommendations)
You can get an idea of the stack depth with something like an aspect that can be woven to your code (load time weaver to allow advising all loaded code except system class loader). The aspect would work around all executed code and would be able to note when you are calling a method and when you return. You can use this to capture most of your stack usage (you'll miss anything loaded from the system class loader, e.g. java.*). While not perfect, it avoids having to change your code to gather StackTraceElement[] at sample points and also gets you into non-jdk code that you might not have written.
For example (aspectj):
public aspect CallStackAdvice {
pointcut allMethods() : execution(* *(..)) && !within(CallStackLog);
Object around(): allMethods(){
String called = thisJoinPoint.getSignature ().toLongString ();
CallStackLog.calling ( called );
try {
return proceed();
} finally {
CallStackLog.exiting ( called );
}
}
}
public class CallStackLog {
private CallStackLog () {}
private static ThreadLocal<ArrayDeque<String>> curStack =
new ThreadLocal<ArrayDeque<String>> () {
#Override
protected ArrayDeque<String> initialValue () {
return new ArrayDeque<String> ();
}
};
private static ThreadLocal<Boolean> ascending =
new ThreadLocal<Boolean> () {
#Override
protected Boolean initialValue () {
return true;
}
};
private static ConcurrentHashMap<Integer, ArrayDeque<String>> stacks =
new ConcurrentHashMap<Integer, ArrayDeque<String>> ();
public static void calling ( String signature ) {
ascending.set ( true );
curStack.get ().push ( signature.intern () );
}
public static void exiting ( String signature ) {
ArrayDeque<String> cur = curStack.get ();
if ( ascending.get () ) {
ArrayDeque<String> clon = cur.clone ();
stacks.put ( hash ( clon ), clon );
}
cur.pop ();
ascending.set ( false );
}
public static Integer hash ( ArrayDeque<String> a ) {
//simplistic and wrong but ok for example
int h = 0;
for ( String s : a ) {
h += ( 31 * s.hashCode () );
}
return h;
}
public static void dumpStacks(){
//implement something to print or retrieve or use stacks
}
}
And a sample stack might be like:
net.sourceforge.jtds.jdbc.TdsCore net.sourceforge.jtds.jdbc.JtdsStatement.getTds()
public boolean net.sourceforge.jtds.jdbc.JtdsResultSet.next()
public void net.sourceforge.jtds.jdbc.JtdsResultSet.close()
public java.sql.Connection net.sourceforge.jtds.jdbc.Driver.connect(java.lang.String, java.util.Properties)
public void phil.RandomStackGen.MyRunnable.run()
Very slow and has its own memory issues but can be workable to get you the stack information you need.
You can then use the max_stack and max_locals for each method in your stack traces to compute a frame size (see class file format) for the method. Based on the vm spec I believe this should be (max_stack+max_locals)*4bytes for the max frame size for a method (long/double occupy two entries on the operand stack/local vars and is accounted for in max_stack and max_locals).
You can easily javap the classes of interest and see the frame values if you don't have that much in your call stacks. And something like asm provides you with some easy tools to use to do this on a larger scale.
Once you have this computed, you need to estimate additional stack frames for JDK classes that might be called by you at your max stack points and add that to your stack sizes. It wont be perfect but it ought to get you a decent starting point for -Xss tuning without hacking around the JVM/JDK.
One other note: I don't know what JIT/OSR does to frame sizes or stack requirements so do be aware that you may have different impacts from -Xss tuning on a cold vs. warm JVM.
EDIT had a few hours of down time and threw together another approach. This is a java agent that will instrument methods to keep track of a max stack frame size and stack depth. This will be able to instrument most of the jdk classes along with your other code and libraries, giving you better results than the aspect weaver. You need asm v4 for this to work. It was more for the fun of it so file this under plinking java for fun, not profit.
First, make something to track the stack frame size and depth:
package phil.agent;
public class MaxStackLog {
private static ThreadLocal<Integer> curStackSize =
new ThreadLocal<Integer> () {
#Override
protected Integer initialValue () {
return 0;
}
};
private static ThreadLocal<Integer> curStackDepth =
new ThreadLocal<Integer> () {
#Override
protected Integer initialValue () {
return 0;
}
};
private static ThreadLocal<Boolean> ascending =
new ThreadLocal<Boolean> () {
#Override
protected Boolean initialValue () {
return true;
}
};
private static ConcurrentHashMap<Long, Integer> maxSizes =
new ConcurrentHashMap<Long, Integer> ();
private static ConcurrentHashMap<Long, Integer> maxDepth =
new ConcurrentHashMap<Long, Integer> ();
private MaxStackLog () { }
public static void enter ( int frameSize ) {
ascending.set ( true );
curStackSize.set ( curStackSize.get () + frameSize );
curStackDepth.set ( curStackDepth.get () + 1 );
}
public static void exit ( int frameSize ) {
int cur = curStackSize.get ();
int curDepth = curStackDepth.get ();
if ( ascending.get () ) {
long id = Thread.currentThread ().getId ();
Integer max = maxSizes.get ( id );
if ( max == null || cur > max ) {
maxSizes.put ( id, cur );
}
max = maxDepth.get ( id );
if ( max == null || curDepth > max ) {
maxDepth.put ( id, curDepth );
}
}
ascending.set ( false );
curStackSize.set ( cur - frameSize );
curStackDepth.set ( curDepth - 1 );
}
public static void dumpMax () {
int max = 0;
for ( int i : maxSizes.values () ) {
max = Math.max ( i, max );
}
System.out.println ( "Max stack frame size accummulated: " + max );
max = 0;
for ( int i : maxDepth.values () ) {
max = Math.max ( i, max );
}
System.out.println ( "Max stack depth: " + max );
}
}
Next, make the java agent:
package phil.agent;
public class Agent {
public static void premain ( String agentArguments, Instrumentation ins ) {
try {
ins.appendToBootstrapClassLoaderSearch (
new JarFile (
new File ( "path/to/Agent.jar" ) ) );
} catch ( IOException e ) {
e.printStackTrace ();
}
ins.addTransformer ( new Transformer (), true );
Class<?>[] classes = ins.getAllLoadedClasses ();
int len = classes.length;
for ( int i = 0; i < len; i++ ) {
Class<?> clazz = classes[i];
String name = clazz != null ? clazz.getCanonicalName () : null;
try {
if ( name != null && !clazz.isArray () && !clazz.isPrimitive ()
&& !clazz.isInterface ()
&& !name.equals ( "java.lang.Long" )
&& !name.equals ( "java.lang.Boolean" )
&& !name.equals ( "java.lang.Integer" )
&& !name.equals ( "java.lang.Double" )
&& !name.equals ( "java.lang.Float" )
&& !name.equals ( "java.lang.Number" )
&& !name.equals ( "java.lang.Class" )
&& !name.equals ( "java.lang.Byte" )
&& !name.equals ( "java.lang.Void" )
&& !name.equals ( "java.lang.Short" )
&& !name.equals ( "java.lang.System" )
&& !name.equals ( "java.lang.Runtime" )
&& !name.equals ( "java.lang.Compiler" )
&& !name.equals ( "java.lang.StackTraceElement" )
&& !name.startsWith ( "java.lang.ThreadLocal" )
&& !name.startsWith ( "sun." )
&& !name.startsWith ( "java.security." )
&& !name.startsWith ( "java.lang.ref." )
&& !name.startsWith ( "java.lang.ClassLoader" )
&& !name.startsWith ( "java.util.concurrent.atomic" )
&& !name.startsWith ( "java.util.concurrent.ConcurrentHashMap" )
&& !name.startsWith ( "java.util.concurrent.locks." )
&& !name.startsWith ( "phil.agent." ) ) {
ins.retransformClasses ( clazz );
}
} catch ( Throwable e ) {
System.err.println ( "Cant modify: " + name );
}
}
Runtime.getRuntime ().addShutdownHook ( new Thread () {
#Override
public void run () {
MaxStackLog.dumpMax ();
}
} );
}
}
The agent class has the premain hook for instrumentation. In that hook, it adds a class transformer that instruments in the stack frame size tracking. It also adds the agent to the boot class loader so that it can process jdk classes, too. To do that, we need to retransform anything that might be loaded already, like String.class. But, we have to exclude a variety of things that are used by the agent or the stack logging which lead to infinite loops or other problems (some of that was found by trial and error). Finally, the agent adds a shutdown hook to dump the results to stdout.
public class Transformer implements ClassFileTransformer {
#Override
public byte[] transform ( ClassLoader loader,
String className, Class<?> classBeingRedefined,
ProtectionDomain protectionDomain, byte[] classfileBuffer )
throws IllegalClassFormatException {
if ( className.startsWith ( "phil/agent" ) ) {
return classfileBuffer;
}
byte[] result = classfileBuffer;
ClassReader reader = new ClassReader ( classfileBuffer );
MaxStackClassVisitor maxCv = new MaxStackClassVisitor ( null );
reader.accept ( maxCv, ClassReader.SKIP_DEBUG );
ClassWriter writer = new ClassWriter ( ClassWriter.COMPUTE_FRAMES );
ClassVisitor visitor =
new CallStackClassVisitor ( writer, maxCv.frameMap, className );
reader.accept ( visitor, ClassReader.SKIP_DEBUG );
result = writer.toByteArray ();
return result;
}
}
The transformer drives two separate transformations - one to figure out the max stack frame size for each method and one to instrument the method for recording. It might be doable in a single pass but I didn't want to use the ASM tree API or spend more time figuring it out.
public class MaxStackClassVisitor extends ClassVisitor {
Map<String, Integer> frameMap = new HashMap<String, Integer> ();
public MaxStackClassVisitor ( ClassVisitor v ) {
super ( Opcodes.ASM4, v );
}
#Override
public MethodVisitor visitMethod ( int access, String name,
String desc, String signature,
String[] exceptions ) {
return new MaxStackMethodVisitor (
super.visitMethod ( access, name, desc, signature, exceptions ),
this, ( access + name + desc + signature ) );
}
}
public class MaxStackMethodVisitor extends MethodVisitor {
final MaxStackClassVisitor cv;
final String name;
public MaxStackMethodVisitor ( MethodVisitor mv,
MaxStackClassVisitor cv, String name ) {
super ( Opcodes.ASM4, mv );
this.cv = cv;
this.name = name;
}
#Override
public void visitMaxs ( int maxStack, int maxLocals ) {
cv.frameMap.put ( name, ( maxStack + maxLocals ) * 4 );
super.visitMaxs ( maxStack, maxLocals );
}
}
The MaxStack*Visitor classes handle figuring out the max stack frame size.
public class CallStackClassVisitor extends ClassVisitor {
final Map<String, Integer> frameSizes;
final String className;
public CallStackClassVisitor ( ClassVisitor v,
Map<String, Integer> frameSizes, String className ) {
super ( Opcodes.ASM4, v );
this.frameSizes = frameSizes;
this.className = className;
}
#Override
public MethodVisitor visitMethod ( int access, String name,
String desc, String signature, String[] exceptions ) {
MethodVisitor m = super.visitMethod ( access, name, desc,
signature, exceptions );
return new CallStackMethodVisitor ( m,
frameSizes.get ( access + name + desc + signature ) );
}
}
public class CallStackMethodVisitor extends MethodVisitor {
final int size;
public CallStackMethodVisitor ( MethodVisitor mv, int size ) {
super ( Opcodes.ASM4, mv );
this.size = size;
}
#Override
public void visitCode () {
visitIntInsn ( Opcodes.SIPUSH, size );
visitMethodInsn ( Opcodes.INVOKESTATIC, "phil/agent/MaxStackLog",
"enter", "(I)V" );
super.visitCode ();
}
#Override
public void visitInsn ( int inst ) {
switch ( inst ) {
case Opcodes.ARETURN:
case Opcodes.DRETURN:
case Opcodes.FRETURN:
case Opcodes.IRETURN:
case Opcodes.LRETURN:
case Opcodes.RETURN:
case Opcodes.ATHROW:
visitIntInsn ( Opcodes.SIPUSH, size );
visitMethodInsn ( Opcodes.INVOKESTATIC,
"phil/agent/MaxStackLog", "exit", "(I)V" );
break;
default:
break;
}
super.visitInsn ( inst );
}
}
The CallStack*Visitor classes handle instrumenting methods with code to call the stack frame logging.
And then you need a MANIFEST.MF for the Agent.jar:
Manifest-Version: 1.0
Premain-Class: phil.agent.Agent
Boot-Class-Path: asm-all-4.0.jar
Can-Retransform-Classes: true
Finally, add the following to your java command line for the program you want to instrument:
-javaagent:path/to/Agent.jar
You will also need to have the asm-all-4.0.jar in the same directory as the Agent.jar (or change Boot-Class-Path in the manifest to reference the location).
A sample output might be:
Max stack frame size accummulated: 44140
Max stack depth: 1004
This is all a bit crude but works for me to get going.
Note: the stack frame size isn't a total stack size (still don't really know how to get that one). In practice, there are a variety of overheads for the thread stack. I found that I usually needed between 2 and 3 times the reported stack max frame size as a -Xss value. Oh, and be sure to do the -Xss tuning without the agent loaded as it adds to your stack size requirements.
I would reduce the -Xss setting in a test environment until you see a problem. Then add some head room.
Reducing your heap size would give your application more space for thread stacks.
Just switching to a 64-bit OS could give your application more memory as most 32-bit OSes only allow about 1.5 GB for each application, however a 32-bit application on a 64-bit OS can use up to 3-3.5 GB depending on the OS.
There is no readily usable tooling in the Java VM to query the stack depth in bytes. But you can get there. Here are some pointers:
Exceptions contain arrays of stack frames which gives you the methods which were called.
For each method, you can find the Code attribute in the .class file. This attribute contains the frame size per method in the field max_stack.
So what you need is a tool that compiles a HashMap which contains method name + file name + line number as keys and the value max_stack as values. Create an Throwable, fetch the stack frames from it with getStackTrace() and then iterate over the StackTraceElements.
Note:
Each entry on the operand stack can hold a value of any Java virtual machine type, including a value of type long or type double.
So each stack entry is probably 64bits, so you need to multiply max_stack with 8 to get bytes.

Categories