Greetings fellow developers!
Since SO was almost always helpful with my programming problems, I decided to sign up and give it a shot with my most recent problem. It really is a strange phenomenon that neither I nor my collegue can figure out. I'm sorry I can't provide a working sample, but the project is way to complex to break it down, and specific hardware is needed to run it properly. So I'll try my best to explain it.
The foundation of our project is a native library (a 32-Bit Windows C-DLL in this case) to access project-specific hardware via a Java application (JNA). The purpose is to manage and display the proprietary file-system of the hardware (connected via USB) in a Swing UI. This is a pretty common project configuration for us, since we integrated a lot of native libraries and drivers in Java applications.
Summary: Unit-tests for enumerating devices work fine. A module of the native library allocates memory and fills it with structs, each containing information for a connected device. It is not good practice, but since we do not have any influence on this part we have to go with it. I mapped this struct in Java/JNA, call the native function, copy the struct content to a Java transfer class and print it in the console. Works just fine.
Now if there are UI-operations active while enumerating devices, the native library crashes with an access violation. Even if this UI-operations have nothing to do with the library. The JNA error message shows an EXCEPTION_ACCESS_VIOLATION (0xc0000005), which SO research revealed as invalid/empty memory.
Has anyone ever encountered such problems before? We certainly never did. It took me days to narrow down the error source to this part of the code. Debugging is not easy when native libraries are involved.
Is it possible that there is a JVM memory concurrency problem? Since the native library allocates memory by itself and the JVM doesn't know anything about it - so the JVM tries to allocate memory for new Swing components in already used memory?
Code:
The following snippet is from my unit-test, broken down as far as possible. The intended sequence is obvious: Remove nodes from the root-node, load connected devices and add these devices as new nodes. This code crashes with an access violation, but NOT AT THE NATIVE CALL - it crashes as soon as I access tree components.
public void loadDevices(){
SwingUtilities.invokeLater(new Runnable() {
#Override
public void run() {
rootNode.removeAllChildren();
rootNode.add(new LoadingNode());
tree.expandPath(new TreePath(rootNode));
}
});
final List<Device> devices = lib.loadDevices(); // wrapped native call
SwingUtilities.invokeLater(new Runnable() {
#Override
public void run() {
rootNode.removeAllChildren();
if(!devices.isEmpty()){
for (Device dev : devices ) {
DevNode node = new DevNode(dev);
rootNode.add(node);
}
}
}
});
}
Note: The DevNode does not contain any native data, the content of each native struct is copied to a Java transfer object. The GC should not have issues when trying to move object data, because all unmanaged code is handled locally in the lib#loadDevices() method.
When I remove the calls to the SwingUtilities completely and print the resulting device information to the console instead of creating nodes, this part works fine.
As soon as I try to access the JTree or TreeModel members, the code crashes. It doesn't metter if I do this in a call to SwingUtitilies#invokeLater() or in the same thread.
I know this is a very specific problem that hardly anybody would be interested in (what makes it really hard to search for solutions in SO/Google). But maybe I am lucky and somebody has already encountered this problem.
So long
xander
Edit: Originally this code was wrapped in a worker thread, leading to the same results. This is just a snippet of my unit-test.
Edit 2: It seems I didn't make myself clear enough or forgot to mention something important here, sorry. The access to the tree or its model doesn't necessarily have to do with the native library. Look at the code again: The first call to invokeLater does nothing but remove nodes from the tree. Even when I remove the second call to invokeLater, the native library crashes!
I have struggeled a lot with JTree untill I learned this:
On JTree one should not edit the nodes itself, but use the methods provided on the DefaultTreeModel:
setRoot(TreeNode root)
removeNodeFromParent(MutableTreeNode node)
insertNodeInto(MutableTreeNode newChild, MutableTreeNode parent, int index)
Editing the nodes itself can (and will sooner or later) lead to strange behaviour.
This ofcourse when you are using DefaultTreeModel and MutableTreeNode. I would strongly advise to do this as I have seen to many wrong implementations of TreeModel.
final List<Device> devices = lib.loadDevices();
(by assuming that your code by using JNA is able to returns each node separatelly or collection of List<Device>'s one time if ended) should be called from Worker Thread, e.g. Runnable#Thread or SwingWorker
all output from to Worker Thread add to the DefaultTreeModel directly (or by creating a new MutableTreeNode) should be wrapped into invokeLater
no idea whats happens withut posting an SSCCE (List<Device>'s could be list of USB ports), short, runnable, compilable, just about JTree, is Model, and JNI/JNA ...
Related
I am currently dealing with a OutOfMemoryException in my Tomcat application, probably due to a problem with my connection pooling. Since my codebase is rather large it is quite hard to see if there is any point where a connection is only opened but not closed.
Is there any easy way in Eclipse to show me all methods that call the openConnection()-Method, but not the closeConnection()-Method?
This is a tricky problem. Are you able to replicate this on a test system?
In the following I describe how I would approach solving your problem given the information you have provided.
First verify that this is actually the problem, by using a profiler on a system showing the problem while running. The traditional start is to use the stand alone version of the Netbeans profiler - VisualVMM (https://visualvm.github.io/) - where it is easy to see the memory usage over time, and to enable object allocation insights. I am unsure if the current version of VisualVM can tell you where a given object was allocated, but if it can then that might be the fastest way to identify your problematic code. If not then a trial version of a profiler that can, might be worth looking into.
If the profiler confirms that "calling X.openConnection() and then not calling X.closeConnection()" somewhere is the problem, but you still need to find out exactly where and tools like Findbugs - http://findbugs.sourceforge.net/ - and PMD - https://pmd.github.io/ - does not help you (a bit of effort here goes a long way), then I would suggest changing your code so that you collect the information you need.
If you are actually talking about database connections, then this is not an uncommon problem, so tools exist to help you. There is a rather old question at How to check the Database Connection leakage in Java EE application? but which has suggestions on how to approach this. It also has suggestions on how to catch this with tests.
At this point I would look into code - please consider the following non-compilable pseudocode. Write a revised subclass of X which maintains a global
var openConnections = new ArrayList<Map<X, Exception>>():
where each call to openConnection() also adds a single element with the object and a corresponding exception to a list somewhat like this:
public ... openConnection(...) {
openConnections.add(Map.of(this, new Exception()));
return super.openConnection();
}
and then each call to closeConnection() removes it again.
public ... closeConnection(...) {
// Loop over openConnections, if the current map has this as the first key, then
// delete that entry.
return super.closeConnection();
}
You should then at any time be able to see your open connections in the debugger, and where they were invoked from.
If you have the time I would strongly suggest that you rewrite your code to conform to the AutoCloseable interface instead as it allows you to use try-with-resources to let the compiler help you, and hints to static code analysis tools that this is what you want to do every time. See implements Closeable or implements AutoCloseable for a good starter question with answers.
I am looking for a way to find the name of the program (in my code) that will launch when an operating system tries to open a given file. I will not be launching the application I'm just looking for its name. Ideally the routine I'm looking for/building would take a filename and return a string. I am programming in Java 8 on Eclipse and need my jar file to stay cross platform.
Simplest solution I can find is to use SWT's class 'Program'. Although this assumes that I can correctly identify filetype which is another big can of worms I'm not going to into here.
String ext = extractFileType(filename);
Program p2 = Program.findProgram(ext);
if (p2 != null) programName = p2.toString();
But for a number of reasons I DON'T WANT TO USE the SWT library if at all possible. I'm using Swing and and I really don't want my clients to need to download a different application (jar) dependent on their operating system. I'm well aware that the underlying code is operating system/Window Manager dependent.
Anyone know of any other package besides SWT that already does this? I can't find one. Or similar enough I can strip the results to get what I want? Even if it's only for one platform? I'm experimenting with Apache Tika but I don't see anything helpful there.
Any hints on where to look to start write this myself? I know this entails reading the registry on Windows. I need this code to work on the most recent versions of Windows, and OS X. And eventually Linux but Linux windowing systems are not a priority.
Is there a way to link/load SWT in Eclipse to make the cross-dependent part of using SWT this code a little more lightweight and invisible to the end user? I'm not new to coding but am to using Eclipse.
Here is a quick description of my solution. I did a fair amount of hunting around and I deciding on simply using the JNA library. https://github.com/java-native-access/jna and writing my own native library on a Macintosh to get it to work.
Windows: Fairly straight forward usage of JNA. I'm calling FindExecutable & PathFindExtension from JNA.
public interface MyShell32 extends Shell32 {
MyShell32 INSTANCE = (MyShell32) Native.loadLibrary("shell32", MyShell32.class, W32APIOptions.DEFAULT_OPTIONS);
WinDef.HINSTANCE FindExecutable(String lpFile, String lpDirectory, char[] lpResult);
}
{
...
char[] returnBuffer = new char[WinDef.MAX_PATH];
shell.FindExecutable(filename, null, returnBuffer);
app = Native.toString(returnBuffer);
...
}
PathFindExtention() call is similar but returns a pointer so it's more straight forward.
Macintosh: I tried all sorts of things and finally decided to write my own tiny native library to call in objective C
rtnValue = [[NSWorkspace sharedWorkspace] getInfoForFile:filenameNS
application:&AStr
type:&TStr];
This library is tiny (but I may add to it if I need other native calls) but I need to write a C/C++ shell as well as the Objective C to get it to work. I then call this library JNA. Not that different from writing straight JNI but I found it easier to code.
public interface NSWWraper extends Library {
/** The instance **/
NSWWraper INSTANCE = (NSWWraper) Native.loadLibrary("NSWWraper", NSWWraper.class);
// CP_NSWWraper
Pointer FindFileInfo(String filename);
void FreeMem(Pointer memory);
}
I honestly haven't tested this calling this a large number of files so I don't know how much it slows down my code. JNA calls are supposed to be expensive. It's interesting timing on someone asking for my solution as I'd had to put this on back burner and only got it working on Windows yesterday. I was going to incorporate this into the rest of my project today.
Edited to add. I didn't use JINI because I found it's not being very well supported on a Macintosh anymore and JNA was the better solution for Windows and I had to use it anyway.
This is a question of good practice and a smart solution, I need an advice.
I have an app that (and as far as I can read here in Stackoverflow and by Google search):
The app handles kind of documents and I like it possible to
handle more than one document at the same time. (I am used to Win32 where there is a program segment and one segment of data for each instance but that is obviously not the case in Android/Java.)
I see one instance starting the app from the app storage (the tablet) another opening a Gmail or email with an appended document file, a third instance by opening a file from a File handling app like ES file explorer. And I like them all be possible to be flipped in between. The user might like to read more than one document at a time. (correct me if I use the word instance wrong in the Android/Java environment)
The app is built in a JNI section that contains all the data and logics and a Java Android user interface. (The JNI section is designed to be OS independent for implementations in different OS, has a glue c-file.)
The Android section recreates every time the screen is flipped or instances are flipped between
There is only one JNI instance and that is kept even when the Android Java part is recreated and all Java data is wiped out, right now it shows the last read file in all cases flipping in-between pushing the running app button
There are no problems making different instances within the JNI section as long as it is possible to bind them to each Java instance, with an identity or something that I can use as a parameter in the interchange with the JNI section, but how?
I can't save for instance the FilePathName in each instance to identify the instance in the Java section because it will be wiped when the Java section is recreated.
First question is if I am right in my observations reading Stackoverflow and Googled articles?
Second question, any good suggestions in solving the issue? I need an advice
Is there a possibility to identify the instance in all situations as long it is alive?
Any other possible paths, both to the general issue of separating data for each instance or identifying the instances for the JNI to handle the data for each instance?
Jan
We have similar problems with JNI objects in our application. The problem is that JNI link isn't work as ordinary Java object and has to be relesed explicitly. At the same time we have activity that can be destroyed at any moment by Android.
Our current solution is to store JNI objects on Application level with posibility to manage refereces and drop objects as soon as reference is zero. And also destroyed JNI reference if activity is going to be destroyed forever. So this is similar like you did in previous post.
However if you would like to have your application scalable after some time you might understand that this solution isn't ideal.
The first thing that Android system sometimes temprorary destroys activity to save memory. In your case all JNI objects with documents will still consume memory. So the best solution here is to be able documents on JNI level saves its state to bundle. This is especually important if your documents can be changed by user. In that case by saving state of JNI object in onSaveInstanceState you can destroy your JNI object and recreate in onCreate. However here it is important to analize how much time is required to destroy/create JNI object with saved to bundle document as we have to support quickly activity recreation in some case (portrait/landscape mode for example) with some limited bundle (not more 1Mb). In case of long process this solution might be not good.
Also as you would like to have one task - one document system. You should consider case when you have several activities in one task.
The second item that Android isn't call onDestroy() always. And if you do some save operation here data might be lost sometimes.
Hope this information helps you.
I made something working but I don't know if it is good practice?
I am getting an int-instance-tag from JNI and tagging it on the intent by
public void onCreate(Bundle savedInstanceState) {
....
if (savedInstanceState == null) {
// Creating the JNI task and get the JNI task ID
int iInstance = initProgram(...);
// and store the JNI task ID in the intent
getIntent().putExtra(Intent.EXTRA_TEXT, iInstance);
...
}
...
public void onResume() {
super.onResume();
if (JniManagement.resumeInstance(iTask)) {
...
public void onRestoreInstanceState(Bundle savedInstanceState) {
super.onRestoreInstanceState(savedInstanceState);
// Restore state members from saved instance
iTask =
savedInstanceState.getInt(AndroidApp.STATE_TASK_ID);
}
Then we are talking about the lifespan of a task, the user is flip/flopping between windows/tasks with the home button. The issue is to synchronise the JNI data with the task of Java.
Re-appearing in th else section of if (savedInstanceState == null) { we get the JNI task ID from the intent and synchronise the JNI task with it.
And onDestroy() with if(isFinishing()) freeing the instance set of memory in the JNI.
#Override
public void onDestroy() {
super.onDestroy(); // Always call the superclass
if(isFinishing())
Commands.destroyInstance(getIntent().getExtras().getInt(Intent.EXTRA_TEXT, 0));
// Extinguishing the JNI task started during onCreate()
}
The JNI-side
In the JNI-side all memory used by an instance will be put together in a structure. This structure could be pointed at, in an array of pointers to get the right set of data for the right instance integer. The pointer array is realloced when new instances are created and can go on as long there is memory left for a new instance.
This works actually pretty good, always getting the right data to the right activity/instance. And using a File Manager app starting one activity after another by calling work data files, there will be a stack of activities/instances. When the user is leaving them with the end button are pealed off one by one and its memory is extinguished real smooth. Open a file in a Gmail works fine too the same way, however appears as a different activity by the activity button.
As an old Win32 C-fox I love my pointers and set them in all the methods/functions this feels a bit clumsy (only handle the active window screen data). But the Android OS do not have active overlapping windows.
So just synchronising the JNI this way to the right Java activity/instance is simply working real smooth.
But is it good practice? Are there any other smooth and good looking solutions?
My question is: is it possible (in ANY way) to analyze and modify call stack (both content of frames and stack content) in runtime?
I'm looking for any possibility - low-level, unsafe or internal API, possibility to write C extension, etc. Only constraint: it should be usable in standard runtime, without debugging or profiling mode. This is the point where I'm doing research "is it possible at all?", not "is it good idea?".
I'd like to gather all local data from a frame, store it somewhere, and then remove that frame from stack, with possibility of restoring it later. Effectively that gives us continuations in JVM, and they will allow fast async frameworks (like gevents from python) and generator constructs (like those from python) to come up.
This may look like repeated question, but I've only found questions that were answered with "use Thread.currentThread().getStackTrace()" or "that should be done with debugging tools". There was similiar question to mine, but it was only answered in context of what asking guy wanted to do (work on async computations), while I need more general (java-stack oriented) answer. This question is similiar too, but as before, it is focused on parallelization, and answers are focused on that too.
I repeat: this is research step in process of coming up with new language feature proposal. I don't wanna risk corrupting anything in JVM - I'm looking for possibility, then I'm gonna analyse possible risks and look out for them. I know that manipulating stack by hand is ugly, but so is creating instances with ommiting consrtuctor - and it is basis for objenesis. Dirty hacks may be dirty, but they may help introducing something cool.
PS. I know that Quasar and Lightwolf exist, but, as above, those are concurrency-focused frameworks.
EDIT
Little clarification: I'm looking for something that will be compatible with future JVM and libraries versions. Preferably we're talking about something that is considered stable public API, but if the solution lies in something internal, yet almost standard or becoming standard after being internal (like sun.misc.Unsafe) - that will do too. If it is doable by C-extension using only C JVM API - that's ok. If that is doable with bytecode manipulation - that's ok too (I think that MAY be possible with ASM).
I think there is a way achieving what you want using JVMTI.
Although you cannot directly do what you want (as stated in a comment above), you may instrument/redefine methods (or entire classes) at run time. So you could just define every method to call another method directly to "restore execution context" and as soon as you have the stack you want, redefine them with your original code.
For example: lets say you want to restore a stack where just A called B and B called C.
When A is loaded, change the code to directly call B. As soon as B is loaded, redefine it to directly call C; Call the topmost method (A); As soon as C gets called (which should be very fast now), redefine A and B to their original code.
If there are multiple threads involved and parameter values that must be restored, it gets a little more complicated, but still doable with JVMTI. However, this would then be worth another question ;-).
Hope this helps. Feel free to contact me or comment if you need clarification on anything.
EDIT:
Although I think it IS doable, I also think this is a lot (!!!) of work, especially when you want to restore parameters, local variables, and calling contexts (like this pointers, held locks, ...).
EDIT as requested: Assume the same stack as above (A calling B calling C). Although A, B, and C have arbitrary code inside them, just redfine them like this: void A() { B(); } void B() { C(); } void C() { redefine(); } As soon as you reach the redefine method, redefine all classes with their original code. Then you have the stack you want.
Not sure in this tool, but you can check http://en.wikipedia.org/wiki/GNU_Debugger.
GDB offers extensive facilities for tracing and altering the execution of computer programs. The user can monitor and modify the values of programs' internal variables, and even call functions independently of the program's normal behavior.
I just faced a strange behavior and I can't even say if it is a JavaFX specific problem, Eclipse problem or even where to start fixing first.
But first things first: I'm writing a JavaFX application (Eclipse Kepler/ Java 1.7), containing some extended ListViews. The program takes about 1sec to load.
The extended ListViews look something like that:
public class NewList<T extends Obj> extends ListView<T>{
public NewList(){
// addEventHandler(KeyEvent.KEY_RELEASED, new EventHandler<KeyEvent>(){
//
// #Override
// public void handle(KeyEvent arg0) {
// if(arg0.getText().equals(KeyCode.DELETE)){
// getItems().remove(getSelectionModel().getSelectedItem());
// }
// }
// });
}
}
There are also some MouseEvents (like drag and drop) inside the same constructor, and they are working as expected.
Now I want to add an EventHandler to my extended ListView (see commend in the code above).
The code takes now about 8sec to start aka 8 times longer as normal. To my surprise 90% of the loading time the program is inside the .show() method of my primaryStage (there is just one). After the loading time, the GUI is extreme slow and the cpu usage is on 100%.
Maybe the Handler is implemented wrongly and is doing some strange stuff during the whole time.
I can't find any solution or even comparable problem on the internet....but now the real problem just begun:
When I delete the Handler and run the program again, it will stay slow! So the program is in the same state as before - just still broken! That makes me crazy, because I can't see any logical explanation for that behavior.
What I have already tried:
delete the bin folder inside the repository and build the code again (+reboot)
reset the repository via git
throw half of the code away and started again. At any point it worked again, unfortunately I was not able to reproduce the effect. Then I implemented the Handler again and the problems started from the beginning...
edit: it looks like there goes something wrong during the building process/updating the binarys. I deleted 99% of the code (>5k LOC), then it worked. I copy/pasted the original project back into my workspace and the hole code worked smoothly - even with the Handler on. So I can't imagine a way to produce a minimalistic setup. btw: I'm not allowed to release the complete project folder (university stuff...)
edit2: I' using win7 64bit with java64. other javaFX programs are working (even then grafic intensive oracle samples). I'm not using any JavaFX specific repository setup or builder
Okay, I faced the same problem a few minutes ago and I think I found the source this time.
I'm running my code ALWAYS in debugging mode. To check the EventHandlers condition, I created some breakpoints in Eclipse. I also created a breakpoint at the following line of code (see above for more code)
public void handle(KeyEvent arg0) {
A mouseover over the breakpoint provides the following information:
Multiple markers at this line
Method breakpoint:NewList [Entry] - handle(KeyEvent)
implements:javafx.event.EventHandler .handle
I don't know what is happening here exactly but I think its something like this:
I'm overriding a JavaFX EventHandler Method and the debugger has to check the breakpoint on every single JavaFX interaction/EventHandler call. So Eclipse cant handle the flood of checks in a propper time and all seems to be very slow.
To clear that out: the breakpoint itself does not have to be called at any time, just the existing of it is enough to cause the problems.
Deactivating the breakpoint or running the application in a non-debugging mode will fix all problems. I think it should be a good idea to avoid all entry-breakpoints in any kind of Listener/EventHandler :)