We Keep Coding

Java 8 lambda mutable variable capture from method parameter?

I'm using AdoptOpenJDK jdk81212-b04 on Ubuntu Linux, running on Eclipse 4.13. I have a method in Swing that creates a lambda inside a lambda; both probably get called on separate threads. It looks like this (pseudocode):
private SwingAction createAction(final Data payload) {
System.out.println(System.identityHashCode(payload));
return new SwingAction(() -> {
System.out.println(System.identityHashCode(payload));
//do stuff
//show an "are you sure?" modal dialog and get a response
//show a file selection dialog
//when the dialog completes, create a worker and show a status:
showStatusDialogWithWorker(() -> new SwingWorker() {
protected String doInBackground() {
save(payload);
}
});
You can see that the lambdas are several layers deep, and that eventually the "payload" which was captured gets saved to a file, more or less.
But before considering the layers and the threads, let's go directly to the problem:
The first time I call createAction(), the two System.out.println() methods print the exact same hash code, indicating that the captured payload inside the lambda is the same I passed to createAction().
If I later call createAction() with a different payload, the two System.out.println() values printed are different! In particular, the second line printed always indicates the same value that was printed in step 1!!
I can repeat this over and over; the actual payload passed will keep getting a different identity hash code, while the second line printed (inside the lambda) will stay the same! Eventually something will click and suddenly the numbers will be the same again, but then they will diverge with the same behavior.
Is Java somehow caching the lambda, as well as the argument that is going to the lambda? But how is this possible? The payload argument is marked final, and besides, lambda captures have to be effectively final anyway!
Is there a Java 8 bug that doesn't recognize a lambda should not be cached if the captured variable is several lambdas deep?
Is there a Java 8 bug that is caching lambdas and lambda arguments across threads?
Or is there something that I don't understand about lambda capture method arguments versus method local variables?
First Failed Workaround Attempt
Yesterday I thought I could prevent this behavior simply by capturing the method parameter locally on the method stack:
private SwingAction createAction(final Data payload) {
final Data theRealPayload = payload;
System.out.println(System.identityHashCode(theRealPayload));
return new SwingAction(() -> {
System.out.println(System.identityHashCode(theRealPayload));
//do stuff
//show an "are you sure?" modal dialog and get a response
//show a file selection dialog
//when the dialog completes, create a worker and show a status:
showStatusDialogWithWorker(() -> new SwingWorker() {
protected String doInBackground() {
save(theRealPayload);
}
});
With that single line Data theRealPayload = payload, if I henceforth used theRealPayload instead of payload suddenly the bug no longer appeared, and every single time I called createAction(), the two printed lines indicate exactly the same instance of the captured variable.
However today this workaround has stopped working.
Separate Bug Fix Addresses Problem; But Why?
I found a separate bug that was throwing an exception inside showStatusDialogWithWorker(). Basically showStatusDialogWithWorker() is supposed to create the worker (in the passed lambda) and show a status dialog until the worker is finished. There was a bug that would create the worker correctly, but fail to create the dialog, throwing an exception that would bubble up and never get caught. I fixed this bug so that the showStatusDialogWithWorker() successfully shows the dialog when the worker is running and then closes it after the worker finishes. I can now no longer reproduce the lambda capture issue.
But why does something inside showStatusDialogWithWorker() relate to the problem at all? When I was printing out System.identityHashCode() outside and inside the lambda, and the values were differing, this was happening before showStatusDialogWithWorker() was being called, and before the exception was being thrown. Why should a later exception make a difference?
Besides, the fundamental question remains: how is it even possible that a final parameter passed by a method and captured by a lambda could ever change?

how is it even possible that a final parameter passed by a method and captured by a lambda could ever change?
It is not. As you have pointed out, unless there is a bug in the JVM, this cannot happen.
This is very difficult to pin down without a minimal reproducible example. Here are the observations you have made:
The first time I call createAction(), the two System.out.println() methods print the exact same hash code, indicating that the captured payload inside the lambda is the same I passed to createAction().
If I later call createAction() with a different payload, the two System.out.println() values printed are different! In particular, the second line printed always indicates the same value that was printed in step 1!!
One possible explanation that fits the evidence is that the lambda being called the second time is in fact the lambda from the first run, and the lambda that was created by the second run has been discarded. That would give the above observations, and would place the bug inside the code that you have not shown here.
Perhaps you could add some extra logging to record:
a) the ids of any lambdas created inside createAction at creation time (I think you would need to change the lambdas into anon classes that implement the callback interfaces with logging in their constructors)
b) the ids of the lambdas at the time of their invocation
I think that the above logging would be sufficient to prove or disprove my theory.
GL!

I don't find anything wrong with the captured lambdas in your code.
Your workaround doesn't change the local capturing as you just declared a new variable an assigned it to to the same reference.
The problem is most likely on your handling of the SwingAction created object.
I wouldn't be surprised if you find that printing the IdentityHashCode of that returned object where you are using it yields consistent values with your payloads. Or in other words, you may be using a previous reference to SwingAction.
Besides, the fundamental question remains: how is it even possible
that a final parameter passed by a method and captured by a lambda
could ever change?
This shouldn't be possible at the reference level, the variable can't be reassigned. The reference passed may be mutable itself though, but that doesn't apply to this case.

A function providing single object to recursive method

The code is posted for review on review board. My intention is not asking to review the code.
[Maze] : https://codereview.stackexchange.com/questions/33155/maze-code-review
In the above code the function solve does nothing but provide a stack object (reference to object of type stack) which would be used by a code executing recursively.
Since there are so many patterns is there a name for such a function which only assists / or does setup for recursive calls ?
If so any do's / dont's / alternatives ?

I think you did just fine. Every recursive algorithm needs some initial values for it's first step. It's common practice to encapsulate this initial call in another method so the caller doesn't have to bother with those values.
If your initial values would be more complicated to set up, you could encapsulate that in additional methods too. Say your stack needs to have some content instead of being empty. You could do something like this:
public List<Coordinate> solve() {
return getMazePath(0, 0, getInitialStack());
}
This way the solve method stays clear and easy as the entry point of your recursion.

How to cluster an instance with Weka's DBSCAN?

I've been trying to use the DBSCAN clusterer from Weka to cluster instances. From what I understand I should be using the clusterInstance() method for this, but to my surprise, when taking a look at the code of that method, it looks like the implementation ignores the parameter:
/**
* Classifies a given instance.
*
* #param instance The instance to be assigned to a cluster
* #return int The number of the assigned cluster as an integer
* #throws java.lang.Exception If instance could not be clustered
* successfully
*/
public int clusterInstance(Instance instance) throws Exception {
if (processed_InstanceID >= database.size()) processed_InstanceID = 0;
int cnum = (database.getDataObject(Integer.toString(processed_InstanceID++))).getClusterLabel();
if (cnum == DataObject.NOISE)
throw new Exception();
else
return cnum;
}
This doesn't seem right. How is that supposed to work? Is there a different method I should be using for clustering? Do I have to run this method sequentially on all instances, in some specific order, if I want to get any useful information out of it?

This has been reported as a bug - [Wekalist] DBScan - Issue/Bug with "clusterInstance()"-Function.
I'm doing some clustering with the DBScan library. Unfortunately it
seems that there is a bug in the function "clusterInstance()". The
function doesn't return the number of the assigned cluster but only
returns the cluster-number of the first database element (or the
second on the second call, the third on the third call, and so on.)
and NOT the assigned instance.
It simply cannot work because the assigned variable is never used in
the function.
The response reads:
DBScan and Optics are contributions to Weka. It's probably best if you
contact the authors to see if they can suggest a bug fix. The code and
package info (Weka 3.7) has contact information:
http://weka.sourceforge.net/packageMetaData/optics_dbScan/index.html
I'm afraid I am unfamiliar with the DBScan algorithm and the code is quite old now (2004), you might be lucky and find that you are still able to contact the authors at LMU Munich.
I did find numerous copies of it via Google Code Search and GitHub but I could not find an example where it had been fixed. While searching I did notice several other implementations of DBScan that you could examine to work out how this one could be fixed (e.g. ELKI's DBSCAN)
As I have said I am unfamiliar with DBScan but looking at the JavaDocs gave me the impression that actual clustering is invoked by calling buildClusterer(Instances instances). Examining the source code there seems to be much more going on inside the buildClusterer method than the clusterInstance method. OPTICS.java contains a clusterInstance method too and that one just throws an exception. If your are lucky maybe you can get by without a functioning clusterInstance method.
I found an example of Weka's DBScan being used here: DBSCANClustering.java

The example posted by Mark shows well how to use the DBScan class.
The method that does the actual clustering is DBScan.buildClusterer(Instances instances).
The DBScan.clusterInstance(Instance instance) is supposed to return the number of the assigned cluster for a given instance (after you ran the buildClusterer method). But it's true the parameter is actually ignored, so I guess it won't do what it's supposed to do.

As Mark answered, this is obviously a bug. As long as you query about instances in the exact same order in which they were inserted into the clusterer it's okay; but it won't work in any other case.
A co-worker solved this by writing her own version of the DBScan class: essentially identical (copy-pasted), except that she maintains a mapping between instances and cluster labels. This mapping can be produced by iterating over the contents of the database instance. The appropriate cluster for an instance can then be immediately retrieved from that mapping.
Editing this method is also a good opportunity to change the throw new Exception into something more sensible in this context, such as return -1.

Java dead code elimination... is this code at risk of being optimized out?

So, I'm using an API which is a little unfriendly in certain ways. Basically, this API creates a resource which can be fetched later. This resource may or may not still exist when we go to fetch it later.
To fetch the previously created resource, you have to use a result guid like so:
String resultKey = "12345";
PersistedResult r = mFactory.getPersistedResult(resultKey);
Now, the tricky thing here is that getPersistedResult does NOT throw an exception when called with an invalid guid... PersistedResult is a lazy loader and will only fail when one of its methods is called (causing the object to load itself).
So, to try and determine whether or not the resource is valid, I'm doing the following:
PersistedResult r = null;
if (!StringUtils.isEmpty(resultKey)) {
try {
r = mFactory.getPersistedResult(resultKey);
r.getResultCount(); // Triggers exception if result key was invalid.
} catch (Exception e) {
// handle exception
}
}
Is my call to getResultCount at risk of being optimized out because I'm not using the value?
Calling any method on PersistedResult goes to an external database, just in case that matters.
Thanks

The compiler can't assume that getResultCount() has no side-effects -- therefore it can't remove the call.

No, why would it be? Either getResultCount won't be inlined, in which case it's a black box and has to be executed because it can do anything, or it will get inlined, in which case the compiler can see that it could potentially throw an exception, and will perform that action.
The fact that it has a return value doesn't matter. If that was a factor, then any function of any complexity would be at risk of getting optimized out if the caller doesn't check its return value.

Optimizations at runtime (or compile time, same) are not allowed to give you different results than it would be when nothing is optimized (apart from runtime or memory savings). If here your exception is not thrown because of optimization, this is definitively another behavior, and thus would be a bug.
(Note than in multithreaded environments this is a bit relaxed when concerning the relations between different threads.)

No. Because optimize cannot change semantic of code.

Debugging stack data not assigned to a named variable

Is there a way to view stack elements like un-assigned return values or exceptions that not assigned to a local variable? (e.g. throw new ...)
For example, suppose I have code along the lines of:
public String foo(InputStream in) throws IOException {
NastyObj obj = null;
try {
obj = new NastyObj(in);
return (obj.read());
} finally {
if (obj != null) obj.close();
}
}
Is there any way to view the return or exception value without stepping to a higher level frame where it is assigned? This is particularly relevant with exceptions because you often have to step back up through a number of frames to find an actual handler.
I usually use the Eclipse debugging environment, but any answer is appreciated. Also, if this cannot be done, can you explain why? (JVM, JPDA limitation?)

The answer seems to be that both JPDA/JDI and Eclipse are deficient.
I’ve verified the following, but I’m not going to bother posting the code unless someone really needs it.
For the most part, JDI mirrors the structure of the code. Named locals for a given scope can be obtained through the debuggee thread’s current StackFrame. Unscoped locals and method arguments can be obtained through the debuggee thread’s current Method. However, in general where the documentation refers to a LocalVariable, it is a named local.
It IS possible to get the return value of a function, if the function is returning normally and you are using a 1.6 debugging setup (RFE). If the debugger listens for MethodExitEvent, then event.returnValue() gets the value returned after all the method has finished executing. According to the documentation, this event is NOT generated when a method terminates via a thrown exception.
The conclusion is that if you use JDI directly under Java 1.6, you can get the return value of a function before it's assigned to a local, if it returns normally. The Eclipse and Netbeans UIs don't support this. You can't get information about Exceptions that have been thrown but not caught yet.

Couldn't you just catch the IOException (and name it) and then rethrow it? (Don't know Java, but that's what I'd do in C++ and Python.) Of course, this answer is invalid if you can't edit the given code or are inspecting the state right now and need to know what it looks like... But if you can edit the code, that seems like the way to go.