We Keep Coding

OptaPlanner chained variables not considering other problem facts due to a hard constraint

I have a domain set up for chained variables and I'm trying to schedule multiple Tasks to a Shift, very similar to the Call Center example of assigning Calls to Agents. Each Task has a Delivery associated to it and each Delivery has ~7 tasks.
These Tasks are chained together because they are required to be in a certain order and certain tasks are collapsable. The problem is that I have a hard constraint stating that all Tasks on a Delivery must be within the same Shift because it wouldn't make sense for some Tasks on a Delivery to be on multiple Shifts. This constraint is causing Optaplanner to put all Tasks from multiple Deliveries onto 1 Shift and never consider other Shifts, which later down the line will break other hard constraints (ex. there's not enough time to have 10 Deliveries on 1 Shift).
I believe this is happening because during initialization, the first Task gets put onto a Shift and then the hard constraint prevents all other Tasks on a Delivery to try other Shifts since it will be breaking the hard constraint. Removing this constraint fixes this issue and Tasks consider multiple Shifts but now all Tasks on a Delivery are not together on the same Shift which will not work.
I'm not sure how else I could write this constraint or if using a weighted constraint would work either since this constraint must not be broken. I've considered using a custom move that uses a CompositeMove to move all Tasks on a Delivery to a new Shift in 1 Move but am struggling to implement this since I haven't found any resource online that uses CompositeMove with a chained variable.
I'm wondering what else I can try. Thanks.

The reason why you're having this issue is that your data model actually allows a state that should not be allowed. (These extra tasks can be part of different shifts, even though in practice it must not be allowed to happen.) Once OptaPlanner gets you in that state, it will be hard - if not impossible - to get out.
The way I see it, you have several options:
Write move filters that would prevent OptaPlanner from creating this wrong state. That said, move filters are overhead. Still, this is only a part of the solution; to help OptaPlanner actually improve your solution, you may also need to...
Write custom moves instead of OptaPlanner's generic moves. It will be the job of these moves to never ever get the solution into the wrong state. These will be somewhat difficult to write; handling chaining is no fun.
Preferrably though, you should change your data model so that this state is impossible. One idea comes to mind. You can have two levels of tasks - one that keeps the actual chain ("parent tasks"), and the other ("subtasks") that are just pointing to their parents; they don't form a chain, they just reference the parent. If the parent changes its position in the chain, the subtasks still point to the parent - they always "move together".
Changing the data model is the preferred way, because it is the only approach where you do not need to work around invalid states. Every time you need to filter moves or generate complex custom moves, you waste CPU time. If your data model makes certain situations impossible, you do not waste time preventing or fixing them.

Java -> volatile and final: Volatile as flushing-all-memory-content

Take a look on that answer here (1):
https://stackoverflow.com/a/2964277/2182302 (Java Concurrency : Volatile vs final in "cascaded" variables?)
and on my old question here (2):
one java memoryFlushing volatile: A good programdesign?
So as i understand (see (2)) i can use volatile variables as memory barrier/flusher for ALL memory content not only for the referenced one by the volatile keyword.
now the accepted answer in (1) says that it would only flush the memory where the volatile-keyowrd is attached on.
So what is correct now?, and if the flushing-all principle in (2) is correct, why i cant then attach volatile to variables in combination with final?

Neither answer is correct, because you're thinking about it the wrong way. The concept of 'flush the memory' is simply made up. It's nowhere in the Java Virtual Machine Specification. It's just Not A Thing. Yes, many CPU/architectures do work that way, but the JVM does not.
You need to program to the JVM spec. Failure to do so means you write code that works perfectly fine on your machine, every time, and then you upload it to your server and it fails there. This is a horrible scenario: Buggy code, but bugs that cannot ever be trigged by tests. Yowza, those are bad.
So, what is in the JVM spec?
Not the concept of 'flushing'. What it does have, is the concept of HBHA: Happens-Before/Happens-After. Here's how it works:
There is a list of specific interactions which sets up that some line of code is defined to 'happen before' (HB/HA = Happens before/Happens after) some other line. An idea of this list is given below.
For any two lines which have an HBHA relationship, it would be impossible for the HA line to observe any state being such that it appears as if the HB line has not run yet. It's basically saying: HB lines occur before HA lines, except not quite that strong: You cannot observe the opposite (i.e. HB changes variable X, the HA line does not see this change to X, that'd be observing the opposite, that's impossible). Except timing-wise. In reality, HB/HA does not actually mean that lines get executed earlier or later: If you have 2 lines with an HB/HA relationship which have no effect on each other (one writes variable X. The other reads completely different variable Y), the JVM/CPU working together is free to reorder as much as it wants.
For any two lines with no defined HB/HA relationship, the JVM and CPU are free to do whatever it pleases. Including things that just cannot be explained with a simplistic 'flushing' model.
For example:
int a = 0, b = 0;
void thread1() {
a = 10;
b = 20;
}
void thread2() {
System.out.println(b);
System.out.println(a);
}
In the above, no HB/HA relationship has been established between thread 1 modifying the state of a/b, and thread 2 reading them.
Therefore, it is legal for a JVM to print 20 0, even though this cannot be explained with basic flushing notions: It is legal for the JVM to 'flush' b but not a.
It is somewhat unlikely for you to be capable of writing this code and actually observing that 20/0 print on any JVM version or any hardware, but the point is: It is allowed, and some day (or probably, it already exists), some exotic combo of JVM+hardware+OS version+state of the machine combines to actually make this happen, so if your code breaks if this sequence of events occurs, then you wrote a bug.
In effect, if one line mutates state, and another line reads it, and those 2 lines have no HB/HA, you messed up, and you need to fix your bug. Even (especially!) if you can't manage to write a test that actually proves it.
The trick here is that volatile reads do establish HB/HA, and as that is the only mechanism that the JVMS spec has to sync stuff up, yes, this has the effect of guaranteeing that you 'see all changes'. But this is not, at all, a good idea. Especially because the JVMS also says that the hotspot compiler is free to eliminate lines that have no side-effect.
So now we're going to have to get into a debate on whether 'establishes HBHA' is a side-effect. It probably is, but now we get to the rule of optimizations:
Write idiomatic code.
Whenever azul, the openjdk core dev team, etc are looking at improving the considerable optimization chops of the hotspot compiler, they look at real life code. It's like a gigantic pattern matcher: They look for patterns in code and finds ways to optimize them. They don't just write detectors for everything imaginable: They strongly prefer writing optimizers for patterns that commonly show up in real life java code. After all, what possible point is there spending time and effort optimizing a construction that almost no java code actually contains?
This gets us to the fundamental issue with using throw-away volatile reads as a way to establish HB/HA: Nobody does it that way, so the odds that at some point the JVMS is updated (or simply the conflicting rules are 'interpreted' as meaning: Yeah, hotspot can eliminate a pointless read, even if it did establish an HB/HA that is now no longer there) are quite high - you're also far more likely to run into JVM bugs if you do things in unique ways. After all, if you do things in ways that are well trodden, the bug would have been reported and fixed ages ago.
How to establish HB/HA:
The natural rule: Within a single thread, code cannot be observed to run in any way except sequentially, i.e. within one thread, all lines have HB/HA with each other in the obvious fashion.
synchronized blocks: If one thread exits a sync block and then another thread enters one on the same reference, then the sync-block-exit in A Happens-Before the sync-block-enter in B.
volatile reads and writes.
Some exotic stuff, such as: thread.start() happens-before the first line that thread's run() method, or all code in a thread is guaranteed to HB before thread.yield() on that thread finishes. These tend to be obvious.
Thus, to answer the question, is it good programming design?
No, it is not.
Establish HB/HA in the proper ways: Find something appropriate in java.util.concurrent and use it. From a simple lock to a queue to a fork/join pool for the entire job. Alternatively, stop sharing state. Alternatively, share state with mechanisms that are designed for concurrent access in more natural ways than HB/HA is, such as a database (transactions), or a message queue.

Camunda BPM execution and variable scope misunderstanding

I work with the camunda BPM process engine and think it is important to understand some concepts. At the moment I struggle a little bit with the concept of Process Executions and Variable Scopes.
To understand what happens during a process execution I designed the following demo process and marked the activities inside the same execution with the same color. I could do this because I debugged the execution id inside each activity.
I understand most of it. What surprised me is that an input parameter opens a new execution (Task 1.3). Thanks meyerdan for clarification on this.
What I do not understand is that "Task 2.2" is inside the same execution of "Task 2.1". A quote from the camunda documentation about Executions is
Internally, the process engine creates two concurrent executions
inside the process instance, one for each concurrent path of
execution.
So I would have exepcted that Task 2.1 / Task 2.2 and Task 3.1 each live inside an own execution.
Is anyone able to explain this?
My main motivation to understand this is the impact it has on process variable scopes. I did not figure out so far what the Java API methods
VariableScope#getVariable / VariableScope#setVariable
VariableScope#getVariableLocal / VariableScope#setVariableLocal
really do. I first thought that the "Local" variants only refer to the current execution and the other ones only refer to the process instance execution - but that seems to be only half of the truth. These are getters and setters where I miss JavaDoc painfully ;-) Bonus points for also explaining this!
Thanks!
You will find the process in a Maven project with an executable JUnit test on GitHub.

Have a look at Variable Scopes and Variable Visibility
A quote from the documentation (Java Object API) about the setVariable method:
Note that this code sets a variable at the highest possible point in
the hierarchy of variable scopes. This means, if the variable is
already present (whether in this execution or any of its parent
scopes), it is updated. If the variable is not yet present, it is
created in the highest scope, i.e. the process instance. If a variable
is supposed to be set exactly on the provided execution, the local
methods can be used.

Java byecode maniulation to detect potential deadlocks

I've been caught by yet another deadlock in our Java application and started thinking about how to detect potential deadlocks in the future. I had an idea of how to do this, but it seems almost too simple.
I'd like to hear people's views on it.
I plan to run our application for several hours in our test environment, using a typical data set.
I think it would be possible to perform bytecode manipulation on our application such that, whenever it takes a lock (e.g. entering a synchronized block), details of the lock are added to a ThreadLocal list.
I could write an algorithm that, at some later point, compares the lists for all threads and checks if any contain the same pair of locks in opposite order - this would be reported as a deadlock possibility. Again, I would use bytecode manipulation to add this periodic check to my application.
So my question is this: is this idea (a) original and (b) viable?

This is something that we talked about when I took a course in concurrency. I'm not sure if your implementation is original, but the concept of analysis to determine potential deadlock is not unique. There are dynamic analysis tools for Java, such as JCarder. There is also research into some analysis that can be done statically.
Admittedly, it's been a couple of years since I've looked around. I don't think JCarder was the specific tool we talked about (at least, the name doesn't sound familiar, but I couldn't find anything else). But the point is that analysis to detect deadlock isn't an original concept, and I'd start by looking at research that has produced usable tools as a starting point - I would suspect that the algorithms, if not the implementation, are generally available.

I have done something similar to this with Lock by supplying my own implementation.
These days I use the actor model, so there is little need to lock the data (as I have almost no shared mutable data)

In case you didn't know, you can use the Java MX bean to detect deadlocked threads programmatically. This doesn't help you in testing but it will help you at least better detect and recover in production.
ThreadMXBean threadMxBean = ManagementFactory.getThreadMXBean();
long[] deadLockedThreadIds = threadMxBean.findMonitorDeadlockedThreads();
// log the condition or even interrupt threads if necessary
...

That way you can find some deadlocks, but never prove their absence. I'd better develop static checking tool, a kind of bytecode analizer, feeded with annotations for each synchronized method. Annotations should show the place of the annotated method in the resource graph. The task is then to find loops in the graph. Each loop means deadlock.

Java logging across multiple threads

We have a system that uses threading so that it can concurrently handle different bits of functionality in parallel. We would like to find a way to tie all log entries for a particular "transaction" together. Normally, one might use 'threadName' to gather these together, but clearly that fails in a multithreaded situation.
Short of passing a 'transaction key' down through every method call, I can't see a way to tie these together. And passing a key into every single method is just ugly.
Also, we're kind of tied to Java logging, as our system is built on a modified version of it. So, I would be interested in other platforms for examples of what we might try, but switching platforms is highly unlikely.
Does anyone have any suggestions?
Thanks,
Peter
EDIT: Unfortunately, I don't have control over the creation of the threads as that's all handled by a workflow package. Otherwise, the idea of caching the ID once for each thread (on ThreadLocal maybe?) then setting that on the new threads as they are created is a good idea. I may try that anyway.

You could consider creating a globally-accessible Map that maps a Thread's name to its current transaction ID. Upon beginning a new task, generate a GUID for that transaction and have the Thread register itself in the Map. Do the same for any Threads it spawns to perform the same task. Then, when you need to log something, you can simply lookup the transaction ID from the global Map, based on the current Thread's name. (A bit kludgy, but should work)

This is a perfect example for AspectJ crosscuts. If you know the methods that are being called you can put interceptors on them and bind dynamically.
This article will give you several options http://www.ibm.com/developerworks/java/library/j-logging/

However you mentioned that your transaction spans more than one thread, take a look at how log4j cope with binding additional information to current thread with MDC and NDC classes. It uses ThreadLocal as you were advised before, but interesting thing is how log4j injects data into log messages.
//In the code:
MDC.put("RemoteAddress", req.getRemoteAddr());
//In the configuration file, add the following:
%X{RemoteAddress}
Details:
http://onjava.com/pub/a/onjava/2002/08/07/log4j.html?page=3
http://wiki.apache.org/logging-log4j/NDCvsMDC

How about naming your threads to include the transaction ID? Quick and Dirty, admittedly, but it should work (until you need the thread name for something else or you start reusing threads in a thread pool).

If you are logging, then you must have some kind of logger object. You should have a spearate instance in each thread.
add a method to it called setID(String id).
When it is initialized in your thread, set a unique ID using the method.
prepend the set iD to each log entry.

A couple people have suggested answers that have the newly spawned thread somehow knowing what the transaction ID is. Unless I'm missing something, in order to get this ID into the newly spawned thread, I would have to pass it all the way down the line into the method that spawns the thread, which I'd rather not do.
I don't think you need to pass it down, but rather the code responsible for handing work to these threads needs to have the transactionID to pass. Wouldn't the work-assigner have this already?