Have a look at this process -
When execution arrives at the gateway which is there just after the common process, how can i identify whether the execution is for A or B?
Initially both A and B are started in parallel and will follow a common process after completion.
Can these two parallel executions have different set of variables (both inside and outside the common process) ?
Scoping of variables is explained in the documentation here:
https://docs.camunda.org/manual/latest/user-guide/process-engine/variables/
You can set variables to a specific scope, e.g. in an out data mapping or in a execution listener using a script like: execution.setVariable("aVariable", "aValue","aSubProcess");
A and B have their own task scope and can access the process scope (default).
You can have different set of variables (only) in different scopes. Those you could create via sub process or within the data structure you carry in the parent scope (distinguished by a key).
The fact that you do not synchronize the two branches later again indicates that they may only be in the same process definition because of the common embedded sub process. If your intention is reuse, but with different data scopes, then I would work with call activities instead. They offer globally reusable sub processes and have their own data scope. You could then also split these parallel alternatives into two independent definitions, which only make use of the same call activity.
Transient variables are an option if you want data not to be persistent and only live for the duration of the (Camunda) execution.
The example seems a bit artificial. Suggestions for the best approach are usually easier when discussing concrete business processes.
Related
The main question is about static fields and singleton instances (for configs, etc.) - are instances of one process running in different threads, as usual servlet requests?
If look deeper - do different #ProcessApplication run in one JVM and will see the same singletons? I don't think so. I know exactly that their classes don't see each other and can have equal names (because of different classLoaders?)
Haven't found any meaningful info on these important themes about Camunda, will appreciate your answers.
I had this same question for one of our scenario while back, and read their Javadoc as mentioned here for a servlet container. Extracting Javadoc,
Invocation Semantics
When the {#link #execute(java.util.concurrent.Callable)} method is
invoked, the servlet process application modifies the context
classloader of the current Thread to the classloader that loaded the
application-provided subclass of this class. This allows,
the process engine to resolve {#link JavaDelegate} implementations using the classloader of the process application
This pretty much explain everything you want to know, since the behavior is very similar to how a web container operates. If you want to know how other container implementations behaves, you can check the respective Javadocs of classes in this package.
To answer your question:
Yes. Thread-safety is required for the shared-resources accessed by JavaDelegate in the same process application. According to the documentation (see below) they create a new instance of delegate each time a task is going to be executed.
Note!
Each time a delegation class referencing activity is executed, a
separate instance of this class will be created. This means that each
time an activity is executed there will be used another instance of
the class to call execute(DelegateExecution).
Therefore, at any time there can be many delegate instances actively running due to the multiple invocation of Process Definitions. So, if they are accessing a shared resource, then they need to be synchronized (thread-safe), because, that shared resources (static or singleton) is local to the process application and loaded by respective application classloader according to above Invocation Semantics Javadoc.
Hope this helps.
This may be a stupid question. However, I would like to know if I have something like this - rdd.mapPartitions(func). Should the logic in func be threadsafe?
Thanks
The short answer is no, it does not have to be thread safe.
The reason for this is that spark divides the data between partitions. It then creates a task for each partition and the function you write would run within that specific partition as a single threaded operation (i.e. no other thread would access the same data).
That said, you have to make sure you do not create thread "unsafety" manually by accessing resources which are not the RDD data. For example, if you create a static object and access that, it might cause issues as multiple tasks might run in the same executor (JVM) and access it as well. That said, you shouldn't be doing something like that to begin with unless you know exactly what you are doing...
Any function passed to the mapPartitions (or any other action or transformation) has to be thread safe. Spark on JVM (this is not necessarily true for guest languages) uses executor threads and doesn't guarantee any isolation between individual tasks.
This is particularly important when you use resources which are not initialized in the function, but passed with the closure like for example objects initialized in the main function, but referenced in the function.
It goes without saying you should not modify any of the arguments unless it is explicitly allowed.
When you do "rdd.mapPartitions(func)", the func may actually execute in a different jvm!!! Thread does not have significance across JVM.
If you are running in local mode, and using global state or thread unsafe functions, the job might work as expected but the behaviours is not defined or supported.
My situation in short: I want to perform parallel tests. Lets say I have 4 credential pairs and 32 tests that will be run in 8 parallel threads (on parallel method level). So I have standart producer-consumer situation but beetween parallel tests.
My idea is to have list of credential objects (on which I will synchronize test threads in "beforeMethod" phase). First 4 threads will get their credentials and remove them from list. All other threads will see empty list and wait ON it.
First test that will finish its execution and will add used credentials back to list on "afterMethod" phase and invoke notify on this list, and so on. But the problem is that I dont have any proper place where I can locate this list of credentials + I want to use simple "mvn test" for starting this process. Any ideas how can I add such synchronization? If there is some similar functionality in TestNG - please tell.
Solved. Main problem was to override TestNG lifecycle + save maven test management. So I started to dig TestNG source code and came to next solution:
Extend IExecutionListener which will store needed data (queue) as a public static final member.
Same interface IExecutionListener participates in TestNG lifecycle: it has lifecycle callbacks onExecutionStart and onExecutionFinish. First one were used to populate queue with credentials from file, and second one is to ease GC work - clean this queue.
Populated queue was used inside of init method annotated with #BeforeMethod. Note: do not store value from queue as a plain member as you'll face overwriting of this member from different threads that performs tests from same class but different tests. Use ThreadLocal to provide thread isolation of this variable.
Special data structure that serves good for my case is BlockingQueue (in LinkedBlockingQueue implementation) with take and put as getter/setter.
Also do not forget to add your custom listener to your xml
I work with the camunda BPM process engine and think it is important to understand some concepts. At the moment I struggle a little bit with the concept of Process Executions and Variable Scopes.
To understand what happens during a process execution I designed the following demo process and marked the activities inside the same execution with the same color. I could do this because I debugged the execution id inside each activity.
I understand most of it. What surprised me is that an input parameter opens a new execution (Task 1.3). Thanks meyerdan for clarification on this.
What I do not understand is that "Task 2.2" is inside the same execution of "Task 2.1". A quote from the camunda documentation about Executions is
Internally, the process engine creates two concurrent executions
inside the process instance, one for each concurrent path of
execution.
So I would have exepcted that Task 2.1 / Task 2.2 and Task 3.1 each live inside an own execution.
Is anyone able to explain this?
My main motivation to understand this is the impact it has on process variable scopes. I did not figure out so far what the Java API methods
VariableScope#getVariable / VariableScope#setVariable
VariableScope#getVariableLocal / VariableScope#setVariableLocal
really do. I first thought that the "Local" variants only refer to the current execution and the other ones only refer to the process instance execution - but that seems to be only half of the truth. These are getters and setters where I miss JavaDoc painfully ;-) Bonus points for also explaining this!
Thanks!
You will find the process in a Maven project with an executable JUnit test on GitHub.
Have a look at Variable Scopes and Variable Visibility
A quote from the documentation (Java Object API) about the setVariable method:
Note that this code sets a variable at the highest possible point in
the hierarchy of variable scopes. This means, if the variable is
already present (whether in this execution or any of its parent
scopes), it is updated. If the variable is not yet present, it is
created in the highest scope, i.e. the process instance. If a variable
is supposed to be set exactly on the provided execution, the local
methods can be used.
We're seeing OptimisticLockingExceptions in a Camunda process with the following Scenario:
The process consists of one UserTask followed by one Gateway and one ServiceTask. The UserTask executes
runtimeService.setVariable(execId, "object", out);`.
taskService.complete(taskId);
The following ServiceTask uses "object" as input variable (does not modify it) and, upon completion throws said OptimisticLockingException. My problem seems to originate from the fact, that taskService.complete() immediately executes the ServiceTask, prior to flushing the variables set in the UserTask.
I've had another, related issue, which occured, when in one UserTask I executed runtimeService.setVariable(Map<Strong, Boolean>) and tried to access the members of the Map as transition-guards in a gateway following that UserTask.
I've found the following article: http://forums.activiti.org/content/urgenterror-updated-another-transaction-concurrently which seems somehow related to my issue. However, I'm not clear on the question whether this is (un)wanted behaviour and how I can access a DelegateExecution-Object from a UserTask.
After long and cumbersome search we think, we have nailed two issues with camunda which (added together) lead to the Exception from the original question.
Camunda uses equals on serialized objects (represented by byte-arrays) to determine, whether process variables have to be written back to the database. This even happens when variables are only read and not set. As equals is defined by pointer-identity on arrays, a serializabled-Object is never determined "equal" if it has been serialized more than once. We have found, that a single runtimeService.setVariable() leads to four db-updates at the time of completeTask() (One for setVariable itself, the other three for various camunda-internal validation actions). We think this is a bug and will file a bug report to camunda.
Obviously there are two ways to set variables. One way is to use runtimeService.setVariable(), the other is to use delegateTask/delegateExecution.setVariable(). There is some flaw when using both ways at the same time. While we cannot simplify our setup to a simple unit-test, we have identified several components which have to be involved for the Exception to occur:
2.1 We are using a TaskListener to set up some context-variables at the start of Tasks this task-listener used runtimeService.setVariable() instead of delegateTask.setVariable(). After we changed that, the Exception vanished.
2.2 We used (and still use) runtimeService.setVariable() during Task-Execution. After we switched to completeTask(Variables) and omitted the runtimeService.setVariable() calls, the Exception vanished as well. However, this isn't a permanent solution as we have to store process variables during task execution.
2.3 The exception occured only in combination when process variables where read or written by the delegate<X>.getVariable() way (either by our code or implicitly in the camunda implementation of juel-parsing with gateways and serviceTasks or completeTask(HashMap))
Thanks a lot for all your input.
You could consider using an asynchronous continuation on the service task. This will make sure that the service task is executed inside a new transaction / command context.
Consider reading the camunda documentation on transactions and asynchronous continuations.
The DelegateExecution object is meant for providing service task (JavaDelegate) implementations access to process instance variables. It is not meant to be used from a User Task.