My situation in short: I want to perform parallel tests. Lets say I have 4 credential pairs and 32 tests that will be run in 8 parallel threads (on parallel method level). So I have standart producer-consumer situation but beetween parallel tests.
My idea is to have list of credential objects (on which I will synchronize test threads in "beforeMethod" phase). First 4 threads will get their credentials and remove them from list. All other threads will see empty list and wait ON it.
First test that will finish its execution and will add used credentials back to list on "afterMethod" phase and invoke notify on this list, and so on. But the problem is that I dont have any proper place where I can locate this list of credentials + I want to use simple "mvn test" for starting this process. Any ideas how can I add such synchronization? If there is some similar functionality in TestNG - please tell.
Solved. Main problem was to override TestNG lifecycle + save maven test management. So I started to dig TestNG source code and came to next solution:
Extend IExecutionListener which will store needed data (queue) as a public static final member.
Same interface IExecutionListener participates in TestNG lifecycle: it has lifecycle callbacks onExecutionStart and onExecutionFinish. First one were used to populate queue with credentials from file, and second one is to ease GC work - clean this queue.
Populated queue was used inside of init method annotated with #BeforeMethod. Note: do not store value from queue as a plain member as you'll face overwriting of this member from different threads that performs tests from same class but different tests. Use ThreadLocal to provide thread isolation of this variable.
Special data structure that serves good for my case is BlockingQueue (in LinkedBlockingQueue implementation) with take and put as getter/setter.
Also do not forget to add your custom listener to your xml
Related
Have a look at this process -
When execution arrives at the gateway which is there just after the common process, how can i identify whether the execution is for A or B?
Initially both A and B are started in parallel and will follow a common process after completion.
Can these two parallel executions have different set of variables (both inside and outside the common process) ?
Scoping of variables is explained in the documentation here:
https://docs.camunda.org/manual/latest/user-guide/process-engine/variables/
You can set variables to a specific scope, e.g. in an out data mapping or in a execution listener using a script like: execution.setVariable("aVariable", "aValue","aSubProcess");
A and B have their own task scope and can access the process scope (default).
You can have different set of variables (only) in different scopes. Those you could create via sub process or within the data structure you carry in the parent scope (distinguished by a key).
The fact that you do not synchronize the two branches later again indicates that they may only be in the same process definition because of the common embedded sub process. If your intention is reuse, but with different data scopes, then I would work with call activities instead. They offer globally reusable sub processes and have their own data scope. You could then also split these parallel alternatives into two independent definitions, which only make use of the same call activity.
Transient variables are an option if you want data not to be persistent and only live for the duration of the (Camunda) execution.
The example seems a bit artificial. Suggestions for the best approach are usually easier when discussing concrete business processes.
Please note: Although this question mentions Java, I think it's an OOP/concurrency problem at heart and can probably be answered by anyone with significant programming experience.
So I'm building a ConfigurationLoader that will read Configuration POJO from a remote service and make it available via an API. A few things:
As soon as the ConfigurationLoader is asked for the Configuration the first time, a background thread (worker) will ping the remote service every, say, 30 seconds, for updates and then apply those updates to the Configuration instance; and
If the Configuration is modified, the background worker will be notified of the change and will push the "new" Configuration to the remote service;
Both the ConfigurationLoader and the Configuration must be thread-safe
So when I think "thread safety" the first thing I think of is immutability, which leads me towards excellent projects like Immutables. The problem is that Configuration can't be immutable because we need to be able to change it on the client-side and then let the loader ripple those changes back to the server.
My next thought was to try and make both ConfigurationLoader and Configuration singletons, but the problem is there is that the ConfigurationLoader takes a lot of arguments to instantiate it, and as this excellent answer points out, a singleton that takes arguments in construction is not a true singleton.
// Groovy pseudo-code
class Configuration {
// Immutable? Singleton? Other?
}
class ConfigurationLoader {
// private fields
Configuration configuration
ConfigurationLoader(int fizz, boolean buzz, Foo foo, List<Bar> bars) {
super()
this.fizz = fizz
this.buzz = buzz;
// etc.
}
Configuration loadConfiguration() {
if(configuration == null) {
// Create background worker that will constantly update
// 'configuration'
}
}
}
What are my options here? How do I create both the loader and the config to be thread-safe, where the config is changeable by the client-side (on-demand) or asynchronously by a background worker thread?
The problem is that Configuration can't be immutable because we need to be able to change it
It can still be immutable, you just create a new one for every change ("copy-on-write").
What are my options here?
First thing you'll have to think about: How do you want to react to configuration changes in concurrently running tasks? Basically, you have three options:
Ignore configuration change until the task is done
I.e. some directory your codes writes files to - finish writing the current file to the current target dir, put new files in the new dir. Writing some bytes into /new/path/somefile won't be a good idea if you never created that file. Your best option for this is probably an immutable Configuration object that you store in a field of your task instance (i.e. at task creation - in that case you can also make that field final for clarity). This usually works best if your code is designed as a collection of isolated small tasks.
Pros: Config never changes within a single task, so this is simple to get tread-safe and easy to test.
Cons: Config updates never make it to already running tasks.
Make your tasks check for config changes
I.e. your task regularly sends some data to an email address. Have a central storage for your config (like in your pseudo-code) and re-fetch it in some interval (i.e. between collecting data and sending the mail) from your task code. This usually works best for long-running/permanent tasks.
Pros: Config can change during a task run, but still somewhat simple to get safe - just make sure you have some memory barrier in place for reading the config (make your private configuration field volatile, use an AtomicReference, guard it with a lock, whatever).
Cons: Task code will be harder to test than first option. Config values may still be outdated between checks.
Signal config changes to your tasks
Basically option two, but the other way around. Whenever config changes, interrupt your tasks, handle the interrupt as a "config needs updating" message, continue/restart with new config.
Pros: Config values are never outdated.
Cons: This is the hardest to get right. Some might even argue that you cannot get this right, because interruption should only be used for task abortion. There is only very minor benefits (if at all) over the second option if you place your task's update checks at the right spots. Don't do this if you don't have a good reason to.
You need a singleton to pull this off, but your singleton isn't the immutable thing. Its the threadsafe thing. Make your singleton (Configuration) contain a simple Properties object or something and protect access to this with synchronization. Your Configuration Loader somehow knows of this Configuration singleton and functions to set, under synchronization, new instances of the Properties object when it detects change.
I'm pretty sure Apache Commons Configuration does something like this.
We're seeing OptimisticLockingExceptions in a Camunda process with the following Scenario:
The process consists of one UserTask followed by one Gateway and one ServiceTask. The UserTask executes
runtimeService.setVariable(execId, "object", out);`.
taskService.complete(taskId);
The following ServiceTask uses "object" as input variable (does not modify it) and, upon completion throws said OptimisticLockingException. My problem seems to originate from the fact, that taskService.complete() immediately executes the ServiceTask, prior to flushing the variables set in the UserTask.
I've had another, related issue, which occured, when in one UserTask I executed runtimeService.setVariable(Map<Strong, Boolean>) and tried to access the members of the Map as transition-guards in a gateway following that UserTask.
I've found the following article: http://forums.activiti.org/content/urgenterror-updated-another-transaction-concurrently which seems somehow related to my issue. However, I'm not clear on the question whether this is (un)wanted behaviour and how I can access a DelegateExecution-Object from a UserTask.
After long and cumbersome search we think, we have nailed two issues with camunda which (added together) lead to the Exception from the original question.
Camunda uses equals on serialized objects (represented by byte-arrays) to determine, whether process variables have to be written back to the database. This even happens when variables are only read and not set. As equals is defined by pointer-identity on arrays, a serializabled-Object is never determined "equal" if it has been serialized more than once. We have found, that a single runtimeService.setVariable() leads to four db-updates at the time of completeTask() (One for setVariable itself, the other three for various camunda-internal validation actions). We think this is a bug and will file a bug report to camunda.
Obviously there are two ways to set variables. One way is to use runtimeService.setVariable(), the other is to use delegateTask/delegateExecution.setVariable(). There is some flaw when using both ways at the same time. While we cannot simplify our setup to a simple unit-test, we have identified several components which have to be involved for the Exception to occur:
2.1 We are using a TaskListener to set up some context-variables at the start of Tasks this task-listener used runtimeService.setVariable() instead of delegateTask.setVariable(). After we changed that, the Exception vanished.
2.2 We used (and still use) runtimeService.setVariable() during Task-Execution. After we switched to completeTask(Variables) and omitted the runtimeService.setVariable() calls, the Exception vanished as well. However, this isn't a permanent solution as we have to store process variables during task execution.
2.3 The exception occured only in combination when process variables where read or written by the delegate<X>.getVariable() way (either by our code or implicitly in the camunda implementation of juel-parsing with gateways and serviceTasks or completeTask(HashMap))
Thanks a lot for all your input.
You could consider using an asynchronous continuation on the service task. This will make sure that the service task is executed inside a new transaction / command context.
Consider reading the camunda documentation on transactions and asynchronous continuations.
The DelegateExecution object is meant for providing service task (JavaDelegate) implementations access to process instance variables. It is not meant to be used from a User Task.
I have a thread cleaner in my code that is being created if the DB capacity was exceeded, the capacity is checked on every insertion to the DB. I would like to add more functionality to this cleaner and clean also when number of files exceeding, lets say 10000 files. The new functionality should run scheduled.
I want to be able to clean the DB in 2 ways:
1. On demand.
2. Scheduled, every day on X hour.
Which concurrent java class to use?
How can I make sure that the same thread will be used by the 2 ways above?
Code that would perform cleanup of DB should be completely separated out of scheduling (single responsibility principle), so that you could execute it at any time from some other code.
As for scheduling, I would suggest you looking at Quartz scheduler, and get familiar with CRON so that you could extract it to properties to have possibility to change scheduling trigger without modifying your code.
You should synchronize your code so that no more than one cleanup gets performed at the same time, this should be easy with standard synchronize.
If you wish to make it very simple and don't want to add new dependencies, you can go with standard Java solution: Timer. Timer#scheduleAtFixedRate can provide fixed rate execution. Which means you'll have to add extra code whenever new requirements will show up (e.g., don't schedule at weekend).
We have a system that uses threading so that it can concurrently handle different bits of functionality in parallel. We would like to find a way to tie all log entries for a particular "transaction" together. Normally, one might use 'threadName' to gather these together, but clearly that fails in a multithreaded situation.
Short of passing a 'transaction key' down through every method call, I can't see a way to tie these together. And passing a key into every single method is just ugly.
Also, we're kind of tied to Java logging, as our system is built on a modified version of it. So, I would be interested in other platforms for examples of what we might try, but switching platforms is highly unlikely.
Does anyone have any suggestions?
Thanks,
Peter
EDIT: Unfortunately, I don't have control over the creation of the threads as that's all handled by a workflow package. Otherwise, the idea of caching the ID once for each thread (on ThreadLocal maybe?) then setting that on the new threads as they are created is a good idea. I may try that anyway.
You could consider creating a globally-accessible Map that maps a Thread's name to its current transaction ID. Upon beginning a new task, generate a GUID for that transaction and have the Thread register itself in the Map. Do the same for any Threads it spawns to perform the same task. Then, when you need to log something, you can simply lookup the transaction ID from the global Map, based on the current Thread's name. (A bit kludgy, but should work)
This is a perfect example for AspectJ crosscuts. If you know the methods that are being called you can put interceptors on them and bind dynamically.
This article will give you several options http://www.ibm.com/developerworks/java/library/j-logging/
However you mentioned that your transaction spans more than one thread, take a look at how log4j cope with binding additional information to current thread with MDC and NDC classes. It uses ThreadLocal as you were advised before, but interesting thing is how log4j injects data into log messages.
//In the code:
MDC.put("RemoteAddress", req.getRemoteAddr());
//In the configuration file, add the following:
%X{RemoteAddress}
Details:
http://onjava.com/pub/a/onjava/2002/08/07/log4j.html?page=3
http://wiki.apache.org/logging-log4j/NDCvsMDC
How about naming your threads to include the transaction ID? Quick and Dirty, admittedly, but it should work (until you need the thread name for something else or you start reusing threads in a thread pool).
If you are logging, then you must have some kind of logger object. You should have a spearate instance in each thread.
add a method to it called setID(String id).
When it is initialized in your thread, set a unique ID using the method.
prepend the set iD to each log entry.
A couple people have suggested answers that have the newly spawned thread somehow knowing what the transaction ID is. Unless I'm missing something, in order to get this ID into the newly spawned thread, I would have to pass it all the way down the line into the method that spawns the thread, which I'd rather not do.
I don't think you need to pass it down, but rather the code responsible for handing work to these threads needs to have the transactionID to pass. Wouldn't the work-assigner have this already?