I have set the Kubernetes cronJob to prevent concurrent runs like here using parallelism: 1, concurrencyPolicy: Forbid, and parallelism: 1. However, when I try to create a cronJob manually I am allowed to do that.
$ kubectl get cronjobs
...
$ kubectl create job new-cronjob-1642417446000 --from=cronjob/original-cronjob-name
job.batch/new-cronjob-1642417446000 created
$ kubectl create job new-cronjob-1642417446001 --from=cronjob/original-cronjob-name
job.batch/new-cronjob-1642417446001 created
I was expecting that a new cronjob would not be created. Or it could be created and fail with a state that references the concurrencyPolicy. If the property concurrencyPolicy is part of the CronJob spec, not the PodSpec, it should prevent a new job to be created. Why it does not?
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: cronjob-name
annotations:
argocd.argoproj.io/sync-wave: "1"
spec:
schedule: "0 * * * *"
suspend: false
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 3
concurrencyPolicy: Forbid
jobTemplate:
spec:
parallelism: 1
completions: 1
backoffLimit: 3
template:
spec:
restartPolicy: Never
After reading the official documentation about the kubectl create -f I didn't find a way to prevent that. Is this behavior expected? If it is, I think I should check inside my Docker image (app written in Java) if there is already a cronjob running. How would I do that?
The concurrencyPolicy: Forbidden spec only prevents concurrent pod creations and executions of the same CronJob. It does not apply across separate CronJobs even though they effectively execute the same commands using the same Docker image. As far as Kubernetes is concerned, they are different jobs. If this was Java, what Kubernetes does is if (stringA == stringB) and not if (stringA.equals(stringB)).
If it is, I think I should check inside my Docker image (app written in Java) if there is already a cronjob running. How would I do that?
One way of using that is to use distributed lock mechanism using separate component such as Redis. Here is the link to the guide to utilize Java Redis library redisson for that purpose: https://github.com/redisson/redisson/wiki/8.-distributed-locks-and-synchronizers. Below is code sample taken from that page:
RLock lock = redisson.getLock("myLock");
// wait for lock aquisition up to 100 seconds
// and automatically unlock it after 10 seconds
boolean res = lock.tryLock(100, 10, TimeUnit.SECONDS);
if (res) {
// do operation
} else {
// some other execution is doing it so just chill
}
Related
I'm trying to use ReactiveRedisOperations from spring-data-redis 2.1.8 to do transactions, for example:
WATCH mykey
val = GET mykey
val = val + 1
MULTI
SET mykey $val
EXEC
But I cannot seem to find a way to do this when browsing the docs or the ReactiveRedisOperations. Is this not available in the reactive client, or how can you achieve this?
TL;DR: There's no proper support for Redis Transactions using the Reactive API
The reason lies in the execution model: How Redis executes transactions and how the reactive API is supposed to work.
When using transactions, a connection enters transactional state, then commands are queued and finally executed with EXEC. Executing queued commands with exec makes the execution of the individual commands conditional on the EXEC command.
Consider the following snippet (Lettuce code):
RedisReactiveCommands<String, String> commands = …;
commands.multi().then(commands.set("key", "value")).then(commands.exec());
This sequence shows command invocation in a somewhat linear fashion:
Issue MULTI
Once MULTI completes, issue a SET command
Once SET completes, call EXEC
The caveat is with SET: SET only completes after calling EXEC. So this means we have a forward reference to the exec command. We cannot listen to a command that is going to be executed in the future.
You could apply a workaround:
RedisReactiveCommands<String, String> commands = …
Mono<TransactionResult> tx = commands.multi()
.flatMap(ignore -> {
commands.set("key", "value").doOnNext(…).subscribe();
return commands.exec();
});
The workaround would incorporate command subscription within your code (Attention: This is an anti-pattern in reactive programming). After calling exec(), you get the TransactionResult in return.
Please also note: Although you can retrieve results via Mono<TransactionResult>, the actual SET command also emits its result (see doOnNext(…)).
That being said, it allows us circling back to the actual question: Because these concepts do not work well together, there's no API for transactional use in Spring Data Redis.
Suppose you have Spark + Standalone cluster manager. You opened spark session with some configs and want to launch SomeSparkJob 40 times in parallel with different arguments.
Questions
How to set reties amount on job failures?
How to restart jobs programmatically on failure? This could be useful if jobs failure due lack of resources. Than I can launch one by one all jobs that require extra resources.
How to restart spark application on job failure? This could be useful if job lack resources even when it's launched simultaneously. Than to change cores, CPU etc configs I need to relaunch application in Standalone cluster manager.
My workarounds
1) I pretty sure the 1st point is possible, since it's possible at spark local mode. I just don't know how to do that in standalone mode.
2-3) It's possible to hand listener on spark context like spark.sparkContext().addSparkListener(new SparkListener() {. But seems SparkListener lacks failure callbacks.
Also there is a bunch of methods with very poor documentation. I've never used them, but perhaps they could help to solve my problem.
spark.sparkContext().dagScheduler().runJob();
spark.sparkContext().runJob()
spark.sparkContext().submitJob()
spark.sparkContext().taskScheduler().submitTasks();
spark.sparkContext().dagScheduler().handleJobCancellation();
spark.sparkContext().statusTracker()
You can use SparkLauncher and control the flow.
import org.apache.spark.launcher.SparkLauncher;
public class MyLauncher {
public static void main(String[] args) throws Exception {
Process spark = new SparkLauncher()
.setAppResource("/my/app.jar")
.setMainClass("my.spark.app.Main")
.setMaster("local")
.setConf(SparkLauncher.DRIVER_MEMORY, "2g")
.launch();
spark.waitFor();
}
}
See API for more details.
Since it creates process you can check the Process status and retry e.g. try following:
public boolean isAlive()
If Process is not live start again, see API for more details.
Hoping this gives high level idea of how we can achieve what you mentioned in your question. There could be more ways to do same thing but thought to share this approach.
Cheers !
check your spark.sql.broadcastTimeout and spark.broadcast.blockSize properties, try to increase them .
I am specifying a NodeInitializationAction for Dataproc as follows:
ClusterConfig clusterConfig = new ClusterConfig();
clusterConfig.setGceClusterConfig(...);
clusterConfig.setMasterConfig(...);
clusterConfig.setWorkerConfig(...);
List<NodeInitializationAction> initActions = new ArrayList<>();
NodeInitializationAction action = new NodeInitializationAction();
action.setExecutableFile("gs://mybucket/myExecutableFile");
initActions.add(action);
clusterConfig.setInitializationActions(initActions);
Then later:
Cluster cluster = new Cluster();
cluster.setProjectId("wide-isotope-147019");
cluster.setConfig(clusterConfig);
cluster.setClusterName("cat");
Then finally, I invoke the dataproc.create operation with the cluster. I can see the cluster being created, but when I ssh into the master machine ("cat-m" in us-central1-f), I see no evidence of the script I specified having been copied over or run.
So this leads to my questions:
What should I expect in terms of evidence? (edit: I found the script itself in /etc/google-dataproc/startup-scripts/dataproc-initialization-script-0).
Where does the script get invoked from? I know it runs as the user root, but beyond that, I am not sure where to find it. I did not find it in the root directory.
At what point does the Operation returned from the Create call change from "CREATING" to "RUNNING"? Does this happen before or after the script gets invoked, and does it matter if the exit code of the script is non-zero?
Thanks in advance.
Dataproc makes a number of guarantees about init actions:
each script should be downloaded and stored locally in:
/etc/google-dataproc/startup-scripts/dataproc-initialization-script-0
the output of the script will be captured in a "staging bucket" (either the bucket specified via --bucket option, or a Dataproc auto-generated bucket). Assuming your cluster is named my-cluster, if you describe master instance via gcloud compute instances describe my-cluster-m, the exact location is in dataproc-agent-output-directory metadata key
Cluster may not enter RUNNING state (and Operation may not complete) until all init actions execute on all nodes. If init action exits with non-zero code, or init action exceeds specified timeout, it will be reported as such
similarly if you resize a cluster, we guarantee that new workers do not join cluster until each worker is fully configured in isolation
if you still don't belive me :) inspect Dataproc agent log in /var/log/google-dataproc-agent-0.log and look for entries from BootstrapActionRunner
I want to add jobs from my java code in eclipse to a running cluster of EMR for saving startup time (creating ec2, bootstrapping...).
I know how to run a new cluster from java code but it's terminating after all jobs are done.
RunJobFlowRequest runFlowRequest = new RunJobFlowRequest()
.withName("Some name")
.withInstances(instances)
// .withBootstrapActions(bootstrapActions)
.withJobFlowRole("EMR_EC2_DefaultRole")
.withServiceRole("EMR_DefaultRole")
.withSteps(firstJobStep, secondJobStep, thirdJobStep)
.withLogUri("s3n://path/to/logs");
// Run the jobs
RunJobFlowResult runJobFlowResult = mapReduce
.runJobFlow(runFlowRequest);
String jobFlowId = runJobFlowResult.getJobFlowId();
You have to set KeepJobFlowAliveWhenNoSteps parameter to TRUE, otherwise the cluster will be terminated after executing all the steps. If this property is set, the cluster will continue in waiting state after executing all the steps.
Add .withKeepJobFlowAliveWhenNoSteps(true) to the existing code.
Refer this doc for further details.
We have a simple performance test flow in an application.
We login
Search based on some criteria
repeat searches for different parameters.
We are using Jmeter to do a performance testing. We need to have multiple threads running to test this in a scalable manner.
The way we currently have this arranged is:
-Test Plan
- Thread Group
- Cookie Manager
- Login To application
- Search on param 1
- Search on param 2
- results summary table
- Summary report
So basically we have summary return table and report present on plan level while cookie manager is present on thread group level.
When I run for one thread it runs fine and completes well. When I scale it to multiple threads, as soon as the next thread kicks off, the session for the last thread is invalidated. This causes failures for all the already running threads due to newly spawned thread.
I reached this result with observation:
1. If I run multiple threads, only last thread has got valid responses in result summary tree
2. If I run with 2 threads with ramp up period as 10 seconds, which means each thread gets time to finish itself, then both of them run successfully.
As per my understanding each thread login's into application and since cookie manager is at thread level, the values will be maintained for session id for each thread respectively? But what is causing the override of the session id value between threads?
Any help will be much appreciated.
Copied from jmeter documentation:
The last element is a HTTP Cookie
Manager . A Cookie Manager should be
added to all web tests - otherwise
JMeter will ignore cookies. By adding
it at the Thread Group level, we
ensure that all HTTP requests will
share the same cookies.
From chapter "4.2.2 Logic Controllers" in http://jmeter.apache.org/usermanual/test_plan.html.
EDIT: I guess you should use http://jmeter.apache.org/usermanual/component_reference.html#Simple_Controller to group your requests together with Cookie Manager.
I think that Andrey's answer cannot help. He quotes that each request will use the same cookies BUT according to jmeter manual:
Each JMeter thread has its own "cookie storage area".
As far as I understand the question, you want each thread to share the same session ID cookie. So it seems to me you need to have two thread groups and execute them consecutively. First thread group (with a single thread that executes once only) should login and save the session cookie value to a global parameter (perhaps you need to use jmeter's scripting capabilities).
Then set that cookie in the cookie manager of the second thread group.
Hope that helps.
Try to increase the ramp up time. I ran into the same issue where the ramp up time was about 1 second then I increased it to 3 seconds per thread and it ran fine.
Try this:
Open the user.properties present in the bin folder of JMeter
Edit it and add the following line:
CookieManager.check.cookies=false
Save it and run the script. I hope it will solve your problem.
First change your code to:
jmeter.properties
CookieManager.save.cookies=true
CookieManager.name.prefix=mycookie_
Next, add a HTTP cookie manager in the same thread group as your java sampler.
Then in your java sampler add:
JMeterVariables jmv = JMeterContextService.getContext().getVariables();
Iterator<Map.Entry<String,Object>> it = jmv.getIterator();
while(it.hasNext()){
Map.Entry<String,Object> v = it.next();
System.out.println("name: " + v.getKey() + " value: " + v.getValue());
}