I've an HTTP triggered azure function (Java) that performs a series of action. In tmy application UI, I've a button which triggers this function to initiate the task.
Everything works as expected.
Now I've to perform this operation in user defined schedules. That is from the UI, user can specify the interval (say every 3 Hrs) at which the function need to be executed. As this schedule will be custom and dynamic I cannot rely on the timer-triggered azure functions. Also the same function need to be executed at different intervals with different input parameters.
How can I dynamically create schedules and invoke the azure function on the scheduled time? Does Azure have an option to run the function with specific events something like (AWS cloud watch rule + lambda invocation)?
EDIT: Its different from the suggested question as it changes the schedule of an existing function.And I think configuring a new schedule will break the previously configured schedules for the function. I want to run the same function in different schedules as per the user configuration and should not break any of the previous schedules set for the function.
You can have a try to modify the function.json, change the cron expression in function.json. Please refer to the steps below:
Use Kudu API to change function.json https://github.com/projectkudu/kudu/wiki/REST-API
PUT https://{functionAppName}.scm.azurewebsites.net/api/vfs/{pathToFunction.json}, Headers: If-Match: "*", Body: new function.json content.
Then send request to apply changes.
POST https://{functionAppName}.scm.azurewebsites.net/api/functions/synctriggers
You can use a durable function for this, applying the Monitor Pattern (shamelessly copied from this MSDN documentation).
This orchestration function sets a dynamic timer trigger using context.CreateTimer.
Code is in C#, but hopefully there is something you can use here.
[FunctionName("MonitorJobStatus")]
public static async Task Run(
[OrchestrationTrigger] IDurableOrchestrationContext context)
{
int jobId = GetJobId();
int pollingInterval = context.GetInput<int>();
DateTime expiryTime = GetExpiryTime();
while (context.CurrentUtcDateTime < expiryTime)
{
var jobStatus = await context.CallActivityAsync<string>("GetJobStatus", jobId);
if (jobStatus == "Completed")
{
// Perform an action when a condition is met.
await context.CallActivityAsync("SendAlert", machineId);
break;
}
// Orchestration sleeps until this time.
var nextCheck = context.CurrentUtcDateTime.AddSeconds(pollingInterval);
await context.CreateTimer(nextCheck, CancellationToken.None);
}
// Perform more work here, or let the orchestration end.
}
Related
So I have a java app that calls 2 APIs.
Call an API to get a request a file to be generated
Call the second API to get the file.
The first API returns the credentials to get the file. The Second one returns the file but it may take a few seconds or minutes to be generated. What is the best way to account for the time delay between asking the file to be generated and the file being available to pull? Some retry logic should be necessary but on the initial call it always returns a 4xx HTTP response. What's the best way to make this api call maybe there's a better way than using RestTemplate to sequentially call the 2 apis? I thought of adding a short time delay before the 2nd call but I was wondering if there is a better library or async method I can use that's more efficient.
I'd appreciate any 2 cents thanks!
If the author of these two apis is a partner of yours, I think there's a better way, like, the file generator call a callback api of yours and then you call the second API to get the file. And as a supplement, considering unexpected exceptions during above process, a retry schedule to fetch the missed file is probably necessary.
But if you just want to implement the retry and async code more gracefully, the following is my idea
//call the first API
ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor();
CompletableFuture<File> completionFuture = new CompletableFuture<>();
final ScheduledFuture<?> checkFuture = executor.scheduleAtFixedRate(() -> {
//Call the second API to get the file
//if("have got the file successfully") {
// completionFuture.complete(the file);
//} else {
// //do nothing
//}
}, 5, 1, TimeUnit.SECONDS);//set a reasonable schedule policy
completionFuture.whenComplete((file, thrown) -> {
//do something when the retry schedule complete
checkFuture.cancel(false);
});
completionFuture.orTimeout(10, TimeUnit.SECONDS);//setting a timeout policy is probably necessary
I have my RabbitListener witch received results of calculations. This is the lot of data so i batched it and next send a command to axon commandGateway. I wonder if it is possible for the listener to wait a few seconds before sending next command. I note that i can't use threadSleep here. It could be work something like this method for every batch
private void myMethod() {
commandGateway.send(batchedData)
//wait 10seconds
What is the reason for having a waiting period here?
If the reason is to wait before Axon Framework has processed all data present in the command, you can use commandGateway.sendAndWait(Object command, ...) instead. This will make the current thread wait for the command to have been executed.
If it is a mechanism to batch the data, I would suggest keeping an in-memory List to queue the items, and then sending a command every 10 seconds using the Spring scheduling mechanism. I created a small example in Kotlin to illustrate this:
#Service
class CalculationBatcher(
private val commandGateway: CommandGateway
) {
private val calculationQueue = LinkedList<Any>()
fun queueCalculation(calculation: Any) {
calculationQueue.add(calculation)
}
#Scheduled(fixedRate = 10000) // Send every 10 seconds
#PreDestroy // When destroying the application, send remaining events
fun sendCalculations() {
// Use pop here on the LinkedList while having items to prevent threading issues
val calculationsToSend = LinkedList<Any>()
while (calculationQueue.isNotEmpty()) {
calculationsToSend.push(calculationQueue.pop())
}
commandGateway.sendAndWait<Any>(MyEventsCommand(calculationsToSend), 10, TimeUnit.SECONDS)
}
data class MyEventsCommand(val events: List<Any>)
}
I hope this helps. If the reason was something else, let me know!
I am new to Vertx and was exploring request-reply using event bus.
I want to implement below flow
User requests for a data
controller sends a message on event bus to a redis-processor verticle
redis-processor will wait for n seconds till value is available in redis (there will be a background process which will keep on refreshing cache, hence the wait)
redis-processor will send reply back to controller
controller responds to user
In short I want to do something like this:
Now I want to implement this in Vertx since vertx can run asynchronously. Using event bus I can isolate controller from processor. So controller can accept multiple user request and stay responsive under load.
(I hope I am right with this!)
I have implemented this in very crude fashion in java-vertx. Stuck in below part.
//receive request from controller
vertx.eventBus().consumer(REQUEST_PROCESSOR, evtHandler -> {
String txnId = evtHandler.body().toString();
LOGGER.info("Received message:: {}", txnId);
this.redisAPI.get(txnId, result -> { // <=====
String value = result.result().toString();
LOGGER.info("Value in redis : {}", value);
evtHandler.reply(value); // reply to controller
});
});
pls see line denoted by arrow. How can I wait for x seconds without blocking event loop?
Please help.
Thats actually very simple, you need a timer. Please see docs for details but you will need more or less something like this:
vertx.setTimer(1000, id -> {
this.redisAPI.get(txnId, result -> {
String value = result.result().toString();
LOGGER.info("Value in redis : {}", value);
evtHandler.reply(value); // reply to controller
});
});
You might want to store the timer IDs somewhere so that you can cancel them or that at least you know something is running when a shutdown request comes in for your verticle to delay it. But this all depends on your needs.
As #mohamnag said, you could use a Vertx timer
here is another example on how to user timer.
Note that the timer value is in ms.
As an improvement to the, I will recommend checking that the callback has succeeded before attempting to get the value from redisAPI. This is done using the succeeded() method.
In an asynchronous environment getting that result could fail due to several issues (network errors etc)
vertx.setTimer(n * 1000, id -> {
this.redisAPI.get(txnId, result -> {
if(result.succeeded()){ // the callback succeeded to get a value from redis
String value = result.result().toString();
LOGGER.info("Value in redis : {}", value);
evtHandler.reply(value); // reply to controller
} else {
LOGGER.error("Value could not be gotten from redis : {}", result.cause());
evtHandler.fail(someIntegerCode, result.cause()); // reply with failure related info
}
});
});
I have a requirement where I need to get the status of AWS Glue crawler, which is an async request, and based on when the jobs get completed, I would fire certain events. The catch here is that I do not want to use polling. On looking further, AWS docs suggests to use CompletableFuture object to deal with async request in AWS. But when I try to use, I am not able to form CompletableFuture object as it gives me Type mismatch. I have this code :
GetCrawlerMetricsRequest metricsRequest =
new GetCrawlerMetricsRequest().withCrawlerNameList(Arrays.asList("myJavaCrawler"));
GetCrawlerMetricsResult jsonOb = awsglueClient.getCrawlerMetrics(metricsRequest);
CompletableFuture<GetCrawlerMetricsResult> futureResponse = CompletableFuture<GetCrawlerMetricsResult>awsglueClient.getCrawlerMetricsAsync(metricsRequest);
But futureResponse object shows error stating FutureTask cannot be casted to CompletableFuture.
I am following the approach given here
I am not sure how can I make this working. Based on this futureResponse object, I can then use .whenApply function to trigger the certain job which I want to execute such as pushing the above response into a Kafka Queue. Any ideas?
It seems like you are using AWS SDK v1 when the doc you mentioned shows how to do it using v2 (which has 'Developer Preview' status, so it's not recommended for production). Here is a doc showing how to make async calls in v1
For your use case I would recommend another approach. Glue posts few types of events and one of them is "Crawler Succeeded". So you can create a CloudWatch rule to catch these events and trigger a lambda which will make a call to start appropriate job
I would like to implement a page that be displayed to the user whilst a system command is run. As soon as the command completes the user should be routed to another page.
What are some strategies to implement this?
(A solution without javascript would be ideal)
It can definitely be done. You want to look at Asynchronous programming with HTTP in the documentation, it explains how to do this in a non-blocking way. You will need a little bit of javascript for the redirecting part though.
And I don't know what you mean with "system command" but you probably want to create a job for it, so you can trigger it with a request. You can then poll it until it's finished and then redirect the user. But really the documentation does an infinitely better job at explaining it then I'm doing now.
Here's an example of a controller action where I assume your system command returns some kind of String output for the user. When the Job is completed it will sent a response to the user, thus triggering the success handler in the javascript example.
public static void executeSystemCommand(String input) {
Promise<String> outputPromise = new SystemCommandJob(input).now();
String output = await(outputPromise);
renderText(output);
}
Basically if you're using jQuery's $.ajax you can use the complete event to poll the data (just do the request again if it didn't succeed within the timeout time) and use the success/done event to redirect the user when the application responds to indicate that the "system command" is done running.
Example of a function you could use:
function poll(){
$.ajax({
url: "/systemcommand",
success: function(data){
// redirect to next page here
document.location.href = '/output'
},
complete: poll,
timeout: 20000
});
};
There is also a great example on long polling in javascript on StackOverflow.