We Keep Coding

Best way to delay a call an external API via Java Spring Boot?

So I have a java app that calls 2 APIs.
Call an API to get a request a file to be generated
Call the second API to get the file.
The first API returns the credentials to get the file. The Second one returns the file but it may take a few seconds or minutes to be generated. What is the best way to account for the time delay between asking the file to be generated and the file being available to pull? Some retry logic should be necessary but on the initial call it always returns a 4xx HTTP response. What's the best way to make this api call maybe there's a better way than using RestTemplate to sequentially call the 2 apis? I thought of adding a short time delay before the 2nd call but I was wondering if there is a better library or async method I can use that's more efficient.
I'd appreciate any 2 cents thanks!

If the author of these two apis is a partner of yours, I think there's a better way, like, the file generator call a callback api of yours and then you call the second API to get the file. And as a supplement, considering unexpected exceptions during above process, a retry schedule to fetch the missed file is probably necessary.
But if you just want to implement the retry and async code more gracefully, the following is my idea
//call the first API
ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor();
CompletableFuture<File> completionFuture = new CompletableFuture<>();
final ScheduledFuture<?> checkFuture = executor.scheduleAtFixedRate(() -> {
//Call the second API to get the file
//if("have got the file successfully") {
// completionFuture.complete(the file);
//} else {
// //do nothing
//}
}, 5, 1, TimeUnit.SECONDS);//set a reasonable schedule policy
completionFuture.whenComplete((file, thrown) -> {
//do something when the retry schedule complete
checkFuture.cancel(false);
});
completionFuture.orTimeout(10, TimeUnit.SECONDS);//setting a timeout policy is probably necessary

passing an Akka stream to an upstream service to populate

I need to call an upstream service (Azure Blob Service) to push data to an OutputStream, which then i need to turn around and push it back to the client, thru akka. Without akka (and just servlet code), i'd just get the ServletOutputStream and pass it to the azure service's method.
The closest i can try to stumble upon, and clearly this is wrong, is something like this
Source<ByteString, OutputStream> source = StreamConverters.asOutputStream().mapMaterializedValue(os -> {
blobClient.download(os);
return os;
});
ResponseEntity resposeEntity = HttpEntities.create(ContentTypes.APPLICATION_OCTET_STREAM, preAuthData.getFileSize(), source);
sender().tell(new RequestResult(resposeEntity, StatusCodes.OK), self());
The idea is i'm calling an upstream service to get an outputstream populated by calling
blobClient.download(os);
It seems like the the lambda function gets called and returns, but then afterwards it fails, because there's no data or something. As if i'm not supposed to be have that lambda function do the work, but perhaps return some object that does the work? Not sure.
How does one do this?

The real issue here is that the Azure API is not designed for back-pressuring. There is no way for the output stream to signal back to Azure that it is not ready for more data. To put it another way: if Azure pushes data faster than you are able to consume it, there will have to be some ugly buffer overflow failure somewhere.
Accepting this fact, the next best thing we can do is:
Use Source.lazySource to only start downloading data when there is downstream demand (aka. the source is being run and data is being requested).
Put the download call in some other thread so that it continues executing without blocking the source from being returned. Once way to do this is with a Future (I'm not sure what Java best practices are, but should work fine either way). Although it won't matter initially, you may need to choose an execution context other than system.dispatcher - it all depends on whether download is blocking or not.
I apologize in advance if this Java code is malformed - I use Akka with Scala, so this is all from looking at the Akka Java API and Java syntax reference.
ResponseEntity responseEntity = HttpEntities.create(
ContentTypes.APPLICATION_OCTET_STREAM,
preAuthData.getFileSize(),
// Wait until there is downstream demand to intialize the source...
Source.lazySource(() -> {
// Pre-materialize the outputstream before the source starts running
Pair<OutputStream, Source<ByteString, NotUsed>> pair =
StreamConverters.asOutputStream().preMaterialize(system);
// Start writing into the download stream in a separate thread
Futures.future(() -> { blobClient.download(pair.first()); return pair.first(); }, system.getDispatcher());
// Return the source - it should start running since `lazySource` indicated demand
return pair.second();
})
);
sender().tell(new RequestResult(responseEntity, StatusCodes.OK), self());

The OutputStream in this case is the "materialized value" of the Source and it will only be created once the stream is run (or "materialized" into a running stream). Running it is out of your control since you hand the Source to Akka HTTP and that will later actually run your source.
.mapMaterializedValue(matval -> ...) is usually used to transform the materialized value but since it is invoked as a part of materialization you can use that to do side effects such as sending the matval in a message, just like you have figured out, there isn't necessarily anything wrong with that even if it looks funky. It is important to understand that the stream will not complete its materialization and become running until that lambda completes. This means problems if download() is blocking rather than forking off some work on a different thread and immediately returning.
There is however another solution: Source.preMaterialize(), it materializes the source and gives you a Pair of the materialized value and a new Source that can be used to consume the already started source:
Pair<OutputStream, Source<ByteString, NotUsed>> pair =
StreamConverters.asOutputStream().preMaterialize(system);
OutputStream os = pair.first();
Source<ByteString, NotUsed> source = pair.second();
Note that there are a few additional things to think of in your code, most importantly if the blobClient.download(os) call blocks until it is done and you call that from the actor, in that case you must make sure that your actor does not starve the dispatcher and stop other actors in your application from executing (see Akka docs: https://doc.akka.io/docs/akka/current/typed/dispatchers.html#blocking-needs-careful-management ).

How to continuously receive and parse the JSON from REST API in spring boot

There is a remote server keep bringing about the data in JSON format. Here is a REST API named
http://192.168.1.101:8000/v1/status,and if I want to collect the data continuously in Spring Boot.Here is a possible JSON from the REST API:
{
"run-status": 0,
"opr-mode": 0,
"ready": false,
"not-ready-reason": 1,
"alarms":["ps", "prm-switch"]
}
I want to keep collecting or just subscribe the REST API, if there is a JSON and then collect it.

There are two main approaches of achieving what you are looking for:
Polling - If this service already exists and you do not have control
over the code, then this might be your only option. You constantly
poll the given URL to check if data has been changed.
In spring, you can use #Scheduled annotation to execute and poll
at any given frequency (using cron expression or fixed delays).
https://www.baeldung.com/spring-scheduled-tasks - provides a detail
of how to create a scheduled tasks.
Webhooks - If you have control over your server code, you can use
webhooks to notify subscriber about availability of data. It is a
callback mechanism where caller will receive a notification about
data changes on the server, and subscriber can then call server to
fetch data immediately.
More about Polling and Webhooks can be found on this URL: https://dzone.com/articles/webhooks-vs-polling-youre-better-than-this-1

Make a "while" cycle what calls your function then goes to sleep (if needed) for the time you want.
Or just while (true) {}

How to use CompletableFuture with AWS Glue job status?

I have a requirement where I need to get the status of AWS Glue crawler, which is an async request, and based on when the jobs get completed, I would fire certain events. The catch here is that I do not want to use polling. On looking further, AWS docs suggests to use CompletableFuture object to deal with async request in AWS. But when I try to use, I am not able to form CompletableFuture object as it gives me Type mismatch. I have this code :
GetCrawlerMetricsRequest metricsRequest =
new GetCrawlerMetricsRequest().withCrawlerNameList(Arrays.asList("myJavaCrawler"));
GetCrawlerMetricsResult jsonOb = awsglueClient.getCrawlerMetrics(metricsRequest);
CompletableFuture<GetCrawlerMetricsResult> futureResponse = CompletableFuture<GetCrawlerMetricsResult>awsglueClient.getCrawlerMetricsAsync(metricsRequest);
But futureResponse object shows error stating FutureTask cannot be casted to CompletableFuture.
I am following the approach given here
I am not sure how can I make this working. Based on this futureResponse object, I can then use .whenApply function to trigger the certain job which I want to execute such as pushing the above response into a Kafka Queue. Any ideas?

It seems like you are using AWS SDK v1 when the doc you mentioned shows how to do it using v2 (which has 'Developer Preview' status, so it's not recommended for production). Here is a doc showing how to make async calls in v1
For your use case I would recommend another approach. Glue posts few types of events and one of them is "Crawler Succeeded". So you can create a CloudWatch rule to catch these events and trigger a lambda which will make a call to start appropriate job

Correct way to handle a REST API path that can be called synchronously or asynchronously

I'm working on a Spring Boot REST API that handles document and can launch a check on a document.
I have a document resource: /doc:
Create a doc with POST /doc
Rest of the CRUD actions with /doc/{id}
Now I can launch a check on a doc, check can be seen either as an action or as a sub-resource.
It's pretty straightforward to launch (create) a check on a document: POST /doc/{id}/check
The check can however take some time so I want to give the user the choice to launch a synchronous or asynchronous check.
How would I handle this path wise?
Should the user choose sync or async check through a query parameter on POST /doc/{id}/check?
Should I create 2 separate paths?
Also in the case of an async check, I would create a temporary Task resource that can be pooled to know the status of the check.
But then if both check and task are returned from the same path it gets confusing, no?
I read an article that says the resource returned in async should be a check resource filled as much as possible but with a link to the task that can be pooled.
That seems like a good way; I would return a partial check if async with a link to the /task/{id} associated with the check.
However I'm still confused as to what path my API should offer to let the user pick between sync and async checks.
How would you handle it path and resource wise?

Basically it's up to you. Usually if it's a big chunk of data you want to query like /resource/{id} most APIs I have used use GET for synchronous requests and POST for async request returning task or job ID.
For POST in your case if the creation/checking takes time I would consider always doing it asynchronous and returning HTTP 202 Accepted and doc/{id}/check/{id} url where the user can see the result if it is ready or some status that it is still working.
If you want to give them a choice to wait or not it's up to you how to do it. There is a standard header that can be used to modify behavior. For example Expect: 202-accepted for async calls and no header or Expect: 201-created for synchronous calls. This makes the API a bit less clear even though it is a standard. Most people (including me) would probably stick to adding a parameter to the URL for clarification. I don't think it should be in the POST data because it should be data related to the object you are creating

There are multiple questions here. I would try to answer one by one
Checking the health of a resource can be done with query param
/doc/{id} - GET Get the resource details
/doc/{id}?healthCheck=true&async=true GET - Get the resource details and trigger an async health check
For the async health check the response as you mentioned will be 202 and the response contains the link to the health status URL
HTTP/1.1 202 Accepted
Location: /doc/12345/status
If the client sends a GET request to this endpoint, the response should contain the current status of the request. Optionally, it could also include an estimated time to completion or a link to cancel the operation.
Reference
https://learn.microsoft.com/en-us/azure/architecture/best-practices/api-design