How to put an object into S3 using Webflux asynchronously? - java

An article, AWS S3 with Java – Reactive, describes how to use the AWS SDK 2.0 client with Webflux.
In the example, they use the following handler to upload to S3 then return a HTTP Created response:
#PostMapping
public Mono<ResponseEntity<UploadResult>> uploadHandler(#RequestHeader HttpHeaders headers,
#RequestBody Flux<ByteBuffer> body) {
long length = headers.getContentLength();
String fileKey = UUID.randomUUID().toString();
Map<String, String> metadata = new HashMap<String, String>();
CompletableFuture future = s3client
.putObject(PutObjectRequest.builder()
.bucket(s3config.getBucket())
.contentLength(length)
.key(fileKey.toString())
.contentType(MediaType.APPLICATION_OCTET_STREAM.toString())
.metadata(metadata)
.build(),
AsyncRequestBody.fromPublisher(body));
return Mono.fromFuture(future)
.map((response) -> {
checkResult(response);
return ResponseEntity
.status(HttpStatus.CREATED)
.body(new UploadResult(HttpStatus.CREATED, new String[] {fileKey}));
});
}
This works as intended. Trying to learn WebFlux, I expected that the following would complete the HTTP upload to S3 asynchronously in the same thread that the subscribe method is called on:
#PostMapping
public Mono<ResponseEntity<UploadResult>> uploadHandler(#RequestHeader HttpHeaders headers, #RequestBody Flux<ByteBuffer> body) {
long length = headers.getContentLength();
String fileKey = UUID.randomUUID().toString();
Map<String, String> metadata = new HashMap<String, String>();
Mono<PutObjectResponse> putObjectResponseMono = Mono.fromFuture(s3client
.putObject(PutObjectRequest.builder()
.bucket(s3config.getBucket())
.contentLength(length)
.key(fileKey.toString())
.contentType(MediaType.APPLICATION_OCTET_STREAM.toString())
.metadata(metadata)
.build(),
AsyncRequestBody.fromPublisher(body)));
putObjectResponseMono
.doOnError((e) -> {
log.error("Error putting object to S3 " + Thread.currentThread().getName(), e);
})
.subscribe((response) -> {
log.info("Response from S3: " + response.toString() + "on " + Thread.currentThread().getName());
});
return Mono.just(ResponseEntity
.status(HttpStatus.CREATED)
.body(new UploadResult(HttpStatus.CREATED, new String[]{fileKey})));
}
The HTTP POST completes as expected, but the S3 put request fails with this log message:
2020-06-10 12:31:22.275 ERROR 800 --- [tyEventLoop-0-4] c.b.aws.reactive.s3.UploadResource : Error happened on aws-java-sdk-NettyEventLoop-0-4
software.amazon.awssdk.core.exception.SdkClientException: 400 BAD_REQUEST "Request body is missing: public reactor.core.publisher.Mono<org.springframework.http.ResponseEntity<com.baeldung.aws.reactive.s3.UploadResult>> com.baeldung.aws.reactive.s3.UploadResource.uploadHandler(org.springframework.http.HttpHeaders,reactor.core.publisher.Flux<java.nio.ByteBuffer>)"
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:97) ~[sdk-core-2.10.27.jar:na]
at software.amazon.awssdk.core.internal.util.ThrowableUtils.asSdkException(ThrowableUtils.java:98) ~[sdk-core-2.10.27.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.retryIfNeeded(AsyncRetryableStage.java:125) ~[sdk-core-2.10.27.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.lambda$execute$0(AsyncRetryableStage.java:107) ~[sdk-core-2.10.27.jar:na]
........
I suspect the explanation involves the request to S3 being run on its own thread, but I'm stumped working out what is going wrong, can you shed any light on it?

try this
#RequestBody Flux<ByteBuffer> body
>>> replace #RequestBody byte[]
and
AsyncRequestBody.fromPublisher(body)
>>> replace .fromBytes(body)
and if you want to subscribe from another thread, use: .subscribeOn({Schedulers})

Related

Throws java.lang.IllegalStateException: block()/blockFirst()/blockLast() when retrying invalid token

I am trying to update the token when I get a response with 401 status code.
In order to do that, I used web client. I know that this mainly used to do reactive development but since resttemplate will soon be deprecated I went for this option.
The issue I am facing is that when it does call the api endpoint to get the new token, it throws a 'java.lang.IllegalStateException: block()/blockFirst()/blockLast() '. And make sense as it stated in the exception message It is not supported in thread reactor-http-nio-3.
I saw that there is a map and flatmap option, but I couldn't figure out how to use it inside the doBeforeRetry() to make it process in a different stream.
I need to have that new token before retrying.
So the question is : How can I get the token via another call and then still do the retry ?
I was able to make it work by using a try catch but I would like to find the solution how to use it inside that retry method.
I also try to block the token request by replacing the token response by a Mono and block it by using myMono.toFuture().get() as stated here block()/blockFirst()/blockLast() are blocking error when calling bodyToMono AFTER exchange()
Here is the code :
Method responsible for the call :
public String getValueFromApi(HashMap<String, Object> filter) {
String response = "";
response = webclient
.post()
.uri(endpoint)
.header("token", token.getToken())
.bodyValue(filter)
.retrieve()
.bodyToMono(String.class)
.retryWhen(Retry.max(3).doBeforeRetry(
retrySignal -> tokenService.getTokenFromApi(env)
).filter(InvalidTokenException.class::isInstance))
.block();
return response;
}
Method that retrieve the token :
public void getTokenFromApi(Environment env) {
HashMap<String, String> requestBody = new HashMap<>();
requestBody.put("name", "name");
requestBody.put("password", "password");
String response = WebClient
.builder()
.baseUrl(BASE_PATH)
.defaultHeader(HttpHeaders.CONTENT_TYPE, "application/json")
.build()
.post()
.uri(tokenUri)
.body(BodyInserters.fromValue(requestBody))
.retrieve()
.bodyToMono(String.class)
.block();
getTokenFromResponse(response);
}
private void getTokenFromResponse(String reponse) {
JsonObject tokenObject = new Gson().fromJson(reponse, JsonObject.class);
setToken(tokenObject.get("token").getAsString());
}
WebClient Builder :
#Bean
public WebClient webClientForApi(WebClient.Builder webClientBuilder) {
return webClientBuilder
.clientConnector(new ReactorClientHttpConnector(httpClient))
.filter(errorHandler())
.filter(logRequest())
.clone()
.baseUrl(BASE_PATH)
.defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE)
.defaultHeader(HttpHeaders.ACCEPT, "application/json")
.build();
}
public ExchangeFilterFunction errorHandler() {
return ExchangeFilterFunction.ofResponseProcessor(clientResponse -> {
if (clientResponse.statusCode().equals(HttpStatus.UNAUTHORIZED)) {
return Mono.error(InvalidTokenException::new);
} else if (clientResponse.statusCode() == HttpStatus.INTERNAL_SERVER_ERROR) {
return Mono.error(ApiInternalServerException::new);
} else {
return Mono.just(clientResponse);
}
});
}
private ExchangeFilterFunction logRequest() {
return ExchangeFilterFunction.ofRequestProcessor(clientRequest -> {
logger.info("Request: {} {}", clientRequest.method(), clientRequest.url());
clientRequest.headers().forEach((name, values) -> values.forEach(value -> logger.info("{}={}", name, value)));
return Mono.just(clientRequest);
});
}

How to get binary data from RestTemplate or WebClient in spring boot?

I need to call an API from my microservice which returns binary file data which i have to return back to the requester system
I tried using below code:
#RequestMapping(value="/getDocument", method=RequestMethod.POST)
private ResponseEntity<byte[]> receiveFile(){
MultiValueMap<String, String> map = new LinkedMultiValueMap<String, String>();
map.add("mobile","XXXXXXXXX9");
WebClient webClient = WebClient.create();
ResponseEntity<byte[]> apiResponse = webClient.post()
.uri(new URI("https://api.myapp.net.in/getDocument"))
.header("username", "xyz")
.contentType(MediaType.APPLICATION_FORM_URLENCODED)
//.accept(MediaType.parseMediaType("application/pdf"))
.body(BodyInserters.fromFormData(map))
.retrieve()
.toEntity(byte[].class)
.block();
return apiResponse;
}
When i execute the above code and try to consume from Postman i am getting error as unable to open file but when i directly call the https://api.myapp.net.in/getDocument API via Postman i am able to download the pdf file properly.
Please let me know where am i going wrong.

Spring Boot POST endpoint returning 405 instead of 200

I am running the following Cucumber step definition:
Given("we have set the following data:", (final DataTable dataTable) -> {
final var map = dataTable.asLists(String.class).stream().skip(1)
.collect(toMap(row -> row.get(0), row -> row.get(1)));
final var bodyPublisher = HttpRequest.BodyPublishers.ofString(
new ObjectMapper().writeValueAsString(map));
final var request = newBuilder(
create("http://localhost:" + applicationPort + ORGANIZATION_ENDPOINT))
.header("Content-Type", MediaType.APPLICATION_JSON_VALUE)
.header("Authorization", BASIC_AUTHORIZATION_USER_PASSWORD_BASE64)
.POST(bodyPublisher).build();
assertThat(newHttpClient().send(request, ofString()).statusCode(), is(200));
});
That is hitting the following endpoint:
#PostMapping(value = "/api/v1/endpoint", consumes = MediaType.APPLICATION_JSON_VALUE)
public Map<String, Map> setData(#RequestBody final Map<String, String> map) {
validateInputMap(map);
log.info(this.getClass().getSimpleName() + "-> setData: " + map);
service.addData(map);
return service.retrieve(data);
}
However it is returning a 405 METHOD NOT ALLOWED instead of 200, so it is hitting the endpoint but for some reason not as a POST.
Any hint?
Thank you!
EDIT:
The body publisher body is:
{
"Jane Morgan":"John Smith",
"John Smith":"Jack Diaz",
"Jack Diaz":"Sarah Logan",
"Jacob Hare":"John Smith"
}

Get http response code and all available body

I want ot implement WebFlux example client which can make request with http params and get the response body and http response code. I tried this:
public ClientResponse execute(NotificationMessage nm)
Mono<String> transactionMono = Mono.just(convertedString);
return client.post().uri(builder -> builder.build())
.header(HttpHeaders.USER_AGENT, "agent")
.body(transactionMono, String.class).exchange().block();
}
private static String convert(Map<String, String> map) throws UnsupportedEncodingException {
String result = map.entrySet().stream().map(e -> encode(e.getKey()) + "=" + encode(e.getValue()))
.collect(Collectors.joining("&"));
return result;
}
private static String encode(String s) {
try {
return URLEncoder.encode(s, "UTF-8");
} catch (UnsupportedEncodingException e) {
throw new IllegalStateException(e);
}
}
Can you give me some advice after .exchange() how I can get the http status code and all available body.
From the ClientResponse object returned by exchange you can use response.statusCode() to get the status and use response.bodyToMono() or bodyToFlux() to get the actual body. You should avoid using .block() in reactive programming and use .subscribe() or .flatMap() or other operators to get the data from Mono or Flux objects. Read more about reactive programming and Project reactor (used by spring webflux) here.
For eg:
public Mono<Data> execute(NotificationMessage nm)
return client.post().uri(builder -> builder.build())
.header(HttpHeaders.USER_AGENT, "agent")
.body(transactionMono, String.class).exchange()
.flatMap(response -> {
HttpStatus code = response.statusCode();
Data data = response.bodyToMono(Data.class);
return data;
});
}

Play Framework and Office 365 OAuth

I am working on writing a Play Framework application that interfaces with Azure Active Directory. I'm starting with simply pulling some events but I can't get past the initial token refresh request.
private static void getEventsFromOffice365(){
System.out.println("getting from O365");
Promise<String> promise = WS.url("https://login.microsoftonline.com/common/oauth2/token")
.setBody("grant_type=refresh_token&refresh_token=[refresh token]&scope=openid+offline_access+https%3A%2F%2Foutlook.office.com%2Fmail.read+https%3A%2F%2Foutlook.office.com%2Fcalendars.read+https%3A%2F%2Foutlook.office.com%2Fcontacts.read&redirect_uri=https%3A%2F%2Foauthplay.azurewebsites.net%2F&client_id=[client id]&client_secret=[client secret]")
.setContentType("application/x-www-form-urlencoded")
.post("")
.map(
new Function<WSResponse, String>() {
public String apply(WSResponse response) {
System.out.println("Done");
String result = response.getBody();
System.out.println("Result:" + result);
System.out.println("json:" + response.getStatus());
return result;
}
});
}
For some reason whenever I run this I get the following response from Microsoft.
{"error":"invalid_request","error_description":"AADSTS90014: The request body must contain the following parameter: 'grant_type'.\r\nTrace ID: 6a3c1620-6f4d-4c53-a077-cf1f842c0332\r\nCorrelation ID: 0caba711-d434-4ce9-b15e-a56e27ea5a0f\r\nTimestamp: 2015-11-02 23:31:17Z","error_codes":[90014],"timestamp":"2015-11-02 23:31:17Z","trace_id":"6a3c1620-6f4d-4c53-a077-cf1f842c0332","correlation_id":"0caba711-d434-4ce9-b15e-a56e27ea5a0f"}
As you can see I have the grant_type declared in the post body. Why am I getting this error and how can I solve it?
From the official documents,you could use the setQueryParameter
to pass your parameters. I suggest you can refer to code as following :
WSRequestHolder req=WS.url("https://login.microsoftonline.com/common/oauth2/token");
req.setQueryParameter("grant_type", refresh_token );
req.setQueryParameter("refresh_token ", refresh_token);
req.setQueryParameter("redirect_uri",REDIRECT_URI);
req.setQueryParameter("scope", scope);
req.setQueryParameter("client_secret ", client_secret);
req.setQueryParameter("client_id ", client_id);
req.setContentType("application/x-www-form-urlencoded")
req.map(
new Function<WSResponse, String>() {
public String apply(WSResponse response) {
System.out.println("Done");
String result = response.getBody();
System.out.println("Result:" + result);
System.out.println("json:" + response.getStatus());
return result;
}
});
At the same time, I suggest you can try to use execute method instead of post method like this :
. execute(“post”)
Any results, please let me know.

Categories