We Keep Coding

Custom-Webclient Spock test throws unwanted NullPointerException in GET call

Using the custom WebClient below:
#Slf4j
#RequiredArgsConstructor
#Component
public class TransitApiClient {
private final TransitApiClientProperties transitApiClientProperties;
private final WebClient transitApiWebClient;
private final OAuth2CustomClient oAuth2CustomClient;
public ResponseEntity<Void> isOfficeOfTransitValidAndNational(String officeId){
try {
final String url = UriComponentsBuilder.fromUriString(transitApiClientProperties.getFindOfficeOfTransit())
.queryParam("codelistKey", "CL173")
.queryParam("itemCode", officeId)
.build()
.toUriString();
return transitApiWebClient.get()
.uri(url)
.header(AUTHORIZATION, getAccessTokenHeaderValue(oAuth2CustomClient.getJwtToken()))
.retrieve()
.onStatus(status -> status.value() == HttpStatus.NO_CONTENT.value(),
clientResponse -> Mono.error( new InvalidOfficeException(null,
"Invalid Office exception occurred while invoking :" + transitApiClientProperties.getFindOfficeOfTransit() + officeId)))
.toBodilessEntity()
.block();
} catch (WebClientResponseException webClientResponseException) {
log.error("Technical exception occurred while invoking :" + transitApiClientProperties.getFindOfficeOfTransit(), webClientResponseException);
throw new TechnicalErrorException(null, "Technical exception occurred while trying to find " + transitApiClientProperties.getFindOfficeOfTransit(), webClientResponseException);
}
}
with its intended usage to hit an endpoint, and check if it returns a 200 code with a body or 204 NoContent code, and react accordingly with some custom exceptions.
I've implemented the groovy-spock test below :
class TransitApiClientSpec extends Specification {
private WebClient transitApiWebClient
private TransitApiClient transitApiClient
private OAuth2CustomClient oAuth2CustomClient
private TransitApiClientProperties transitApiClientProperties
private RequestBodyUriSpec requestBodyUriSpec
private RequestHeadersSpec requestHeadersSpec
private RequestBodySpec requestBodySpec
private ResponseSpec responseSpec
private RequestHeadersUriSpec requestHeadersUriSpec
def setup() {
transitApiClientProperties = new TransitApiClientProperties()
transitApiClientProperties.setServiceUrl("https://test-url")
transitApiClientProperties.setFindOfficeOfTransit("/transit?")
transitApiClientProperties.setUsername("username")
transitApiClientProperties.setPassword("password")
transitApiClientProperties.setAuthorizationGrantType("grantType")
transitApiClientProperties.setClientId("clientId")
transitApiClientProperties.setClientSecret("clientSecret")
oAuth2CustomClient = Stub(OAuth2CustomClient)
oAuth2CustomClient.getJwtToken() >> "token"
transitApiWebClient = Mock(WebClient)
requestHeadersSpec = Mock(RequestHeadersSpec)
responseSpec = Mock(ResponseSpec)
requestHeadersUriSpec = Mock(RequestHeadersUriSpec)
transitApiClient = new TransitApiClient(transitApiClientProperties, transitApiWebClient, oAuth2CustomClient)
}
def "request validation of OoTra and throw InvalidOfficeException"(){
given :
def officeId = "testId"
def uri = UriComponentsBuilder
.fromUriString(transitApiClientProperties.getFindOfficeOfTransit())
.queryParam("codelistKey", "CL173")
.queryParam("itemCode", officeId)
.build()
.toUriString()
1 * transitApiWebClient.get() >> requestHeadersUriSpec
1 * requestHeadersUriSpec.uri(uri) >> requestHeadersSpec
1 * requestHeadersSpec.header(HttpHeaders.AUTHORIZATION, "Bearer token") >> requestHeadersSpec
1 * requestHeadersSpec.retrieve() >> responseSpec
1 * responseSpec.onStatus() >> Mono.error( new InvalidOfficeException(null,null) )
when :
def response = transitApiClient.isOfficeOfTransitValidAndNational(officeId)
then :
thrown(InvalidOfficeException)
}
But instead of an InvalidOfficeException being thrown, a java.lang.NullPointerException is thrown.
It seems to be triggered when during the test run, the program calls the following :
return transitApiWebClient.get()
.uri(url)
.header(AUTHORIZATION, getAccessTokenHeaderValue(oAuth2CustomClient.getJwtToken()))
.retrieve()
.onStatus(status -> status.value() == HttpStatus.NO_CONTENT.value(),
clientResponse -> Mono.error( new InvalidOfficeException(null,
"Invalid Office exception occurred while invoking :" + transitApiClientProperties.getFindOfficeOfTransit() + officeId)))
.toBodilessEntity() <---------------------- **HERE**
.block();
I understand that I haven't mocked its behavior but seems to me that some other mock hasn't been done correctly.

I can only recommend not to mock WebClient calls, as the necessary steps are a pain to mock, as you have seen yourself, requiring a lot of intermediary mocks without actually adding much value. This basically repeats the implementation, thus locking it in, which is not a good thing.
What I usually do is to extract all code that interacts with WebClient into a client class, and only mock this class interactions in my code. From the looks of it this is what you are already doing with TransitApiClient. For these client classes, I would recommend testing them with MockServer, WireMock, or any of the other frameworks. This way you actually make sure that the correct request/responses are sent/received, and you don't have to awkwardly deal with the WebClient interface.

Nested json type won't serialize in jersey

This bounty has ended. Answers to this question are eligible for a +100 reputation bounty. Bounty grace period ends in 22 hours.
James Wierzba wants to draw more attention to this question.
I'm running a dropwizard/jersey java restful web app.
I have an endpoint that is defined like this in api.yaml:
swagger: '2.0'
info:
version: 0.0.1
basePath: /
schemes:
- https
- http
consumes:
- application/json
- application/x-protobuf
produces:
- application/json
- application/x-protobuf
paths:
/v1/event:
post:
summary: receive a event
operationId: receiveEvent ## this value names the generated Java method
parameters:
- name: event
in: body
schema:
$ref: "#/definitions/Event"
responses:
200:
description: success
schema:
type: object
$ref: '#/definitions/EventResponse'
definitions:
Stream:
properties:
vendor:
type: "string"
Event:
properties:
eventCity:
type: "string"
streams:
type: "array"
items:
$ref: "#/definitions/Stream"
EventResponse:
required:
- statusCode
properties:
statusCode:
type: "integer"
Endpoint is defined like so
#POST
#Consumes({ "application/json", "application/x-protobuf" })
#Produces({ "application/json", "application/x-protobuf" })
#Path("/event")
void receiveEvent(
#Suspended AsyncResponse response,
#Valid Event.EventModel event
);
When issuing json POST request, I cannot get the streams field to get serialized/deserialized property.
This is the payload
{
"eventCity": "San Diego",
"streams": [
{
"vendor": "CBS"
}
]
}
I test like this with curl
curl -X POST -H "Content-Type: application/json" -d '{"eventCity": "San Diego", "streams": [{"vendor": "CBS"}]}' https://localhost:8990/v1/event
In the server request handler:
#Override
public void receiveEvent(AsyncResponse response, Event.EventModel event) {
System.out.println(event.getEventCity());
System.out.println(event.getStreamCount()); // <-- this returns 0? why is the inner 'streams' list not getting serialized? it should have one element
}
And the output:
San Diego
0
Another observation, is that when I issue the same post, but with a protobuf payload, it works. The streams list is populated.
The protobuf was generated like so
// create proto
Stream.StreamModel stream = Stream.StreamModel.newBuilder()
.setVendor("CBS")
.build();
Event.EventModel event = Event.EventModel.newBuilder()
.setEventCity("San Diego")
.addStream(stream)
.build();
// write to file
FileOutputStream fos = new FileOutputStream("/Users/jameswierzba/temp/proto.out");
stream.writeTo(fos);
The output in the endpoint is as expected:
San Diego
1
This is the full generated code for the Event class: https://gist.github.com/wierzba3/84face6c21c4fb6ce554f90707ba6ef9
This is the full generated doe for the Stream class: https://gist.github.com/wierzba3/32664312df87c64049b281daab928f94

You can't use the same class as the payload for protobuf and json.
If you inspect the Event generated for protobuf processing you will notice that the the stream is stored as stream_ and has a setStream that returns a Builder as follows. A JSON deserialiser can't work with it:
/**
* <code>repeated .com.apple.amp.social.linearmasterplaylist.refresh.model.StreamModel stream = 2;</code>
*/
public Builder setStream(
int index, com.apple.amp.social.linearmasterplaylist.refresh.model.Stream.StreamModel value) {
if (streamBuilder_ == null) {
if (value == null) {
throw new NullPointerException();
}
ensureStreamIsMutable();
stream_.set(index, value);
onChanged();
} else {
streamBuilder_.setMessage(index, value);
}
return this;
}
A JSON deserialiser (probably jackson-databind) needs a definition of Event like this generated with openapi-generator-cli. In this case the stream and its setter look like this: (note that the #JsonProperty("streams") is redundant as the property is named streams)
#JsonProperty("streams")
#Valid
private List<Stream> streams = null;
and
public void setStreams(List<Stream> streams) {
this.streams = streams;
}
I have included the other model definitions here to allow you to try them with you controller to show that JSON is correctly consumed.
It is possible to write code to inspect the incoming media type and fork the processing. Eg the following generated by openapi-generator. I would recommend a separate controller method for each media type and some sort of mapping so that a single service layer call will suffice.
#RequestMapping(
method = RequestMethod.POST,
value = "/v1/event",
produces = { "application/json", "application/x-protobuf" },
consumes = { "application/json", "application/x-protobuf" }
)
default ResponseEntity<EventResponse> receiveEvent(
#Parameter(name = "event", description = "") #Valid #RequestBody(required = false) Event event
) {
getRequest().ifPresent(request -> {
for (MediaType mediaType: MediaType.parseMediaTypes(request.getHeader("Accept"))) {
if (mediaType.isCompatibleWith(MediaType.valueOf("application/json"))) {
String exampleString = "{ \"statusCode\" : 0 }";
ApiUtil.setExampleResponse(request, "application/json", exampleString);
break;
}
if (mediaType.isCompatibleWith(MediaType.valueOf("application/x-protobuf"))) {
String exampleString = "Custom MIME type example not yet supported: application/x-protobuf";
ApiUtil.setExampleResponse(request, "application/x-protobuf", exampleString);
break;
}
}
});
return new ResponseEntity<>(HttpStatus.NOT_IMPLEMENTED);
}
On a separate note, but I doubt it will help: it looks like the protobuf is using stream (singular) (eg getStreamList()) and the swagger get uses streams (plural).

So, the issue turned out to be a bug in our companies internal implementation of the code generation module.
Workflow looks like this: api.yaml specified -> .proto file generated -> .java code generated.
To be explicit. Using the schema in question. Note I changed the list to have name myStreamArrayInput to illustrate the issue at hand.
Event:
properties:
eventCity:
type: "string"
myStreamArrayInput:
type: "array"
items:
$ref: "#/definitions/Stream"
This would generate a proto like this:
message Event {
optional string eventCity = 1;
/*
this name should be `myStreamArrayInput` or something
close to it. but it is using the name of the TYPE
instead of the name specified by the dev in api.yaml
*/
repeated StreamModel stream;
}
And then this would generate a java class like this
public final class Event {
public static final class EventModel extends
com.google.protobuf.GeneratedMessageV3 implements EventModelOrBuilder {
private EventModel() {
eventCity_ = "";
stream_ = "";
}
}
}
YMMV, since this is an bug in our own internal code generation engine

import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.databind.ObjectMapper;
import javax.ws.rs.*;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import java.util.ArrayList;
import java.util.List;
#Path("/myresource")
public class MyResource {
#GET
#Produces(MediaType.APPLICATION_JSON)
public Response getJson() {
ObjectMapper mapper = new ObjectMapper();
List<MyObject> myObjects = new ArrayList<>();
MyObject myObject = new MyObject("foo", new NestedObject("bar"));
myObjects.add(myObject);
String json;
try {
json = mapper.writeValueAsString(myObjects);
} catch (Exception e) {
return Response.status(Response.Status.INTERNAL_SERVER_ERROR).build();
}
return Response.ok(json, MediaType.APPLICATION_JSON).build();
}
private static class MyObject {
private String name;
private NestedObject nestedObject;
public MyObject(String name, NestedObject nestedObject) {
this.name = name;
this.nestedObject = nestedObject;
}
#JsonProperty("name")
public String getName() {
return name;
}
#JsonProperty("nested_object")
public NestedObject getNestedObject() {
return nestedObject;
}
}
private static class NestedObject {
private String value;
public NestedObject(String value) {
this.value = value;
}
#JsonProperty("value")
public String getValue() {
return value;
}
}
}

Apache Camel SFTP get file content null

Java Code
#Component
#RequiredArgsConstructor
public class WireInboundFileListener extends RouteBuilder {
private final JwtWireTokenService jwtWireTokenService;
private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
#Value("${wire.ftp.protocol}")
private String ftpProtocol;
#Value("${wire.ftp.server}")
private String ftpServer;
#Value("${wire.ftp.server.port}")
private String ftpServerPort;
#Value("${wire.ftp.username}")
private String ftpUsername;
#Value("${wire.ftp.password}")
private String ftpPassword;
#Value("${wire.ftp.private.key.file}")
private String ftpPrivateKeyFile;
#Value("${wire.ftp.private.key.passphrase}")
private String privateKeyPassphrase;
#Value("${wire.file.inbound.dir}")
private String ftpListenDir;
#Value("${wire.file.inbound.url}")
private String inboundUrl;
#Override
public void configure() {
var fromFtpUri = String.format("%s:%s:%s/%s?username=%s&delete=true&antInclude=*.txt",
ftpProtocol, ftpServer, ftpServerPort, ftpListenDir, ftpUsername);
log.info("SFTP inbound listen dir : " + ftpListenDir);
if (Environment.getExecutionEnvironment().equals(Environment.ExecutionEnvironment.AWS)) {
fromFtpUri += "&privateKeyFile=" + ftpPrivateKeyFile + "&privateKeyPassphrase=" + privateKeyPassphrase;
} else {
fromFtpUri += "&password=" + ftpPassword;
}
from(fromFtpUri)
//.delay(10000) event I put delay but still got file content null
.convertBodyTo(String.class)
.process(exchange -> {
final var requestBody = new HashMap<String, Object>();
final var content = exchange.getIn().getBody();
final var fileName = exchange.getIn().getHeader("CamelFileName");
requestBody.put("content", content);
requestBody.put("name", fileName);
exchange.getIn().setBody(OBJECT_MAPPER.writeValueAsString(requestBody));
})
.to("log:com.test.wire.inbound.listener.SftpRoute")
.setHeader(Exchange.HTTP_METHOD, constant("POST"))
.setHeader("Content-Type", constant("application/json"))
.setHeader("Authorization", method(this, "clientToken"))
.to(inboundUrl + "?throwExceptionOnFailure=false")
.log("Response body from wire inbound : ${body}")
.end();
}
public String clientToken() {
return "Bearer " + jwtWireTokenService.getToken();
}
}
Success Request
2022-04-20 03:46:47.910 INFO 1 --- [read #6 - Delay] c.test.wire.inbound.listener.SftpRoute : Exchange[ExchangePattern: InOnly, BodyType: String, Body: {
"name": "sample-inbound-transfer-20220414024722.txt",
"content": "file content"
}]
Fail Request
2022-04-21 09:36:54.148 INFO 1 --- [read #4 - Delay] c.test.wire.inbound.listener.SftpRoute : Exchange[ExchangePattern: InOnly, BodyType: String, Body: {
"name": "sample-inbound-transfer-20220414024722.txt",
"content": ""
}]
Main issue
final var content = exchange.getIn().getBody();// sometimes get null, sometimes can get file contents
When I test to drop a file to SFTP server in local it worked as I expected in Sucess Request because the process to upload seemed fast because it is in local (FileZilla).
But when I test to drop a file again to SFTP server that hosts in real server sometimes it works and sometimes it didn't work Fail Request. Seem like the SFTP consuming file issue. Could you please help me on that? Thanks

The file is probably (sometimes) consumed by your Camel route before the file upload is finished.
What you can try is to configure your endpoint to poll the files only if it has exclusive read-lock on the file (i.e. the file is not in-progress or being written); see the readLock parameter.
I think the default is no read lock; you should set readLock=changed...but I do not remember is such mode is also possible on endpoints of SFTP type.

Sending output of one websocket client as input to another

Quick search on SO failed to find me a similar question so here we go
I basically want RSocket's requestChannel syntax with Webflux so I am able to process the received Flux outside of WebSocketClient.execute() method and write something like this (with session being opened only when the returned flux is subscribed to, proper error propagation, automatic completion and closing of the WS session when both inbound and outbound fluxes are complete -
either completed by the server side or cancelled by the consumer)
service /f wraps its received string messages in 'f(...)': 'str' -> 'f(str)'
service /g does the same with 'g(...)' and the following test passes:
private final DataBufferFactory dataBufferFactory = new DefaultDataBufferFactory();
private WebSocketMessage serializeString(final String text) {
return new WebSocketMessage(Type.TEXT, dataBufferFactory.wrap(text.getBytes(StandardCharsets.UTF_8)));
}
#Test
void test() {
var requests = 5;
var input = Flux.range(0, requests).map(String::valueOf);
var wsClient = new ReactorNettyWebSocketClient(
HttpClient.from(TcpClient.create(ConnectionProvider.newConnection())));
var f = requestChannel(wsClient, fUri, input.map(this::serializeString))
.map(WebSocketMessage::getPayloadAsText);
var g = requestChannel(wsClient, gUri, f.map(this::serializeString))
.map(WebSocketMessage::getPayloadAsText);
var responses = g.take(requests);
var expectedResponses = Stream.range(0, requests)
.map(i -> "g(f(" + i + "))")
.toJavaArray(String[]::new);
StepVerifier.create(responses)
.expectSubscription()
.expectNext(expectedResponses)
.verifyComplete();
}

And this seems to work for me... so far
public static Flux<WebSocketMessage> requestChannel(
WebSocketClient wsClient, URI uri, Flux<WebSocketMessage> outbound) {
CompletableFuture<Flux<WebSocketMessage>> recvFuture = new CompletableFuture<>();
CompletableFuture<Integer> consumerDoneCallback = new CompletableFuture<>();
var executeMono = wsClient.execute(uri,
wss -> {
recvFuture.complete(wss.receive().log("requestChannel.receive " + uri, Level.FINE));
return wss.send(outbound)
.and(Mono.fromFuture(consumerDoneCallback));
}).log("requestChannel.execute " + uri, Level.FINE);
return Mono.fromFuture(recvFuture)
.flatMapMany(recv -> recv.doOnComplete(() -> consumerDoneCallback.complete(1)))
.mergeWith(executeMono.cast(WebSocketMessage.class));
}
Rather interested if there're any flaws with this solution I haven't stumbled upon yet

Can't call soap metod from SoapEngine to Java Soap WebService

I have a problem with me web service which I created in Java. This web service is to send data to my mobile application created in Swift.
For now I just want to test communication between application and WebService, so I create one simple method which returns a string.
My WebService:
#WebService(serviceName = "mSetiWebService", targetNamespace = "http://javawebservices.dawidratajczak.pl")
#SOAPBinding(style = Style.DOCUMENT, use = Use.LITERAL, parameterStyle = ParameterStyle.WRAPPED)
public class mSetiWebService {
#WebMethod(operationName = "getMyOtherStuff", action="getMyOtherStuffAction")
public String getMyOtherStuff() {
return "Hello from WebService !";
}
}
The java package(namespace) name is pl.dawidratajczak.webservices. I test this WebService by Boomerang Plugin for chrome and it works fine. But when I use SOAPEngine for Swift a get an error.
My Swift code:
let soap = SOAPEngine()
soap.userAgent = "SOAPEngine"
soap.actionNamespaceSlash = true
soap.version = SOAPVersion.VERSION_1_1
soap.responseHeader = true // use only for non standard MS-SOAP service
soap.requestURL("http://localhost:8080/mSetiSoapService/mSetiWebService",
soapAction: "getMyOtherStuff",
completeWithDictionary: { (statusCode : Int,
dict : [NSObject : AnyObject]!) -> Void in
let result:Dictionary = dict as Dictionary
NSLog("%#", result)
}) { (error : NSError!) -> Void in
NSLog("%#", error)
}
The error is simple: (I change . in address to _)
SOAPEngine Server response: Unexpected wrapper element getMyOtherStuff found. Expected {[httpjavawebservices_dawidratajczak_pl}getMyOtherStuff.
2016-08-26 20:24:59.835 SETI Mobile[4658:200526] Error Domain=NSOSStatusErrorDomain Code=0 "Unexpected wrapper element getMyOtherStuff found. Expected {http://javawebservices_dawidratajczak_pl}getMyOtherStuff." UserInfo={NSLocalizedDescription=Unexpected wrapper element getMyOtherStuff found. Expected {javawebservices_dawidratajczak_pl}getMyOtherStuff.}**
When I put the soapAction like the error expect {url}getMyOtherStuff I get another error that character { is unrecognized. At the end when I put all url and soapActionme function is not found.
What I do wrong?