We Keep Coding

Nested json type won't serialize in jersey

This bounty has ended. Answers to this question are eligible for a +100 reputation bounty. Bounty grace period ends in 22 hours.
James Wierzba wants to draw more attention to this question.
I'm running a dropwizard/jersey java restful web app.
I have an endpoint that is defined like this in api.yaml:
swagger: '2.0'
info:
version: 0.0.1
basePath: /
schemes:
- https
- http
consumes:
- application/json
- application/x-protobuf
produces:
- application/json
- application/x-protobuf
paths:
/v1/event:
post:
summary: receive a event
operationId: receiveEvent ## this value names the generated Java method
parameters:
- name: event
in: body
schema:
$ref: "#/definitions/Event"
responses:
200:
description: success
schema:
type: object
$ref: '#/definitions/EventResponse'
definitions:
Stream:
properties:
vendor:
type: "string"
Event:
properties:
eventCity:
type: "string"
streams:
type: "array"
items:
$ref: "#/definitions/Stream"
EventResponse:
required:
- statusCode
properties:
statusCode:
type: "integer"
Endpoint is defined like so
#POST
#Consumes({ "application/json", "application/x-protobuf" })
#Produces({ "application/json", "application/x-protobuf" })
#Path("/event")
void receiveEvent(
#Suspended AsyncResponse response,
#Valid Event.EventModel event
);
When issuing json POST request, I cannot get the streams field to get serialized/deserialized property.
This is the payload
{
"eventCity": "San Diego",
"streams": [
{
"vendor": "CBS"
}
]
}
I test like this with curl
curl -X POST -H "Content-Type: application/json" -d '{"eventCity": "San Diego", "streams": [{"vendor": "CBS"}]}' https://localhost:8990/v1/event
In the server request handler:
#Override
public void receiveEvent(AsyncResponse response, Event.EventModel event) {
System.out.println(event.getEventCity());
System.out.println(event.getStreamCount()); // <-- this returns 0? why is the inner 'streams' list not getting serialized? it should have one element
}
And the output:
San Diego
0
Another observation, is that when I issue the same post, but with a protobuf payload, it works. The streams list is populated.
The protobuf was generated like so
// create proto
Stream.StreamModel stream = Stream.StreamModel.newBuilder()
.setVendor("CBS")
.build();
Event.EventModel event = Event.EventModel.newBuilder()
.setEventCity("San Diego")
.addStream(stream)
.build();
// write to file
FileOutputStream fos = new FileOutputStream("/Users/jameswierzba/temp/proto.out");
stream.writeTo(fos);
The output in the endpoint is as expected:
San Diego
1
This is the full generated code for the Event class: https://gist.github.com/wierzba3/84face6c21c4fb6ce554f90707ba6ef9
This is the full generated doe for the Stream class: https://gist.github.com/wierzba3/32664312df87c64049b281daab928f94

You can't use the same class as the payload for protobuf and json.
If you inspect the Event generated for protobuf processing you will notice that the the stream is stored as stream_ and has a setStream that returns a Builder as follows. A JSON deserialiser can't work with it:
/**
* <code>repeated .com.apple.amp.social.linearmasterplaylist.refresh.model.StreamModel stream = 2;</code>
*/
public Builder setStream(
int index, com.apple.amp.social.linearmasterplaylist.refresh.model.Stream.StreamModel value) {
if (streamBuilder_ == null) {
if (value == null) {
throw new NullPointerException();
}
ensureStreamIsMutable();
stream_.set(index, value);
onChanged();
} else {
streamBuilder_.setMessage(index, value);
}
return this;
}
A JSON deserialiser (probably jackson-databind) needs a definition of Event like this generated with openapi-generator-cli. In this case the stream and its setter look like this: (note that the #JsonProperty("streams") is redundant as the property is named streams)
#JsonProperty("streams")
#Valid
private List<Stream> streams = null;
and
public void setStreams(List<Stream> streams) {
this.streams = streams;
}
I have included the other model definitions here to allow you to try them with you controller to show that JSON is correctly consumed.
It is possible to write code to inspect the incoming media type and fork the processing. Eg the following generated by openapi-generator. I would recommend a separate controller method for each media type and some sort of mapping so that a single service layer call will suffice.
#RequestMapping(
method = RequestMethod.POST,
value = "/v1/event",
produces = { "application/json", "application/x-protobuf" },
consumes = { "application/json", "application/x-protobuf" }
)
default ResponseEntity<EventResponse> receiveEvent(
#Parameter(name = "event", description = "") #Valid #RequestBody(required = false) Event event
) {
getRequest().ifPresent(request -> {
for (MediaType mediaType: MediaType.parseMediaTypes(request.getHeader("Accept"))) {
if (mediaType.isCompatibleWith(MediaType.valueOf("application/json"))) {
String exampleString = "{ \"statusCode\" : 0 }";
ApiUtil.setExampleResponse(request, "application/json", exampleString);
break;
}
if (mediaType.isCompatibleWith(MediaType.valueOf("application/x-protobuf"))) {
String exampleString = "Custom MIME type example not yet supported: application/x-protobuf";
ApiUtil.setExampleResponse(request, "application/x-protobuf", exampleString);
break;
}
}
});
return new ResponseEntity<>(HttpStatus.NOT_IMPLEMENTED);
}
On a separate note, but I doubt it will help: it looks like the protobuf is using stream (singular) (eg getStreamList()) and the swagger get uses streams (plural).

So, the issue turned out to be a bug in our companies internal implementation of the code generation module.
Workflow looks like this: api.yaml specified -> .proto file generated -> .java code generated.
To be explicit. Using the schema in question. Note I changed the list to have name myStreamArrayInput to illustrate the issue at hand.
Event:
properties:
eventCity:
type: "string"
myStreamArrayInput:
type: "array"
items:
$ref: "#/definitions/Stream"
This would generate a proto like this:
message Event {
optional string eventCity = 1;
/*
this name should be `myStreamArrayInput` or something
close to it. but it is using the name of the TYPE
instead of the name specified by the dev in api.yaml
*/
repeated StreamModel stream;
}
And then this would generate a java class like this
public final class Event {
public static final class EventModel extends
com.google.protobuf.GeneratedMessageV3 implements EventModelOrBuilder {
private EventModel() {
eventCity_ = "";
stream_ = "";
}
}
}
YMMV, since this is an bug in our own internal code generation engine

import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.databind.ObjectMapper;
import javax.ws.rs.*;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import java.util.ArrayList;
import java.util.List;
#Path("/myresource")
public class MyResource {
#GET
#Produces(MediaType.APPLICATION_JSON)
public Response getJson() {
ObjectMapper mapper = new ObjectMapper();
List<MyObject> myObjects = new ArrayList<>();
MyObject myObject = new MyObject("foo", new NestedObject("bar"));
myObjects.add(myObject);
String json;
try {
json = mapper.writeValueAsString(myObjects);
} catch (Exception e) {
return Response.status(Response.Status.INTERNAL_SERVER_ERROR).build();
}
return Response.ok(json, MediaType.APPLICATION_JSON).build();
}
private static class MyObject {
private String name;
private NestedObject nestedObject;
public MyObject(String name, NestedObject nestedObject) {
this.name = name;
this.nestedObject = nestedObject;
}
#JsonProperty("name")
public String getName() {
return name;
}
#JsonProperty("nested_object")
public NestedObject getNestedObject() {
return nestedObject;
}
}
private static class NestedObject {
private String value;
public NestedObject(String value) {
this.value = value;
}
#JsonProperty("value")
public String getValue() {
return value;
}
}
}

Parse String API response containing GeoJson FeatureCollection to Angular and add them as pins with pop-up info on leaflet map

Below is the service method (JsonObjectBuilderService) that converts an object (FeatureCollectionForGeoJson) to a jsonStr. This service method is used in the Get RequestMapping to send a response to the front-end.
The FeatureCollectionForGeoJson object is a class mapped for GeoJson FeatureCollection.
The GeometryForGeoJson is another class that contains the string type with "Point" value and the array that contains the latitude and longitude for the point.
The PropertyForGeoJson class contains information/properties about that pin that will be displayed in the pop-up when the pin is clicked on on the map.
#Getter
#Setter
#ToString
#NoArgsConstructor
#AllArgsConstructor
public class FeatureForGeoJson {
private final String type = "Feature";
private GeometryForGeoJson geometry;
private PropertyForGeoJson properties;
}
#Service
public class JsonObjectBuilderService {
public String transformObjectToGeoJson(FeatureCollectionForGeoJson featureCollectionForGeoJson){
ObjectMapper Obj = new ObjectMapper();
String jsonStr = null;
try {
jsonStr = Obj.writeValueAsString(featureCollectionForGeoJson);
} catch (JsonProcessingException e) {
e.printStackTrace();
} //catch (IOException e) {
return jsonStr;
}
}
This is the GetMapping that sends the response to Angular
#GetMapping("/power-plants")
public ResponseEntity<String> getAllPowerPlants() {
try {
FeatureCollectionForGeoJson powerPlantsToFeatureCollectionForGeoJson ;
//jpa query for the database to return the information
List<PowerPlant> powerPlantList = powerPlantJpaService.findAll();
if (powerPlantList.isEmpty()) {
logger.info("The power plant list is empty.");
return new ResponseEntity<>(HttpStatus.NO_CONTENT);
}
logger.info("The power plant list is populated and has been returned successfully.");
powerPlantsToFeatureCollectionForGeoJson = transformPowerPlantsToFeaturesCollection.transformPowerPlantToGeoJsonElements(powerPlantList);
String objectToGeoJson = jsonObjectBuilderService.transformObjectToGeoJson(powerPlantsToFeatureCollectionForGeoJson);
logger.info(objectToGeoJson);
return new ResponseEntity<>(objectToGeoJson, HttpStatus.OK);
} catch (Exception e) {
return new ResponseEntity<>(null, HttpStatus.INTERNAL_SERVER_ERROR);
}
}
This is how the response looks like in the browser
This is the Angular method that fetches the response.
This is the Angular component where I call the service method that fetches the response and where I want to add the pins to the map with the pop-ups.
How do I take that response from the API (line 27 from Home.component.ts -right above- or the getAll() method from the PowerPlantService) and process it to extract the Point Geometry, to create a pin with it and extract the properties to add to a pop-up to the pin?

if you use angular you should use Observables and not Promises, also avoid to post images of code, now I can't copy/paste you code.
what you want to do is return an observable in getAll(), something like this:
// in component
this.powerPlantService.getAll$().subscribe(
res => this.featureCollection = res,
err => console.log(err)
);
// in service
getAll$(): Observable<any[]> {
return this.http.get(baseUrl).pipe(
map(data => {
// transform your data here, or remove this pipe if you don't need it
return data;
})
);
}
you can transform your features in a flat object like this:
return this.http.get(baseUrl).pipe(
map(features => {
return features.map(f => {
const pointGeometry: any = {
...f.geometry,
...f.properties
};
return pointGeometry;
});
})
);

If you want to know how the back end formats and sends the response, please check in the body of the question.
Below is the service method that performs a GET request to the back end.
export class PowerPlantService {
constructor(private http: HttpClient) { }
getAll() {
return this.http.get(baseUrl);
}
Below is the component method that subscribes to the answer and adds the elements to the map.
#Component({
selector: 'app-home',
templateUrl: './home.component.html',
styleUrls: ['./home.component.css']
})
export class HomeComponent implements OnInit {
private latitude: number = 45.6427;
private longitude: number = 25.5887;
private map!: L.Map;
private centroid: L.LatLngExpression = [this.latitude, this.longitude];
ngOnInit(): void {
this.initMap();
}
constructor(private powerPlantService: PowerPlantService) {
}
private initMap(): void {
this.map = L.map('map', {
center: this.centroid,
zoom: 2.8
});
const tiles = L.tileLayer('https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png',
{
minZoom: 2.8,
attribution: '© OpenStreetMap'
});
tiles.addTo(this.map);
this.powerPlantService.getAll().subscribe((data: any)=>{
console.log(data);
L.geoJSON(data).addTo(this.map)
})

Reactor - understanding thread pools in .flatMap()

I try to understand how does reactive programming really work. I prepared simple demo for this purpose: reactive WebClient from Spring Framework sends requests to simple rest api and this client prints name of thread in each operation.
rest api:
#RestController
#SpringBootApplication
public class RestApiApplication {
public static void main(String[] args) {
SpringApplication.run(RestApiApplication.class, args);
}
#PostMapping("/resource")
public void consumeResource(#RequestBody Resource resource) {
System.out.println(String.format("consumed resource: %s", resource.toString()));
}
}
#Data
#AllArgsConstructor
class Resource {
private final Long id;
private final String name;
}
and the most important - reactive web client:
#SpringBootApplication
public class ReactorWebclientApplication {
public static void main(String[] args) {
SpringApplication.run(ReactorWebclientApplication.class, args);
}
private final TcpClient tcpClient = TcpClient.create();
private final WebClient webClient = WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(HttpClient.from(tcpClient)))
.baseUrl("http://localhost:8080")
.build();
#PostConstruct
void doRequests() {
var longs = LongStream.range(1L, 10_000L)
.boxed()
.toArray(Long[]::new);
var longsStream = Stream.of(longs);
Flux.fromStream(longsStream)
.map(l -> {
System.out.println(String.format("------- map [%s] --------", Thread.currentThread().getName()));
return new Resource(l, String.format("name %s", l));
})
.filter(res -> {
System.out.println(String.format("------- filter [%s] --------", Thread.currentThread().getName()));
return !res.getId().equals(11_000L);
})
.flatMap(res -> {
System.out.println(String.format("------- flatmap [%s] --------", Thread.currentThread().getName()));
return webClient.post()
.uri("/resource")
.syncBody(res)
.header("Content-Type", "application/json")
.header("Accept", "application/json")
.retrieve()
.bodyToMono(Resource.class)
.doOnSuccess(ignore -> System.out.println(String.format("------- onsuccess [%s] --------", Thread.currentThread().getName())))
.doOnError(ignore -> System.out.println(String.format("------- onerror [%s] --------", Thread.currentThread().getName())));
})
.blockLast();
}
}
#JsonIgnoreProperties(ignoreUnknown = true)
class Resource {
private final Long id;
private final String name;
#JsonCreator
Resource(#JsonProperty("id") Long id, #JsonProperty("name") String name) {
this.id = id;
this.name = name;
}
Long getId() {
return id;
}
String getName() {
return name;
}
#Override
public String toString() {
final StringBuilder sb = new StringBuilder("Resource{");
sb.append("id=").append(id);
sb.append(", name='").append(name).append('\'');
sb.append('}');
return sb.toString();
}
}
And the problem is the behaviour is different than I predicted.
I expected that each call of .map(), .filter() and .flatMap() will be executed on main thread and each call of .doOnSuccess() or .doOnError will be executed on a thread from nio thread pool. So I expected logs that look like:
------- map [main] --------
------- filter [main] --------
------- flatmap [main] --------
(and so on...)
------- onsuccess [reactor-http-nio-2] --------
(and so on...)
But the logs I've got are:
------- map [main] --------
------- filter [main] --------
------- flatmap [main] --------
------- map [main] --------
------- filter [main] --------
------- flatmap [main] --------
------- onsuccess [reactor-http-nio-2] --------
------- onsuccess [reactor-http-nio-6] --------
------- onsuccess [reactor-http-nio-4] --------
------- onsuccess [reactor-http-nio-8] --------
------- map [reactor-http-nio-2] --------
------- filter [reactor-http-nio-2] --------
------- flatmap [reactor-http-nio-2] --------
------- map [reactor-http-nio-2] --------
and each next log in .map(), .filter() and .flatMap() was done on thread from reactor-http-nio.
Next incomprehensible fact is the ratio between operations executed on main thread and reactor-http-nio is always different. Sometimes all operations .map(), .filter() and .flatMap() are performed on main thread.

Reactor, like RxJava, can be considered to be concurrency-agnostic. That is, it does not enforce a concurrency model. Rather, it leaves you, the developer, in command. However, that does not prevent the library from helping you with concurrency.
Obtaining a Flux or a Mono does not necessarily mean that it runs in a dedicated Thread. Instead, most operators continue working in the Thread on which the previous operator executed. Unless specified, the topmost operator (the source) itself runs on the Thread in which the subscribe() call was made.
Project Reactor relevant documentation can be found here.
From your code, the following snippet:
webClient.post()
.uri("/resource")
.syncBody(res)
.header("Content-Type", "application/json")
.header("Accept", "application/json")
.retrieve()
.bodyToMono(Resource.class)
Leads to a thread switch from the main to netty's worker pool. Afterward, all the following actions are performed by the netty worker thread.
If you want to control this behavior, you should add a publishOn(...) statement to your code, for example:
webClient.post()
.uri("/resource")
.syncBody(res)
.header("Content-Type", "application/json")
.header("Accept", "application/json")
.retrieve()
.bodyToMono(Resource.class)
.publishOn(Schedulers.elastic())
In this way, any following action will be performed by the elastic scheduler thread pool.
Another example would be a usage of a dedicated scheduler for heavy tasks that following HTTP request execution.
import static com.github.tomakehurst.wiremock.client.WireMock.aResponse;
import static com.github.tomakehurst.wiremock.client.WireMock.get;
import static com.github.tomakehurst.wiremock.client.WireMock.urlEqualTo;
import com.github.tomakehurst.wiremock.WireMockServer;
import java.util.concurrent.TimeUnit;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.springframework.web.reactive.function.client.ClientResponse;
import org.springframework.web.reactive.function.client.WebClient;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
import reactor.core.scheduler.Schedulers;
import ru.lanwen.wiremock.ext.WiremockResolver;
import ru.lanwen.wiremock.ext.WiremockResolver.Wiremock;
import ru.lanwen.wiremock.ext.WiremockUriResolver;
import ru.lanwen.wiremock.ext.WiremockUriResolver.WiremockUri;
#ExtendWith({
WiremockResolver.class,
WiremockUriResolver.class
})
public class ReactiveThreadsControlTest {
private static int concurrency = 1;
private final WebClient webClient = WebClient.create();
#Test
public void slowServerResponsesTest(#Wiremock WireMockServer server, #WiremockUri String uri) {
String requestUri = "/slow-response";
server.stubFor(get(urlEqualTo(requestUri))
.willReturn(aResponse().withStatus(200)
.withFixedDelay((int) TimeUnit.SECONDS.toMillis(2)))
);
Flux
.generate(() -> Integer.valueOf(1), (i, sink) -> {
System.out.println(String.format("[%s] Emitting next value: %d", Thread.currentThread().getName(), i));
sink.next(i);
return i + 1;
})
.subscribeOn(Schedulers.single())
.flatMap(i ->
executeGet(uri + requestUri)
.publishOn(Schedulers.elastic())
.map(response -> {
heavyTask();
return true;
})
, concurrency)
.subscribe();
blockForever();
}
private void blockForever() {
Object monitor = new Object();
synchronized (monitor) {
try {
monitor.wait();
} catch (InterruptedException ex) {
}
}
}
private Mono<ClientResponse> executeGet(String path) {
System.out.println(String.format("[%s] About to execute an HTTP GET request: %s", Thread.currentThread().getName(), path));
return webClient
.get()
.uri(path)
.exchange();
}
private void heavyTask() {
try {
System.out.println(String.format("[%s] About to execute a heavy task", Thread.currentThread().getName()));
Thread.sleep(TimeUnit.SECONDS.toMillis(20));
} catch (InterruptedException ex) {
}
}
}

Spring Cloud Stream message is not received through message queue

I have 2 microservices(images and comments) which communicate one with another(discovered by eureka discovery service) using Spring Cloud Stream with broker rabbit mq. I want to send a message from images microservice to comment microservice.
The problem is that CommentController StreamListener method 'save' is not called.
Image Microservice-> CommentController.java:
public CommentController(MeterRegistry meterRegistry) {
this.meterRegistry = meterRegistry;
this.flux = Flux.<Message<Comment>>create(
emitter -> this.commentSink = emitter,
FluxSink.OverflowStrategy.IGNORE)
.publish()
.autoConnect();
}
#PostMapping("/comments")
public Mono<String> addComment(Mono<Comment> newComment) {
if (commentSink != null) {
return newComment
.map(comment -> {
commentSink.next(MessageBuilder
.withPayload(comment)
.setHeader(MessageHeaders.CONTENT_TYPE,
MediaType.APPLICATION_JSON_VALUE)
.build());
return comment;
})
.flatMap(comment -> {
meterRegistry
.counter("comments.produced", "imageId", comment.getImageId())
.increment();
return Mono.just("redirect:/");
});
} else {
return Mono.just("redirect:/");
}
}
#StreamEmitter
#Output(Source.OUTPUT)
public void emit(FluxSender output) {
output.send(this.flux);
}
Comment Microservice -> CommentService.java
#StreamListener
#Output(Processor.OUTPUT)
public Flux<Void> save(#Input(Processor.INPUT) Flux<Comment> newComment) {
return repository
.saveAll(newComment)
.flatMap(comment -> {
meterRegistry
.counter("comments.consumed", "imageId", comment.getImageId())
.increment();
return Mono.empty();
});
}
CommentService.java->#Service
#EnableBinding(Processor.class)
public class CommentService { ....
Repository which I cloned Chapter7/part 1
Image Microservice -> CommentController.java
Comment Microservice -> CommentService.java

How to consume a java rest to render nv3d candlestick chart with Angular?

I'm trying to use nvd3d candlestick chart with Angular, but I'm not getting to render it when using a rest service built in Java.
How to consume a java rest to render nv3d candlestick chart with Angular?
My rest is returning this:
[{"id":450,"vwap":3821.62,"faixa":69.48,"open":3858.7,"high":3863.29,"low":3793.81,"close":3795.54,"date":19338}]
The component expected this:
[{values:[{"id":450,"vwap":3821.62,"faixa":69.48,"open":3858.7,"high":3863.29,"low":3793.81,"close":3795.54,"date":19338}]}]
My Angular code:
import { Injectable } from '#angular/core';
import { Provider, SkipSelf, Optional, InjectionToken } from '#angular/core';
import { Response, Http } from '#angular/http';
import { Observable } from 'rxjs/Observable';
import 'rxjs/add/operator/map';
import { HttpInterceptorService, RESTService } from '#covalent/http';
import 'rxjs/add/operator/map';
import 'rxjs/add/operator/catch';
import 'rxjs/add/operator/do';
export interface IDolFutDiario {
id: number;
date: number;
open: number;
high: number;
low: number;
close: number;
vwap: number;
faixa: number;
}
#Injectable()
export class DolfudiarioService extends RESTService<IDolFutDiario>{
constructor(private _http: HttpInterceptorService) {
super(_http, {
baseUrl: 'http://localhost:8080',
path: '',
});
}
staticQuery(): Observable<IDolFutDiario[]> {
return this.http.get('http://localhost:8080/dolfutdiarios')
.map(this.extractData)
.catch(this.handleErrorObservable);
}
extractData(res: Response) {
let body = res.json();
return body;
}
private handleErrorObservable (error: Response | any) {
console.error(error.message || error);
return Observable.throw(error.message || error);
}
}
My Java code:
#RestController
public class DolFutRestController {
#Autowired
DolFutDiarioService dolFutDiarioService;
#RequestMapping(value = "dolfutdiarios", method = RequestMethod.GET, produces = MediaType.APPLICATION_JSON_VALUE)
public ResponseEntity<List<DolFutDiario>> list() {
List<DolFutDiario> dolfutdiarios = dolFutDiarioService.listDolFutDiarios();
return ResponseEntity.ok().body(dolfutdiarios);
}
}
PS: When I put the second block of data [[values: ..... , it works.
However when I get from Java Service it does not.
No errors returned as well.

Well, you need to convert the block of data you get to the one you want. It's not going to work if you use the wrong format. The crux of the matter is in this method:
extractData(res: Response) {
let body = res.json();
return body;
}
There you can map your data to what you need; for example, if you want to wrap it in a values object, do it like so:
extractData(res: Response) {
const body = res.json();
return [{ values: body }];
}
Also, try console.log'ing your code in different steps to see what you have and compare to what you need!