I am struggling with indexing jsonB column into Elasicsearch backend, using Hibernate Search 6.0.2
This is my entity:
#Data
#NoArgsConstructor
#Entity
#Table(name = "examples")
public class Example {
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
private UUID id;
#NotNull
#Column(name = "fields")
#Type(type = "jsonb")
private Map<String, Object> fields;
}
and this is my programmatic mapping of elasticsearch backend for Hibernate Search:
#Configuration
#RequiredArgsConstructor
public class ElasticsearchMappingConfig implements HibernateOrmSearchMappingConfigurer {
private final JsonPropertyBinder jsonPropertyBinder;
#Override
public void configure(HibernateOrmMappingConfigurationContext context) {
var mapping = context.programmaticMapping();
var exampleMapping = mapping.type(Example.class);
exampleMapping.indexed();
exampleMapping.property("fields").binder(jsonPropertyBinder);
}
}
I based my custom property binder implementation on Hibernate Search 6.0.2 documentation.
#Component
public class JsonPropertyBinder implements PropertyBinder {
#Override
public void bind(PropertyBindingContext context) {
context.dependencies().useRootOnly();
var schemaElement = context.indexSchemaElement();
var userMetadataField = schemaElement.objectField("metadata");
context.bridge(Map.class, new Bridge(userMetadataField.toReference()));
}
#RequiredArgsConstructor
private static class Bridge implements PropertyBridge<Map> {
private final IndexObjectFieldReference fieldReference;
#Override
public void write(DocumentElement target, Map bridgedElement, PropertyBridgeWriteContext context) {
var map = target.addObject(fieldReference);
((Map<String, Object>) bridgedElement).forEach(map::addValue);
}
}
}
I am aware that documentation defines multiple templates for what an Object in Map can be (like in MultiTypeUserMetadataBinder example), but I really do not know what can be inside. All I know, it is a valid json and my goal is to put it into Elasticsearch as valid json structure under "fields": {...}
In my case jsonB column may contain something like this:
{
"testString": "298",
"testNumber": 123,
"testBoolean": true,
"testNull": null,
"testArray": [
5,
4,
3
],
"testObject": {
"testString": "298",
"testNumber": 123,
"testBoolean": true,
"testNull": null,
"testArray": [
5,
4,
3
]
}
but it throws an exception:
org.hibernate.search.util.common.SearchException: HSEARCH400609: Unknown field 'metadata.testNumber'.
I have also set dynamic_mapping to true in my spring application:
...
spring.jpa.properties.hibernate.search.backend.hosts=127.0.0.3:9200
spring.jpa.properties.hibernate.search.backend.dynamic_mapping=true
...
Any other ideas how can I approach this problem? Or maybe I made an error somewhere?
I am aware that documentation defines multiple templates for what an Object in Map can be (like in MultiTypeUserMetadataBinder example), but I really do not know what can be inside. All I know, it is a valid json and my goal is to put it into Elasticsearch as valid json structure under "fields": {...}
If you don't know what the type of each field is, Hibernate Search won't be able to help much. If you really want to stuff that into your index, I'd suggest declaring a native field and pushing the JSON as-is. But then you won't be able to apply predicates to the metadata fields easily, except using native JSON.
Something like this:
#Component
public class JsonPropertyBinder implements PropertyBinder {
#Override
public void bind(PropertyBindingContext context) {
context.dependencies().useRootOnly();
var schemaElement = context.indexSchemaElement();
// CHANGE THIS
IndexFieldReference<JsonElement> userMetadataField = schemaElement.field(
"metadata",
f -> f.extension(ElasticsearchExtension.get())
.asNative().mapping("{\"type\": \"object\", \"dynamic\":true}");
)
.toReference();
context.bridge(Map.class, new Bridge(userMetadataField));
}
#RequiredArgsConstructor
private static class Bridge implements PropertyBridge<Map> {
private static final Gson GSON = new Gson();
private final IndexFieldReference<JsonElement> fieldReference;
#Override
public void write(DocumentElement target, Map bridgedElement, PropertyBridgeWriteContext context) {
// CHANGE THIS
target.addValue(fieldReference, GSON.toJsonTree(bridgedElement));
}
}
}
Alternatively, you can just declare all fields as strings. Then all features provided by Hibernate Search on string types will be available. But of course things like range predicates or sorts will lead to strange results on numeric values (2 is before 10, but "2" is after "10").
Something like this:
#Component
public class JsonPropertyBinder implements PropertyBinder {
#Override
public void bind(PropertyBindingContext context) {
context.dependencies().useRootOnly();
var schemaElement = context.indexSchemaElement();
var userMetadataField = schemaElement.objectField("metadata");
// ADD THIS
userMetadataField.fieldTemplate(
"userMetadataValueTemplate_default",
f -> f.asString().analyzer( "english" )
);
context.bridge(Map.class, new Bridge(userMetadataField.toReference()));
}
#RequiredArgsConstructor
private static class Bridge implements PropertyBridge<Map> {
private final IndexObjectFieldReference fieldReference;
#Override
public void write(DocumentElement target, Map bridgedElement, PropertyBridgeWriteContext context) {
var map = target.addObject(fieldReference);
// CHANGE THIS
((Map<String, Object>) bridgedElement).forEach(entry -> map.addValue( entry.getKey(), String.valueOf(entry.getValue())));
}
}
}
Related
I use external application which expects an Object that Serializable from me like his function:
externalFunction(Object input);
So I should give that function an input that will be correctly serialized into JSON when the method is invoked (not controlled by me).
But I don't know how data is structured since I receive input from another external application dynamically. So case like this:
1. Get data from 3rd party
2. MyApp should annotate data for Json Serialization
3. Send data to 3rd party as input
4. Response will be produced as JSON
How can I achieve this? How can I give input to the function that is correctly serialized when the function is invoked?
What I tried so far:
So first thing I try is wrap data with some Wrapper like:
public class JsonWrapper<T> implements Serializable
{
public T attributes;
public JsonWrapper( T attributes )
{
this.attributes = attributes;
}
#JsonValue
public T getAttributes( )
{
return attributes;
}
}
So I wrap data like ->
data = getFromThirdParty();
wrapped = new JsonWrapper<>(data);
externalFunction(wrapped);
But it produces a response with "attributes" field which I don't want. Also I tried to use #JsonUnwrapped public T attributes; but the result is same.
I don't want this:
{
"attributes": {
... some fields/values that I don't know, get from 3rd party
}
}
I want like this:
{
... some fields/values that I don't know, get from 3rd party
}
The #JsonUnwrapped annotation doesn't work when T is a Collection (see this answer from the Jackson's creator). But the #JsonValue annotation actually does the trick:
public class JsonWrapper<T> {
#JsonValue
private T value;
public JsonWrapper(T value) {
this.value = value;
}
public T getValue() {
return value;
}
}
If you use Lombok, you can have:
#Getter
#AllArgsConstructor
public class JsonWrapper<T> {
#JsonValue
private T value;
}
Example
Consider the following class:
#Data
#AllArgsConstructor
public class Person {
private String firstName;
private String lastName;
}
When serializing an Person instance, the following result JSON is produced:
ObjectMapper mapper = new ObjectMapper();
JsonWrapper<?> wrapper = new JsonWrapper<>(new Person("John", "Doe"));
String json = mapper.writeValueAsString(wrapper);
{"firstName":"John","lastName":"Doe"}
When serializing a list of Person instances, the following result JSON is produced:
ObjectMapper mapper = new ObjectMapper();
JsonWrapper<?> wrapper = new JsonWrapper<>(
Arrays.asList(
new Person("John", "Doe"),
new Person("Jane", "Poe")
));
String json = mapper.writeValueAsString(wrapper);
[{"firstName":"John","lastName":"Doe"},{"firstName":"Jane","lastName":"Poe"}]
I use Swagger to set up an REST Api Service. For this I used following versions:
swagger (YAML)-file version 2.0
swagger codegen cli version v2.3.0
springfox version 2.5.0
The task of the API Service is to respond with data that we store in our database, as well as its relations.
However, we have circular relations in our data. This means that each Object a can have a relation to Object b, which can have a backward relation to Object a.
Now using Swagger, this will generate an endless long JSON, resulting in an error.
The code looks like this:
public class Service extends DataObject {
#SerializedName("provides")
private List<Utility> provides;
/**
* Get provides
*
* #return provides
**/
#ApiModelProperty(required = true, value = "")
public List<Utility> getProvides() {
return provides;
}
}
public class Utility extends DataObject {
#SerializedName("providedBy")
private List<Service> providedBy;
/**
* Get providedBy
*
* #return providedBy
**/
#ApiModelProperty(required = true, value = "")
public List<Service> getProvidedBy() {
return providedBy;
}
}
Then in the response:
#Override
public ResponseEntity<List<Service>> servicesGet(
#Min(0) #ApiParam(value = "The number of layers of relations the object is enriched by. Lower numbers typically increase performance.", defaultValue = "0") #Valid #RequestParam(value = "layersOfRelations", required = false, defaultValue = "0") final Integer layersOfRelations) {
String accept = this.request.getHeader("Accept");
List<Service> services = Util.getServices();
return new ResponseEntity<List<Service>>(services, HttpStatus.OK);
}
My Question is, where and how can I change the output that will be automatically generated for the return in the method servicesGet()?
I would like to not transform the whole object b into JSON, but rather only its title, so that there will be no endless recursion.
So after some research I found out, that swagger is using Jackson to convert objects into JSON (even though the #SerializedName parameter contained in the generated code was from the gson library, at least in my case).
This means that in this case it I can simply use customization techniques that Jackson provides, and it worked like a charm.
I only had to write a custom Jackson adapter annotation:
#SerializedName("provides")
#JsonSerialize(using = DataObjectAdapter.class)
private List<Utility> provides;
and the adapter looks like this:
public static class DataObjectAdapter extends JsonSerializer<List<Utility>> {
#Override
public void serialize(final List<Utility> arg0, final JsonGenerator arg1, final SerializerProvider arg2)
throws IOException {
arg1.writeStartArray();
for (Utility object : arg0) {
arg1.writeStartObject();
arg1.writeStringField("id", object.getId());
arg1.writeStringField("title", object.getTitle());
arg1.writeStringField("description", object.getDescription());
arg1.writeEndObject();
}
arg1.writeEndArray();
}
}
So now instead of writing the whole object into JSON (and thus recursivly adding all child objects into it, he will only write the information I described in the adapter into JSON.
Similarly a deserializer, where I read out the ID in the JSON and map it to my data, would look like this
public static class ServiceDeserializer extends StdDeserializer<Service> {
public ServiceDeserializer() {
super(Service.class);
}
#Override
public Service deserialize(final JsonParser jp, final DeserializationContext ctxt)
throws IOException, JsonProcessingException {
List<Utility> objects = Util.getUtilities();
JsonNode node = jp.getCodec().readTree(jp);
String id = node.get("id").asText();
String title = node.get("title").asText();
String description = node.get("description").asText();
Service service = new Service(id, title, description);
for (JsonNode utility : node.get("provides")) {
String checkId = utility.get("id").asText();
for (DataObject object : objects) {
if (object.getId().equals(checkId)) {
service.addUtility(object);
break;
}
}
}
return service;
}
}
I want to create a config file that will allow me to define different data generators, each of which will need a different configuration. But, they all share the same method, generateRow, so these classes can all implement an interface. I'm using Jackson version 2.9.4.
To illustrate, here's two sample config files:
{
"data": {
"type": "standard",
"config": {
"rows": 1000,
"columns": 10
}
}
}
and
{
"data": {
"type": "totalSize",
"config": {
"sizeInBytes": 1073741824,
"cellDensityInBytes": 12,
"columns": 5
}
}
}
The first data generator simply creates a file with the given number of rows and columns, the second generator creates a file of a pre-defined size, determining the number of rows needed to satisfy the configured variables (i.e., number of columns and cell density).
So, I created an interface:
import com.fasterxml.jackson.annotation.JsonSubTypes;
import com.fasterxml.jackson.annotation.JsonSubTypes.Type;
import com.fasterxml.jackson.annotation.JsonTypeInfo;
import com.fasterxml.jackson.annotation.JsonTypeInfo.As;
import com.fasterxml.jackson.annotation.JsonTypeInfo.Id;
#JsonTypeInfo(use = Id.NAME, include = As.PROPERTY, property = IGenerateRows.PROPERTY, defaultImpl = StandardRowGenerator.class)
#JsonSubTypes(value = { #Type(StandardRowGenerator.class) })
public interface IGenerateRows {
public static final String PROPERTY = "type";
public String[] generateRow();
}
And I have at least one concrete implementation:
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.annotation.JsonTypeName;
#JsonTypeName(value = StandardRowGenerator.TYPE)
public class StandardRowGenerator {
public static final String TYPE = "standard";
private static final String ROWS = "rows";
private static final String COLUMNS = "columns";
#JsonProperty(value = ROWS, required = true)
private int rows;
#JsonProperty(value = COLUMNS, required = true)
private int columns;
}
What I cannot figure out, is how to handle the config node of a data generator node in my configuration file. How would I correctly set up my concrete classes to define the properties they need to generate data?
In my bootstrap code, I instantiate the entire config object as follows:
new ObjectMapper().readValue(inputStream, DataGeneratorConfig.class);
For brevity, I've omitted getters and setters, and the rest of the config file which isn't pertinent to the question at-hand. If I can provide any additional details or code, let me know.
I'm a little unsure about the underlying implementation of your classes and what data they are genearting etc.
But you are along the right sort of lines, I've pushed what I think is a working example of what you are looking to this repo, note this is using https://projectlombok.org/ to generate the POJOs because Im lazy.
https://github.com/Flaw101/jackson-type-info
it will ignore the "data" node. This is mostly because again Im lazy, the entities could be wrapped in a Data class to handle it. The ObjectMapper in the test enables the features required for this.
It will read/write the data of the config classes. Inline with the exampels you've specified.
There's no quick wins for automagically desearlizing the data. You could maybe just write it to a map -> Object but that's incredibly messy and with tools like lombok/IDE class geneartion making these entities should be seconds of work.
The IGenerateRow looks like,
#JsonTypeInfo(use = Id.NAME, include = As.PROPERTY, property = RowGenerator.PROPERTY, defaultImpl = StandardRowGenerator.class)
#JsonSubTypes(value = { #Type(StandardRowGenerator.class), #Type(TotalSizeGeneartor.class) })
#JsonRootName(value = "data")
public abstract interface RowGenerator {
public static final String PROPERTY = "type";
Config getConfig();
}
And Config is just a marker interface for the concrete impls.
public interface Config {
}
The SimpleTypeGenerator now becomes,
#JsonTypeName(value = StandardRowGenerator.TYPE)
#Data
public class StandardRowGenerator implements RowGenerator {
public static final String TYPE = "standard";
private StandardConfig config;
#Data
public static class StandardConfig implements Config {
private int rows;
private int columns;
}
}
And similar for TotalSize,
#JsonTypeName(value = TotalSizeGeneartor.TYPE)
#Data
public class TotalSizeGeneartor implements RowGenerator {
public static final String TYPE = "totalSize";
private TotalSizeConfig config;
#Data
public static class TotalSizeConfig implements Config {
private long sizeInBytes;
private int cellDensityInBytes;
private int columns;
}
}
These could be improved with more/better generic type information to be able to get the concrete references to config.
The test class reads your two configs in the resource folder, writes them to the object and back to a string comparing the before/after, that there is no null or empty properties, and that the interfaces are of the correct implementation.
Note, this uses the assertThat from AssertJ
public class JacksonTest {
private ObjectMapper mapper;
private String json;
#Before
public void setup() throws Exception {
mapper = new ObjectMapper();
mapper.configure(SerializationFeature.WRAP_ROOT_VALUE, true);
mapper.configure(DeserializationFeature.UNWRAP_ROOT_VALUE, true);
}
#Test
public void testDeserStandard() throws Exception {
json = StringUtils.deleteWhitespace(
new String(Files.readAllBytes(Paths.get("src/main/resources/standard.json")), StandardCharsets.UTF_8));
RowGenerator generator = mapper.readValue(json, RowGenerator.class);
assertThat(generator).hasNoNullFieldsOrProperties().isExactlyInstanceOf(StandardRowGenerator.class);
assertThat(generator.getConfig()).hasNoNullFieldsOrProperties().isExactlyInstanceOf(StandardConfig.class);
assertThat(json).isEqualTo(mapper.writeValueAsString(generator));
System.out.println(generator);
}
#Test
public void testDeserTotalsize() throws Exception {
json = StringUtils.deleteWhitespace(
new String(Files.readAllBytes(Paths.get("src/main/resources/totalsize.json")), StandardCharsets.UTF_8));
RowGenerator generator = mapper.readValue(json, RowGenerator.class);
assertThat(generator).hasNoNullFieldsOrProperties().isExactlyInstanceOf(TotalSizeGeneartor.class);
assertThat(generator.getConfig()).hasNoNullFieldsOrProperties().isExactlyInstanceOf(TotalSizeConfig.class);
assertThat(json).isEqualTo(mapper.writeValueAsString(generator));
System.out.println(generator);
}
}
I'm using Spring Data with a MongoDB to save some documents. When saving documents, I would like that Mongo does not contain empty objects. (How) can this be achieved?
Say I have the following main class:
#Document(collection = "main_doc")
public class MainDoc {
#Id
private String id;
private String title;
private SubDoc subDoc;
}
that contains an object of the following class:
public class SubDoc {
private String title;
private String info;
}
Now if I would try to save the following object:
MainDoc main = new MainDoc();
main.setTitle("someTitle");
main.setSubDoc(new SubDoc());
Note: in reality I do not control the fact that the SubDoc is set like this. It can either be empty or filled in. What I want is that if an element's properties/fields are all NULL, it will not be stored in mongo at all.
This results in something like this in mongo:
{
"_id" : "5a328f9a-6118-403b-a3a0-a55ce52099f3",
"title": "someTitle",
"subDoc": {}
}
What I would like is that if an element contains only null properties, they aren't saved at all, so for the above example I would want the following result:
{
"_id" : "5a328f9a-6118-403b-a3a0-a55ce52099f3",
"title": "someTitle"
}
Saving of documents is done with the help of a repository as following:
#NoRepositoryBean
public interface MainRepo extends CrudRepository<MainDoc, String> {
// save inherited
}
Thanks in advance.
One thing you can do here is to write your custom converter for MainDoc:
public class MainDocConverter implements Converter<MainDoc, DBObject> {
#Override
public DBObject convert(final MainDoc source) {
final BasicDbObject dbObject = new BasicDBObject();
...
if(/* validate is subdoc is not null and not empty */) {
dbOject.put("subDoc", source.getSubDoc());
}
}
}
You can register it in #Configuration file for example:
#Configuration
#EnableMongoRepositories(basePackages = {"package"})
public class MongoConfig {
private final MongoDbFactory mongoDbFactory;
public MongoConfig(final MongoDbFactory mongoDbFactory) {
this.mongoDbFactory = mongoDbFactory;
}
#Bean
public MongoTemplate mongoTemplate() throws Exception {
final MongoTemplate mongoTemplate = new MongoTemplate(mongoDbFactory, getDefaultMongoConverter());
return mongoTemplate;
}
#Bean
public MappingMongoConverter getDefaultMongoConverter() throws Exception {
final MappingMongoConverter converter = new MappingMongoConverter(
new DefaultDbRefResolver(mongoDbFactory), new MongoMappingContext());
converter.setCustomConversions(new CustomConversions(Arrays.asList(new MainDocConverter())));
return converter;
}
}
If you don't want to write a custom converter for your object toy can use default one and and modify it a bit.
final Document document = (Document) getDefaultMongoConverter().convertToMongoType(mainDoc);
if(/* validate is null or is empty */) {
document .remove("subDoc");
}
mongoTemplate().save(document);
Actually it's not the best way. As guys wrote empty object should be stored as {}, but converter can help you with your case.
I use Spring/Spring Boot and Spring MVC with #RestController
I have a composite model objects:
public abstract class BaseQuery {
private final Long characteristicId;
...
}
public abstract class ComparableQuery extends BaseQuery {
private final Object value;
private final String comparisonOperator;
...
}
public class GreaterOrEqualQuery extends ComparableQuery {
public GreaterOrEqualQuery(Long characteristicId, Object value) {
super(characteristicId, value, ">=");
}
}
public class EqualQuery extends ComparableQuery {
public EqualQuery(Long characteristicId, Object value) {
super(characteristicId, value, "=");
}
}
public class GreaterQuery extends ComparableQuery {
public GreaterQuery(Long characteristicId, Object value) {
super(characteristicId, value, ">");
}
}
public class CompositQuery extends BaseQuery {
private final String operator;
private final BaseQuery[] queries;
public CompositQuery(Long characteristicId, Operator operator, BaseQuery... queries) {
super(characteristicId);
this.operator = operator.value;
this.queries = queries;
}
...
}
etc.
The sample usage of this model looks for example like:
Set<BaseQuery> queries = new HashSet<>();
BaseQuery megapixelCharacteristicQuery = new CompositQuery(megapixelCharacteristic.getCharacteristicId(), CompositQuery.Operator.AND, new GreaterOrEqualQuery(megapixelCharacteristic.getCharacteristicId(), 10), new LessOrEqualQuery(megapixelCharacteristic.getCharacteristicId(), 50));
queries.add(megapixelCharacteristicQuery);
queries.add(new EqualQuery(androidCharacteristic.getCharacteristicId(), true));
Serialized JSON object for Set<BaseQuery> queries looks like:
[
{
"operator":"AND",
"queries":[
{
"value":10,
"comparisonOperator":"\u003e\u003d",
"characteristicId":391
},
{
"value":50,
"comparisonOperator":"\u003c\u003d",
"characteristicId":391
}
],
"characteristicId":391
},
{
"value":true,
"comparisonOperator":"\u003d",
"characteristicId":383
}
]
I have to pass this or similar JSON from client application(AngularJS) to my back end REST API endpoint in order to get correctly deserialized model like described above(Set with appropriate entries like CompositQuery or EqualQuery).
Right now my Spring application back end logic at my Rest controller unable to correctly deserialize this JSON with appropriate classes.
Is any way in Spring to provide some meta information(or something else) to this JSON in order to help Spring correctly deserialize this structure ?
You can achieve this using jackson annotations on the superclass like following:
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.annotation.JsonSubTypes;
import com.fasterxml.jackson.annotation.JsonSubTypes.Type;
import com.fasterxml.jackson.annotation.JsonTypeInfo;
#JsonTypeInfo(use = JsonTypeInfo.Id.CLASS, include = JsonTypeInfo.As.PROPERTY, property = "javaclass")
#JsonSubTypes({
#Type(value = GreaterOrEqualQuery.class),
#Type(value = EqualQuery.class)
//and so on...
})
public abstract class BaseQuery {
...
}
This will add javaclass property into json representation which is a fully-qualified name in case of use = JsonTypeInfo.Id.CLASS. In order to simplify the value of this property consider different options for use parameter of #JsonTypeInfo (JsonTypeInfo.Id.NAME for example).