ElasticSearch returning all Field values as null - java

My config:
#Configuration
#EnableElasticsearchRepositories(
basePackages = { "com.aj.new.repositories" })
public class ElasticDataSourceConfig {
#Bean
public RestHighLevelClient client() {
ClientConfiguration clientConfiguration = ClientConfiguration.builder()
.connectedTo("localhost:9200")
.build();
return RestClients.create(clientConfiguration).rest();
}
#Bean
public ElasticsearchRestTemplate elasticsearchTemplate() {
return new ElasticsearchRestTemplate(client());
}
}
Document:
#Document(indexName = "provider_search")
public class Provider {
#Id #Field
private Long id;
#Field(type = FieldType.Keyword)
private String search;
#Field(name = "ProviderName", type = FieldType.Keyword)
private String providerName;
#Field(name = "Address")
private String address;
#Field(name = "City")
private String city;
#Field(name = "State")
private String state;
...getters...
...setters...
}
Usage:
#Service
public class MySearchService {
#Autowired
private ElasticsearchRestTemplate elasticsearchRestTemplate;
public List<Provider> searchProvidersUsingElastic(final String
providerName, final AddressSearchCriteriaBean addressCriteria) {
final NativeSearchQueryBuilder searchQueryBuilder = new
NativeSearchQueryBuilder();
if (providerName != null) {
final String regex = ".*" + providerName + ".*";
searchQueryBuilder.withQuery(regexpQuery("providerName", regex));
}
if (addressCriteria.getState() != null) {
searchQueryBuilder.withFilter(matchQuery("state",
addressCriteria.getState())
.fuzziness(Fuzziness.ONE));
}
SearchHits<Provider> articles =
elasticsearchRestTemplate.search(searchQueryBuilder.build(),
Provider.class, IndexCoordinates.of("provider_search"));
final List<Provider> providers = articles
.stream()
.map(SearchHit::getContent)
.collect(Collectors.toList());
return providers;
}
}
When debugging with or without filters, I get providers with only their ID field populated. Every other field like "search", "state", etc. is null.
This is my first venture in ElasticSearch world and I'm not sure what's wrong here. Any help is appreciated.
Edit:
Provider Mappings from Elasticsearch
{
"provider_search": {
"mappings": {
"properties": {
"Address": {
"type": "text",
"analyzer": "autocomplete"
},
"City": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"FaxNumber": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"PhoneNumber": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"ProviderName": {
"type": "text",
"analyzer": "autocomplete"
},
"State": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"Status": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"Zip": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"fac_dbk": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"search": {
"type": "text",
"analyzer": "autocomplete"
}
}
}
}
}
Please note that for testing purposes, I have not mapped every field of Provider on Java side. If that's problematic, let me know.
Second Update:
I have changed the Provider document to map the Field names as is. Still, everything except id is still null.
#Document(indexName = "provider_search")
public class Provider {
#Id
private Long id;
private String search;
private String ProviderName;
private String Address;
private String City;
private String State;
...getters...
...setters...
}
UPDATE:
Turns out the Elasticsearch index had a bug and the fields I had mapped on the Java side were not available on the ES index. It has been fixed and I'm seeing all the values populate correctly.


You have an inconsistency in using the property names (the properties of the Provider entity) and the corresponding field names (the names in Elasticsearch).
Your properties start with a lowercase letter and then are camelcase (providerName, state), for Spring Data Elasticsearch you define them to map to versions starting with a uppercase letter (ProviderName, State) with the #Field annotations. So Spring Data Elasticsearch expects Elasticsearch to return a value for ProviderName and will then store this in the providerName property.
Are the field names in your index starting with an uppercase letter? What does http://localhost:9200/provider_search/_mappings show? This is a guess, as you did not show what Elasticsearch returns, but would explain your error.

Related

Spring elasticsearch - Map<String, String> index

I have an object that contains a Map<String, String> field:
#Document(indexName = "custom-creative-template")
#Setting(settingPath = "/elasticsearch/custom-creative-template-index.json")
public class CustomCreativeTemplateIndexDto {
Map<String, String> customValues;
}
When querying data for this index, I am getting the following error:
org.springframework.data.mapping.MappingException: Couldn't find PersistentEntity for type java.lang.String!
When using Map<String, Object> however, it works.
That's the Settings file for the index:
{
"index": {
"analysis": {
"normalizer": {
"sortable": {
"filter": [
"lowercase"
]
}
}
}
}
}
I have tried setting the FieldType in the Field annotation to FieldType.Text & FieldType.Flattened but I keep getting the same error.
Mappings:
{
"custom-creative-template": {
"mappings": {
"properties": {
"_class": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"customValues": {
"properties": {
"offerBullets": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"offerDescription": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"key": {
"type": "long"
},
"name": {
"type": "text",
"fields": {
"sort": {
"type": "keyword",
"normalizer": "sortable"
}
}
}
}
}
}
}

Works for me with field type object:
#Document(indexName = "custom-creative-template")
#Setting(settingPath = "/elasticsearch/custom-creative-template-index.json")
public class CustomCreativeTemplateIndexDto {
#Field(type = FieldType.Object)
Map<String, String> customValues;
}

Spring Data Mongodb Aggregation - Group by nested objects and build DTO

I have the following Employee data in MongoDB
{
"_id": {
"$oid": "625f09bb1a96bf42ff4c4006"
},
"employeeId": 1234,
"email": "jason#acme.com",
"firstName": "Jason",
"lastName": "Stuart",
"currentCTC": 1201117.61,
"department": {
"$ref": "department",
"$id": {
"$oid": "625f09bb1a96bf42ff4c4005"
}
}
}
{
"_id": {
"$oid": "625f09bb1a96bf42ff4c4006"
},
"employeeId": 1235,
"email": "jasons#acme.com",
"firstName": "Jasons",
"lastName": "Stuarts",
"currentCTC": 1201117.61,
"department": {
"$ref": "department",
"$id": {
"$oid": "625f09bb1a96bf42ff4c4005"
}
}
}
My Spring #Document looks like this:
// Employee.java
#Data
#Document
public class Employee {
#Id
private String id;
private Long employeeId;
private String email;
private String firstName;
private String middleName;
private String lastName;
private Gender gender;
private double currentCTC;
#DBRef
private Department department;
}
// Department.java
#Document
#Data
public class Department {
#Id
private String id;
private String name;
}
Now, my requirement is to find the sum of salaries Department-wise.. I need the data to be in the following way:
[
{
"department": {
"id": "625f09bb1a96bf42ff4c4006",
"name": "Engineering"
},
"cost": 31894773.01
},
{
"department": {
"id": "625f09bb1a96bf42ff4c4006",
"name": "Marketing"
},
"cost": 4552325.25
}
]
I created an aggregate function like this in Spring Data:
public List<DepartmentCost> getDepartmentCosting() {
GroupOperation groupByDepartment = group("department").sum("currentCTC").as("cost").first("$$ROOT").as("department");
Aggregation aggregation = Aggregation.newAggregation(groupByDepartment);
AggregationResults<DepartmentCost> results = mongoTemplate.aggregate(aggregation, "employee", DepartmentCost.class);
return results.getMappedResults();
}
And my expected DepartmentCost.java
#Data
#Document
public class DepartmentCost {
#DBRef
private Department department;
private double cost;
}
Now when I try this API out, I get the data correctly, but I do not get department name. It comes as null. I get a response like
[
{
"department": {
"id": "625f09bb1a96bf42ff4c4006",
"name": null,
},
"cost": 2241117.6100000003
},
{
"department": {
"id": "625f09bb1a96bf42ff4c400a",
"name": null,
},
"cost": 14774021.43
},
{
"department": {
"id": "625f09bc1a96bf42ff4c4013",
"name": null,
},
"cost": 14879633.97
}
]
How can I get the department details expanded in my model? Please help..

After a couple of attempts, I figured it out. All I had to do was this:
GroupOperation groupByDepartment = group("department").sum("currentCTC").as("cost").first("$department").as("department");
as opposed to:
GroupOperation groupByDepartment = group("department").sum("currentCTC").as("cost").first("$$ROOT").as("department");

Spring Data ElasticSearch: Index field must be Keyword and not Text

I'm having this issue where I need my index to look like this:
"status": {
"type": "keyword",
}
I create my Java class:
#Data
#JsonIgnoreProperties(ignoreUnknown = true)
#JsonInclude(JsonInclude.Include.NON_ABSENT)
#Document(indexName = "#{applicationConfiguration.indexName}")
public class ProductDocument {
#Field(type = FieldType.Keyword)
#NotEmpty
#JsonProperty("status")
private String status;
}
But spring creates the index as follows:
"status": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
How can I make the keyword type the main type instead text?

ElasticSearch Indexing Key/Value Map

I'm using elasticsearch as indexing engine in my java project. I'm asking if there is anyway to index a java.util.Map a key/value Map.
For example i have this java class:
public class NextEvent extends NewtonResource{
private Map<String, String> metadata;
private Instant executionDate;
private String type;
protected Boolean executed;
private List<PatchOperation> jsonPatch;
private List<String> crons;
...
}
I want to create an elasticsearch mapping including the Map of metadata variable. somethings like this:
{
"aliases": {
"posc-alias-nextevent": {}
},
"mappings": {
"nextevent": {
"properties": {
"executed": {
"type": "boolean"
},
"executionDate": {
"type": "date"
},
"type": {
"type": "string",
"index": "not_analyzed"
},
"metadata": {
"type": "nested",
"properties": {
"key": {
"type": "string",
"index": "not_analyzed"
},
"value": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
}
}

Deserialize complex JSON to Java, classes nested multiple levels deep

I am trying to make the Json output from Cucumber into a single Java object. This contains objects nested four levels deep, and I am having trouble deserializing it. I am presently using Jackson, but open to suggestions.
Here is my Json code:
{
"line": 1,
"elements": [
{
"line": 3,
"name": "Converteren centimeters naar voeten/inches",
"description": "",
"id": "applicatie-neemt-maten-in-cm-en-converteert-ze-naar-voet/inch,-en-vice-versa;converteren-centimeters-naar-voeten/inches",
"type": "scenario",
"keyword": "Scenario",
"steps": [
{
"result": {
"duration": 476796588,
"status": "passed"
},
"line": 4,
"name": "maak Maten-object aan met invoer in \"centimeters\"",
"match": {
"arguments": [
{
"val": "centimeters",
"offset": 37
}
],
"location": "StepDefinition.maakMatenObjectAanMetInvoerIn(String)"
},
"keyword": "Given "
},
{
"result": {
"duration": 36319,
"status": "passed"
},
"line": 5,
"name": "ik converteer",
"match": {
"location": "StepDefinition.converteerMaten()"
},
"keyword": "When "
},
{
"result": {
"duration": 49138,
"status": "passed"
},
"line": 6,
"name": "uitvoer bevat maat in \"voeten/inches\"",
"match": {
"arguments": [
{
"val": "voeten/inches",
"offset": 23
}
],
"location": "StepDefinition.uitvoerBevatMaatIn(String)"
},
"keyword": "Then "
}
]
},
{
"line": 8,
"name": "Converteren voeten/inches naar centimeters",
"description": "",
"id": "applicatie-neemt-maten-in-cm-en-converteert-ze-naar-voet/inch,-en-vice-versa;converteren-voeten/inches-naar-centimeters",
"type": "scenario",
"keyword": "Scenario",
"steps": [
{
"result": {
"duration": 84175,
"status": "passed"
},
"line": 9,
"name": "maak Maten-object aan met invoer in \"voeten/inches\"",
"match": {
"arguments": [
{
"val": "voeten/inches",
"offset": 37
}
],
"location": "StepDefinition.maakMatenObjectAanMetInvoerIn(String)"
},
"keyword": "Given "
},
{
"result": {
"duration": 23928,
"status": "passed"
},
"line": 10,
"name": "ik converteer",
"match": {
"location": "StepDefinition.converteerMaten()"
},
"keyword": "When "
},
{
"result": {
"duration": 55547,
"status": "passed"
},
"line": 11,
"name": "uitvoer bevat maat in \"centimeters\"",
"match": {
"arguments": [
{
"val": "centimeters",
"offset": 23
}
],
"location": "StepDefinition.uitvoerBevatMaatIn(String)"
},
"keyword": "Then "
}
]
}
],
"name": "Applicatie neemt maten in cm en converteert ze naar voet/inch, en vice versa",
"description": "",
"id": "applicatie-neemt-maten-in-cm-en-converteert-ze-naar-voet/inch,-en-vice-versa",
"keyword": "Feature",
"uri": "sample.feature"
}
I have tried a number of different approaches. First I used nested inner classes, but it appeared you had to make them static, which I feared would not work since I have multiple instances of the same object within one (multiple "element"-objects in the root, for example). Then I tried putting them in separate classes, with Json annotations. Here's where that got me (omitting setters):
public class CucumberUitvoer {
private String name;
private String description;
private String id;
private String keyword;
private String uri;
private int line;
#JsonProperty("elements")
private List<FeatureObject> elements;
public CucumberUitvoer(){}
}
public class FeatureObject {
private String name;
private String description;
private String id;
private String type;
private String keyword;
private int line;
#JsonProperty("steps")
private List<StepObject> steps;
public FeatureObject() {
}
}
public class StepObject {
#JsonProperty("result")
private ResultObject result;
private String name;
private String given;
private String location;
private String keyword;
private int line;
#JsonProperty("match")
private MatchObject match;
public StepObject(){}
}
public class ResultObject {
private int duration;
private String status;
public ResultObject(){}
}
public class MatchObject {
#JsonProperty("arguments")
private List<ArgumentObject> arguments;
private String location;
public MatchObject(){}
}
public class ArgumentObject {
private String val;
private String offset;
public ArgumentObject(){}
}
For clarification, here's a class diagram of how the nesting works.
This solution gives me the following error:
com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of nl.icaprojecten.TestIntegratieQuintor.JSONInterpreter.CucumberUitvoer out of START_ARRAY token
Here is the code doing the actual mapping:
ObjectMapper mapper = new ObjectMapper();
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
CucumberUitvoer obj1 = null;
try {
obj1 = mapper.readValue(json, CucumberUitvoer.class);
} catch (IOException e) {
e.printStackTrace();
}
Is there a quick fix to this approach to make it work, or should I try something entirely different?

Ok I spent some time debugging and trying to figure out what was the problem, and finally was something pretty obvious.
implements Serializable
Thats the line I added to MatchObject and worked.
When we try to deserialize some object first we have to make those classes implements the interface Serializable

I just tried your sample code and oddly, it works.
Can you please double check your imports, if the JSON is coming in as provided and the getters, setters, constructors are actually there?

You can get the idea from this code to deserialize,
public class testCustomDeSerializer extends JsonDeserializer<test> {
public testCustomDeSerializer() {
this(null);
}
public TestCustomDeSerializer(Class t) {
// super(t);
}
#Override
public Test deserialize(JsonParser p, DeserializationContext ctx) throws IOException, JsonProcessingException {
ObjectCodec objectCodec = p.getCodec();
JsonNode node = objectCodec.readTree(p);
ObjectMapper objectMapper = new ObjectMapper();
Test test= new Test();
test.setId(node.get("line").asText());
List<elements> elementList = new ArrayList<>();
JsonNode elementsNode = node.get("elements");
Iterator<JsonNode> slaidsIterator = elementsNode.elements();
while (slaidsIterator.hasNext()) {
Steps steps= new Steps();
JsonNode slaidNode = slaidsIterator.next();
JsonNode stepNode= (JsonNode) slaidNode.get("Steps");
BoundingPoly in = objectMapper.readValue(stepNode.toString(), Steps.class);
elementsNode.setSteps(in);
/// continue
return
}
Hope it helps

Categories