Related
I have an object that contains a Map<String, String> field:
#Document(indexName = "custom-creative-template")
#Setting(settingPath = "/elasticsearch/custom-creative-template-index.json")
public class CustomCreativeTemplateIndexDto {
Map<String, String> customValues;
}
When querying data for this index, I am getting the following error:
org.springframework.data.mapping.MappingException: Couldn't find PersistentEntity for type java.lang.String!
When using Map<String, Object> however, it works.
That's the Settings file for the index:
{
"index": {
"analysis": {
"normalizer": {
"sortable": {
"filter": [
"lowercase"
]
}
}
}
}
}
I have tried setting the FieldType in the Field annotation to FieldType.Text & FieldType.Flattened but I keep getting the same error.
Mappings:
{
"custom-creative-template": {
"mappings": {
"properties": {
"_class": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"customValues": {
"properties": {
"offerBullets": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"offerDescription": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"key": {
"type": "long"
},
"name": {
"type": "text",
"fields": {
"sort": {
"type": "keyword",
"normalizer": "sortable"
}
}
}
}
}
}
}
Works for me with field type object:
#Document(indexName = "custom-creative-template")
#Setting(settingPath = "/elasticsearch/custom-creative-template-index.json")
public class CustomCreativeTemplateIndexDto {
#Field(type = FieldType.Object)
Map<String, String> customValues;
}
I want to save data in MYSQL DB by creating Entity class and repository from scratch. I am able to save the normal String Data, Integer Data but struggling to save complex JSON data's
for instance:
[
{
"id": "0001",
"type": "donut",
"name": "Cake",
"ppu": 0.55,
"batters":
{
"batter":
[
{ "id": "1001", "type": "Regular" },
{ "id": "1002", "type": "Chocolate" },
{ "id": "1003", "type": "Blueberry" },
{ "id": "1004", "type": "Devil's Food" }
]
},
"topping":
[
{ "id": "5001", "type": "None" },
{ "id": "5002", "type": "Glazed" },
{ "id": "5005", "type": "Sugar" },
{ "id": "5007", "type": "Powdered Sugar" },
{ "id": "5006", "type": "Chocolate with Sprinkles" },
{ "id": "5003", "type": "Chocolate" },
{ "id": "5004", "type": "Maple" }
]
},
{
"id": "0002",
"type": "donut",
"name": "Raised",
"ppu": 0.55,
"batters":
{
"batter":
[
{ "id": "1001", "type": "Regular" }
]
},
"topping":
[
{ "id": "5001", "type": "None" },
{ "id": "5002", "type": "Glazed" },
{ "id": "5005", "type": "Sugar" },
{ "id": "5003", "type": "Chocolate" },
{ "id": "5004", "type": "Maple" }
]
},
{
"id": "0003",
"type": "donut",
"name": "Old Fashioned",
"ppu": 0.55,
"batters":
{
"batter":
[
{ "id": "1001", "type": "Regular" },
{ "id": "1002", "type": "Chocolate" }
]
},
"topping":
[
{ "id": "5001", "type": "None" },
{ "id": "5002", "type": "Glazed" },
{ "id": "5003", "type": "Chocolate" },
{ "id": "5004", "type": "Maple" }
]
}
]
How can I store such JSON's in MYSQL Db?
Should I Create Class for every nested element ?
(I would consider to switch to a NoSQL DB instead of MySQL, but okay...)
//1.
create table users_json(
id int auto_increment primary key,
details json);
2.
public interface SomeRepository extends JpaRepository<AnyEntity, Long> {
#Modifying(clearAutomatically = true)
#Query(value = "insert into users_json (details) values (:param) ", nativeQuery = true)
#Transactional
int insertValue(#Param("param") String param);}
3.
anyRepository.insertValue("{ \"page\": \"1\" , \"name\": \"Zafari\", \"os\": \"Mac\", \"spend\": 100, \"resolution\": { \"x\": 1920, \"y\": 1080 } }");
4.
SELECT id, details->'$.name' FROM users_json;
Storing JSON in MySQL is possible. You can use these 3 column types depending upon the column size.
For your Entity class :
#Entity
#Getter
#Setter
public class Test {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Integer id;
#Column(columnDefinition = "LONGTEXT") // can store upto 4GB
private String longText;
#Column(columnDefinition = "MEDIUMTEXT") // can store upto 64MB
private String mediumText;
#Column(columnDefinition = "TEXT") // can store upto 64KB
private String text;
}
For your Controller method :
#PostMapping(value = "/addData")
public void addData(#RequestBody String payload) {
testRepository.addData(payload);
}
For your Repository Class:
#Repository
public interface TestRepository extends JpaRepository<Test,Integer> {
#Modifying
#Transactional
#Query(value = "INSERT INTO test(text,medium_text,long_text) VALUE(?1,?1,?1)" ,nativeQuery = true)
void addData(String payload);
}
In MYSQL it will look like this,
It depends if you want to store your Json as String or do you want to convert it into DTO instances that are mapped to your entities and use repository to save them to DB? If you want to store JSON as a String than It shouldn't be any different from any other String. If you want to store it as Entities than you need to convert your JSON (de-serialize) into your DTOs and then work with them as regular DTOs. It doesn't matter how they where created. I just answered very similar question. Please see here
I am new to the Elastic search world.Basically I am trying to retrieve the nested objects based on the ID.This is the JSON representation of my document.
{
"_index": "xyz",
"_type": "abc",
"_id": "12",
"_version": 1,
"found": true,
"_source":
{
"lastModifiedBy": "12",
"lastModifiedDate": "2015-12-31T19:45:29.493Z",
"profile":
[
{
"type": "nested",
"views":
[
{
"type": "nested",
"id": "view1",
"name": "view1",
"properties":
[
{
"name": "default",
"value": false
}
],
"widgets":
[
{
"type": "nested",
"id": "graph",
"name": "graph",
"elementId": "ui_graph",
"properties":
[
{
"name": "currency",
"value": "YEN"
}
]
}
]
}
} ] } ]
I am trying to get the widgets based on the view id.This is the my search query.
"query" : {
"term" : {
"_id" : "12"
}
},
"post_filter" : {
"nested" : {
"query" : {
"filtered" : {
"query" : {
"match_all" : { }
},
"filter" : {
"term" : {
"profile.views.id" : "view1"
}
}
}
},
"path" : "profile.views"
}
}
}
I am not sure what is wrong here.But getting "nested object under path [profile.views] is not of nested type]".
Below is my mapping structure
{
"xyz": {
"mappings": {
"abc": {
"properties": {
"lastModifiedBy": {
"type": "string"
},
"lastModifiedDate": {
"type": "date",
"format": "dateOptionalTime"
},
"name": {
"type": "string"
},
"profile": {
"properties": {
"lastModifiedBy": {
"type": "string"
},
"lastModifiedDate": {
"type": "date",
"format": "dateOptionalTime"
},
"type": {
"type": "string"
},
"views": {
"properties": {
"id": {
"type": "string"
},
"isDefault": {
"type": "boolean"
},
"name": {
"type": "string"
},
"properties": {
"properties": {
"name": {
"type": "string"
},
"type": {
"type": "string"
},
"value": {
"type": "string"
}
}
},
"type": {
"type": "string"
},
"viewId": {
"type": "string"
},
"widgets": {
"properties": {
"elementId": {
"type": "string"
},
"id": {
"type": "string"
},
"name": {
"type": "string"
},
"properties": {
"properties": {
"name": {
"type": "string"
},
"type": {
"type": "string"
},
"value": {
"type": "string"
}
}
},
"type": {
"type": "string"
}
}
}
}
}
}
}
}
}
}
}
}
Please help!
You are getting the error because you have not specified type as nested for profile and views. Refer to the Docs for how to created nested objects . You should be defining type as nested for every nested object like this
{
"xyz": {
"mappings": {
"abc": {
"properties": {
"lastModifiedBy": {
"type": "string"
},
"lastModifiedDate": {
"type": "date",
"format": "dateOptionalTime"
},
"name": {
"type": "string"
},
"profile": {
"type": "nested", <--- here, you need this for every nested object
"properties": {
"lastModifiedBy": {
"type": "string"
},
"lastModifiedDate": {
"type": "date",
"format": "dateOptionalTime"
},
"type": {
"type": "string"
},
"views": {
"type": "nested",
"properties": {
"id": {
"type": "string"
},
"isDefault": {
"type": "boolean"
},
"name": {
"type": "string"
},
"properties": {
"type": "nested",
"properties": {
"name": {
"type": "string"
},
"type": {
"type": "string"
},
"value": {
"type": "string"
}
}
},
"type": {
"type": "string"
},
"viewId": {
"type": "string"
},
"widgets": {
"type": "nested",
"properties": {
"elementId": {
"type": "string"
},
"id": {
"type": "string"
},
"name": {
"type": "string"
},
"properties": {
"type": "nested",
"properties": {
"name": {
"type": "string"
},
"type": {
"type": "string"
},
"value": {
"type": "string"
}
}
},
"type": {
"type": "string"
}
}
}
}
}
}
}
}
}
}
}
}
Hope this helps!!
I am new to Elastic Search.
I have a User mapping and associated with the User is a Nested Object extraDataValues. In this object is the id, a string value and another nested object. For example:
"extraDataValues": [
{
"id": 1,
"value": "01/01/2016 00:00:00",
"id": 10,
"label": "Metadata Date",
"displayable": true
},
},
{
"id": 2,
"value": "aaaa",
"id": 11,
"label": "Metadata TextBox",
"displayable": true
},
}
],
As you can see, value field can be a date or a normal string. The problem arises here, I want to be able to sort this value given that it could be either a date or a normal string. Moreover, the date can be in two formats: "dd/MM/yyyy HH:mm:ss", "dd/MM/yyyy". How can I achieve this firstly with Elastic Search (so I can understand the theory) and then Java?
I have tried adding "dynamic_date_formats" : ["dd/MM/yyyy HH:mm:ss", "dd/MM/yyyy"]
to no avail.
The mapping for the Users is:
User Mapping Document
{
"User": {
"properties": {
"fullName": {
"type": "string",
"index": "not_analyzed",
"fields": {
"raw_lower_case": {
"type": "string",
"analyzer": "case_insensitive"
}
}
},
"username": {
"type": "string",
"index": "not_analyzed",
"fields": {
"raw_lower_case": {
"type": "string",
"analyzer": "case_insensitive"
}
}
},
"email": {
"type": "string",
"index": "not_analyzed",
"fields": {
"raw_lower_case": {
"type": "string",
"analyzer": "case_insensitive"
}
}
},
"firstName": {
"type": "string",
"index": "not_analyzed",
"fields": {
"raw_lower_case": {
"type": "string",
"analyzer": "case_insensitive"
}
}
},
"surname": {
"type": "string",
"index": "not_analyzed",
"fields": {
"raw_lower_case": {
"type": "string",
"analyzer": "case_insensitive"
}
}
},
"id": {
"type": "long"
},
"extraDataValues": {
"type": "nested",
"dynamic_date_formats" : ["dd/MM/yyyy HH:mm:ss", "dd/MM/yyyy"],
"properties": {
"extraDataValueObject": {
"properties": {
"id": {
"type": "long"
},
"label": {
"type": "string"
},
"displayable": {
"type": "boolean"
}
}
},
"value": {
"type": "string",
"index": "not_analyzed",
"fields": {
"raw_lower_case": {
"type": "string",
"analyzer": "case_insensitive"
}
}
}
}
}
}
}
}
You can't do that the way you are trying to do it. dynamic_date_formats are used only for dynamically added date fields, not for date fields that you specify in your mapping (from the documentation).
What I would suggest trying out is this mapping:
"value": {
"type": "string",
"fields": {
"date1": {
"type": "date",
"format": "dd/MM/yyyy HH:mm:ss",
"ignore_malformed": "true"
},
"date2": {
"type": "date",
"format": "dd/MM/yyyy",
"ignore_malformed": "true"
}
}
}
Where you have a field which is string (for the string type part of the value) and for it you define two subfields each with a different date format. It's imperative to have for them "ignore_malformed": "true" in case you really have a string instead of a date coming in.
In this way you can index this:
POST /my_index/user/1
{
"value": "aaa"
}
POST /my_index/user/2
{
"value": "01/01/2016 00:00:00"
}
POST /my_index/user/3
{
"value": "02/02/2016"
}
And you could differentiate between which type of date or string was indexed like this in a query:
"query": {
"filtered": {
"filter": {
"exists": {
"field": "value.date2"
}
}
}
}
If ES was able to index something under value.date2 then you get that document back. The same goes for value.date1, of course.
Does anyone knows how to create Avro schema which contains list of objects of some class?
I want my generated classes to look like below :
class Child {
String name;
}
class Parent {
list<Child> children;
}
For this, I have written part of schema file but do not know how to tell Avro to create list of objects of type Children?
My schema file looks like below :
{
"name": "Parent",
"type":"record",
"fields":[
{
"name":"children",
"type":{
"name":"Child",
"type":"record",
"fields":[
{"name":"name", "type":"string"}
]
}
}
]
}
Now problem is that I can mark field children as either Child type or array but do not know how to mark it as a array of objects of type Child class?
Can anyone please help?
You need to use array type for creating the list.
Following is the updated schema that handles your usecase.
{
"name": "Parent",
"type":"record",
"fields":[
{
"name":"children",
"type":{
"type": "array",
"items":{
"name":"Child",
"type":"record",
"fields":[
{"name":"name", "type":"string"}
]
}
}
}
]
}
I had following class and avro maven plugin generated two classes accordingly :
public class Employees{
String accountNumber;
String address;
List<Account> accountList;
}
public class Account {
String accountNumber;
String id;
}
Avro file format :
{
"type": "record",
"namespace": "com.mypackage",
"name": "AccountEvent",
"fields": [
{
"name": "accountNumber",
"type": "string"
},
{
"name": "address",
"type": "string"
},
{
"name": "accountList",
"type": {
"type": "array",
"items":{
"name": "Account",
"type": "record",
"fields":[
{ "name": "accountNumber",
"type": "string"
},
{ "name": "id",
"type": "string"
}
]
}
}
}
]
}
Array as type
{
"type": "record",
"name": "jamesMedice",
"fields": [{
"name": "columns",
"type": {
"type": "array",
"items": {
"type": "record",
"name": "columnValues",
"fields": [{
"name": "personId",
"type": "string",
"default": "null"
},
{
"name": "email",
"type": "string",
"default": "null"
}
]
}
}
}]
}