I have a question about elasticsearch with spring data.
#Data
#NoArgsConstructor
#AllArgsConstructor
#Document(indexName = "my_es_index")
public class MyEsIndex {
private String id;
private Long counter;
private Long timestamp;
}
and repository
public interface MyEsIndexRepository extends ElasticsearchRepository<MyEsIndex, String> {
Optional<MyEsIndex> findFirstByIdOrderByTimestampDesc(String id);
}
so I have a service where I have to search first for previous one saved record to retrieve previous value, always doing search ordered by timestamp.
#Service
#RequiredArgsConstructor
public class MyEsService {
private final MyEsIndexRepository repository;
public MyEsIndex insert(String previousId) {
Long previousCounter =
repository.findFirstByIdOrderByTimestampDesc(previousId).map(MyEsIndex::getCounter).orElse(0L);
var index = new MyEsIndex(UUID.randomUUID().toString(), ++previousCounter,
Instant.now().toEpochMilli());
return repository.save(index);
}
}
and when trying to do the operation receiving
{"error":{"root_cause":[{"type":"query_shard_exception","reason":"No mapping found for [timestamp] in order to sort on","index":"my_es_index"}
is it possible to do initialization for fields in elasticsearch on empty index?
because the solution of init config is not that clear because it will be used only once when starting working with empty index where never saved a record
#Configuration
public class InitElasticsearchConfig {
private final MyEsIndexRepository repository;
#EventListener(ApplicationReadyEvent.class)
public void initIndex() {
if (repository.findAll(PageRequest.of(0, 1)).isEmpty()) {
var initIndex = new MyEsIndex("initId", 0L, 0L);
repository.save(initIndex);
repository.delete(initIndex);
}
}
is it possible to delegate this solution to spring? I didn't find any
When using Spring Data Elasticsearch repositories - as you do - the normal behaviour is that the mapping is written to Elasticsearch after index creation on application startup when the index does not yet exist.
The problem in your code is that you do not define to what types the properties of your entity should be mapped; you need to add #Field annotations to do that:
#Document(indexName = "my_es_index")
public class MyEsIndex {
private String id;
#Field(type = FieldType.Long)
private Long counter;
#Field(type = FieldType.Long)
private Long timestamp;
}
Properties that are not annotated with a #Field annotation are not written to the mapping but left for automatic mapping by Elasticsearch, that's the cause for the sort not working. As there is no document written to the index, Elasticsearch does not know what type it is and how to sort on that.
In your code there is another thing that might probably not match your desired application logic. In Spring Data Elasticsearch an entity needs to have an id property, that's the property that will be used as the document's id in Elasticsearch. This is normally defined by annotating the property with #Id, if that is missing - as in your case - a property with the name of "id" or "document" is used. So in your case the property id is used.
A document's id is unique in Elasticsearch, if you store a new document under an existing id, the previous content will be overwritten. If that's what you want, the you should add the #Id annotation to your property to make it clear that this is the unique id. But in this case then your code findFirstByIdOrderByTimestamp does not make sense, as a find by id will always return at most one document, so the order by is irrelevant, you could just use a findById() then. I assume that the id should be unique as you initialize it with a UUID.
If your id is not unique and you have multiple documents with the same id and different timestamps, the you'll need to add a new unique property to your entity and annotate that with #Id to prevent id to be used as a unique identifier.
Related
I have an entity 'Product' and I want the primary key in ES to be used as a combination of 'id' and 'name' attributes. How can we do that using spring data elastic search.
public class Product {
#Id
private String id;
#Id
private String name;
#Field(type = FieldType.Keyword)
private Category category;
#Field(type = FieldType.Long)
private double price;
#Field(type = FieldType.Object)
private List<ValidAge> age;
public enum Category {
CLOTHES,
ELECTRONICS,
GAMES;
}
}
One way to achieve this would be the following:
first rename your id property, I changed it to documentId here. This is necessary, because in Spring Data
Elasticsearch an id-property can be either annotated with #Id or it can be namend id. As there can only be one
id-property we need to get this out of the way. It can have the name id in Elasticsearch, set by the #Field
annotation, but the Java property must be changed.
second, add a method annotated with #Id and #AccessType(AccessType.Type.PROPERTY) which returns the value you
want to use in Elasticsearch.
third, you need to provide noop-setter for this property. This is necessary because Spring Data Elasticsearchsoe
not check the id property to be read only when populating an entity after save or when reading from the index.
This is a bug in Spring Data Elasticsearch, I'll create an issue for that
So that comes up with an entity like this:
#Document(indexName = "composite-entity")
public class CompositeEntity {
#Field(name="id", type = FieldType.Keyword)
private String documentId;
#Field(type = FieldType.Keyword)
private String name;
#Field(type = FieldType.Text)
private String text;
#Id
#AccessType(AccessType.Type.PROPERTY)
public String getElasticsearchId() {
return documentId + '-' + name;
}
public void setElasticsearchId(String ignored) {
}
// other getter and setter
}
The repository definition would be straight forward:
public interface CompositeRepository extends ElasticsearchRepository<CompositeEntity,
String> {
}
Remember that for every method that needs an Elasticsearch Id, you'll need to create like it's done in the entity
class.
I am not sure about spring data elasticsearch but spring jpa provides the facility of defining composite primary key by using #IdClass where we can define a separate class(let us say class A) in which we can define all the fields which we want to be a part of composite key Then we can use #IdClass(A.class) in entity class and use #Id annotation on all the fields which should be the part of the composite key
you can refer to this article, although I am not sure whether the same concept will be applicable for spring data es - https://www.baeldung.com/jpa-composite-primary-keys
I'm trying to persist method value in the MongoDB collection as property but MongoTemplate is not storing it's value:
#Data
#Document(collection = "collection")
#JsonIgnoreProperties(ignoreUnknown = true)
public class CollectionObject{
#Id
private ObjectId id;
private List<Object> elements;
public int getElementCount() {
return size(elements);
}
}
This method (as property) exists in the JSON after serialization.
When using
#AccessType(AccessType.Type.PROPERTY)
and I'm trying to persist this object I'm receiving error
java.lang.IllegalStateException: Cannot set property elementCount because no setter, no wither and it's not part of the persistence constructor private com.package.CollectionObject()!
Object is stored using
mongoTemplate.insert(OBJECT, COLLECTION)
JsonProperty with READ_ONLY option is not working as well.
Is there any option to achieve that except creating separate field ?
Please note: while I would accept an XML-based solution if that's truly the only way to accomplish what I'm looking for, I would greatly prefer a solution using Dozer's Java API.
I am new to Dozer and am trying to figure out how to use its API. It seems to default to field-level mappings (if the field names match) and to allow for custom mappers and converters in the event that field-level mapping (based on field name) is either not possible or not logical for your application needs.
I have a situation where my app will take a DTO, say, ReportedIssue (an issue reported by a user and sent to my application over HTTP), and an Issue entity (a data entity that will be persisted to a MySQL DB).
Here are my two objects:
#Data
public class ReportedIssue {
private String typeRefId;
private String reporterRefId;
private String info;
}
#Entity
#Table(name = "issues")
#Data
public class Issue {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
#Column(name = "issue_ref_id")
private String refId;
#Column(name = "issue_tracking_number")
private String trackingNumber;
#OneToOne(fetch = FetchType.EAGER, cascade = {CascadeType.PERSIST, CascadeType.MERGE})
#JoinColumn(name = "issue_type_id", referencedColumnName = "issue_type_id")
private IssueType type;
#Column(name = "issue_reported_on")
private Date reportedOn;
#OneToOne(fetch = FetchType.EAGER, cascade = {CascadeType.PERSIST, CascadeType.MERGE})
#JoinColumn(name = "issue_reporter_id", referencedColumnName = "account_id")
private Account reporter;
#Column(name = "issue_info")
private String info;
}
So in the application frontend, a user can report an issue. The frontend sends a JSON version of a ReportedIssue to the backend, where that JSON is deserialized into a ReportedIssue DTO bean. Then I need Dozer to convert my ReportedIssue into an Issue entity that I can then easily save to my MySQL DB.
Here is my best attempt:
public class ReportedIssueConverter extends DozerConverter<ReportedIssue, Issue> {
private AuthService authService;
public ReportedIssueConverter(AuthService authService, Class<ReportedIssue> prototypeA, Class<Issue> prototypeB) {
super(prototypeA, prototypeB);
this.authService = authService;
}
public ReportedIssueConverter(Class<ReportedIssue> prototypeA, Class<Issue> prototypeB) {
super(prototypeA, prototypeB);
}
#Override
public Issue convertTo(ReportedIssue source, Issue destination) {
Issue issue = new Issue();
issue.setRefId(UUID.randomUUID().toString());
issue.setType(IssueUtils.determineType(source));
issue.setReportedOn(DateTimeUtils.nowInUTC());
issue.setReporter(authService.currentUser());
issue.setInfo(destination.getInfo());
return issue;
}
#Override
public ReportedIssue convertFrom(Issue source, ReportedIssue destination) {
throw new UnsupportedOperationException("we currently don't map from issues to reported issues");
}
}
Several concerns here. For one, is such a custom converter even necessary? Or is there a "better" (more standards compliant or using generally-accepted Dozer practices) way to use the Dozer API to perform this conversion? But mainly, this DozerConverter seems to be intended for bi-directional mapping use cases. Whereas, in my application, I will never have an Issue instance and need to map it back to a ReportedIssue DTO instance. So I only need one-way mapping from ReportedIssue --> Issue. Am I using Dozer correctly by throwing an UnsupportedOperationException or is there another interface or API trick I can use to only leverage the one-way mapping I need?
It could actually be done without a custom converter using custom getter methods in your dto class corresponding to fields in Issue. Dozer works by mapping each field in destination class by trying to invoke the getter method of the corresponding name in the source class.
public class ReportedIssue {
// fields.......
public String getRefId() {
UUID.randomUUID().toString()
}
public IssueType getType() {
IssueUtils.determineType(this);
}
// similarly create getters for other required fields.
}
But for reporter field in Issue, you need an AuthService object. I would suggest writing a static method as below:
public static Issue getIssue(AuthService auth, ReportedIssue dto) {
Issue issue = //map using dozer
issue.setReporter(authService.currentUser());
return issue;
}
Gauntham answer will work. Another option:
Implement a com.github.dozermapper.core.BeanFactory
Your custom BeanFactory can handle
Issue issue = new Issue();
issue.setRefId(UUID.randomUUID().toString());
issue.setReportedOn(DateTimeUtils.nowInUTC());
issue.setReporter(authService.currentUser());
Then depending on your preferences, this could also go into the bean factory
issue.setType(IssueUtils.determineType(source));
Or you could handle that separately in the mapping. Something would need to know how to call IssueUtils, so that is either 1) a customer converter or 2) a change to the DTO or entity to have the functionality through a getter or setter.
Finally, this line would be handled in the Dozer Java API mapping
issue.setInfo(destination.getInfo());
Personally, I like Dozer's com.github.dozermapper.core.loader.api.BeanMappingBuilder where you can explicitly tell it how to map 2 beans, specify the bean factory to use and the custom converter for a specific field.
mapping(ReportedIssue.class, Issue.class, oneWay(), wildcard(true), beanFactory(IssueBeanFactory.class.getName()).fields("this", "type", customConverter(IssueTypeConverter.class)
oneWay(), wildcard(boolean), and beanFactory(String) are found in Dozer's TypeMappingOptions and customConverter(Class.class) is found in Dozer's FieldMappingOptions.
oneWay() makes the mapping work only in the direction specified in the BeanMappingBuilder.
wildcard(true) tells Dozer to automatically map matching fields (this is default behavior).
I am new to spring and spring boot and playing around by developing a simple application. I have the following usecase as mentioned below.
I have the following class definition in my spring boot project.
public class Issue {
#Id
#GenericGenerator(name = "sequence_issue_id", strategy = "com.app.mycompany.AgileCenterServices.util.IssueIdGenerator",
parameters = #Parameter(name = "ProjectKey", value = "PeopleCenter" ))
#GeneratedValue(generator = "sequence_issue_id")
#Column(unique = true)
private String id;
private String projectId;
}
I have a IssueIdGenerator class that sets dynamic value to the id parameter in the Issue POJO class.
However, while doing so, I would like to set a prefix to the id parameter.
How should I be sending this dynamic prefix value to the IssueIdGenerator class.
The prefix value is not a fixed value and will be received as an attribute to the Issue POJO class.
Hence I would like to pass this prefix value which is present as an attribute in the Issue POJO class to the IssueIdGenerator class.
The closer you will get (Using a Generator) to your usecase is by defining a Custom Generator.
An example available in https://www.baeldung.com/hibernate-identifiers is:
Let’s create a generator that builds identifiers containing a String
prefix and a number:
public class MyGenerator implements
IdentifierGenerator, Configurable {
private String prefix;
#Override
public Serializable generate(
SharedSessionContractImplementor session, Object obj)
throws HibernateException {
String query = String.format("select %s from %s",
session.getEntityPersister(obj.getClass().getName(), obj)
.getIdentifierPropertyName(),
obj.getClass().getSimpleName());
Stream ids = session.createQuery(query).stream();
Long max = ids.map(o -> o.replace(prefix + "-", ""))
.mapToLong(Long::parseLong)
.max()
.orElse(0L);
return prefix + "-" + (max + 1);
}
#Override
public void configure(Type type, Properties properties,
ServiceRegistry serviceRegistry) throws MappingException {
prefix = properties.getProperty("prefix");
}
}
>
In this example, we override the generate() method from the IdentifierGenerator interface and first find the highest number from
the existing primary keys of the form prefix-XX.
Then we add 1 to the maximum number found and append the prefix
property to obtain the newly generated id value.
Our class also implements the Configurable interface, so that we can
set the prefix property value in the configure() method.
Next, let’s add this custom generator to an entity. For this, we can
use the #GenericGenerator annotation with a strategy parameter that
contains the full class name of our generator class:
#Entity
public class Product {
#Id
#GeneratedValue(generator = "prod-generator")
#GenericGenerator(name = "prod-generator",
parameters = #Parameter(name = "prefix", value = "prod"),
strategy = "com.baeldung.hibernate.pojo.generator.MyGenerator")
private String prodId;
// ...
}
Also, notice we’ve set the prefix parameter to “prod”.
Let’s see a quick JUnit test for a clearer understanding of the id
values generated:
#Test
public void whenSaveCustomGeneratedId_thenOk()
{
Product product = new Product();
session.save(product);
Product product2 = new Product();
session.save(product2);
assertThat(product2.getProdId()).isEqualTo("prod-2");
}
Here, the first value generated using the “prod” prefix was “prod-1”, followed
by “prod-2”.
Another alternative would be to use a Composite Key (JPA specification):
A composite primary key must correspond to either a single persistent
field or property or to a set of such fields or properties as
described below. A primary key class must be defined to represent a
composite primary key. Composite primary keys typically arise when
mapping from legacy databases when the database key is comprised of
several columns. The EmbeddedId and and IdClass annotations are used
to denote composite primary keys. See sections 9.1.14 and 9.1.15.
I have the following object structure:
#Document(collection = "user")
#TypeAlias("user")
public class User {
#Id
private ObjectId id;
private Contact info = new Contact();
}
and here is the Contact pojo:
public class Contact {
#Indexed(unique = true)
private String mail;
}
But for some reasons not known to me, I don't see Spring-data creating a unique index for the property info.mail
To summarize, I have this json structure of user object:
{_id:xxxxx,info:{mail:"abc#xyz.shoes"}}
And I want to create a unique index on info.mail using Spring data with the above pojo structure. Please help.
As far as I remember, annotating embedded fields with #Indexed will not work. #CompoundIndex is the way to go:
#Document(collection = "user")
#TypeAlias("user")
#CompoundIndexes({
#CompoundIndex(name = "contact_email", def = "{ 'contact.mail': 1 }", unique = true)
})
public class User {
#Id
private ObjectId id;
private Contact info = new Contact();
}
In my case I had a fresh spring boot application 2.3.0 with just #Document, #Id and #Indexed annotations. I was able to retrieve and insert documents but it refused to create the index other than the PK. Finally I figured that there is a property that you need to enable.
spring.data.mongodb.auto-index-creation = true
As a matter of fact it even works on nested objects without #Document annotation.
Hope this helps :)
Obsolete answer, this was with and older version of mongodb 1.x.
Had the same issue, it seems that your Contact class is missing the #Document annotation i.e.
#Document
public class Contact {
#Indexed(unique = true)
private String mail;
}
Should work, quote from the spring mongodb reference
Automatic index creation is only done for types annotated with #Document.
Extending #Xenobius's answer:
If any configuration extending AbstractMongoClientConfiguration is set, MongoMappingContext will back off. The result is:
spring.data.mongodb.auto-index-creation = true will not be effective
You will need add this into your own configuration:
#Override
protected boolean autoIndexCreation() {
return true;
}
ref: https://github.com/spring-projects/spring-boot/issues/28478#issuecomment-954627106