ElasticSearch makes index for new records created by UI,but the records created by liquibase file not indexed so it don't appears in search result,ElasticSearch should index all records created by UI and liquibase files,Is there any process for indexing the records in liquibase files.
Liquibase only makes changes to your database. Unless you have some process which listens to the database changes and then updates Elasticsearch, you will not see the changes.
There might be multiple ways to get your database records into Elasticsearch:
Your UI probably calls some back-end code to index a create or an update into Elasticsearch already
Have a batch process which knows which records are changed (e.g. use an updated flag column or a updated_timestamp column) and then index those into Elasticsearch.
The second option can either be done in code using a scripting or back-end scheduled job or you might be able to use Logstash with the jdbc-input plugin.
As Sarwar Bhuiyan and Mogsdad sad
Unless you have some process which listens to the database changes and
then updates Elasticsearch
You can use liquibase to populate elasticsearch(this task will be executed once, just like normal migration). To do this you need to create a customChange:
<customChange class="org.test.ElasticMigrationByEntityName">
<param name="entityName" value="org.test.TestEntity" />
</customChange>
In that java based migration you can call the services you need. Here is an example of what you can do (please do not use code from this example in a production).
public class ElasticMigrationByEntityName implements CustomTaskChange {
private String entityName;
public String getEntityName() {
return entityName;
}
public void setEntityName(String entityName) {
this.entityName = entityName;
}
#Override
public void execute(Database database) {
//We schedule the task for the next execution. We are waiting for the context to start and we get access to the beans
DelayedTaskExecutor.add(new DelayedTask(entityName));
}
#Override
public String getConfirmationMessage() {
return "OK";
}
#Override
public void setUp() throws SetupException {
}
#Override
public void setFileOpener(ResourceAccessor resourceAccessor) {
}
#Override
public ValidationErrors validate(Database database) {
return new ValidationErrors();
}
/* ===================== */
public static class DelayedTask implements Consumer<ApplicationContext> {
private final String entityName;
public DelayedTask(String entityName) {
this.entityName = entityName;
}
#Override
public void accept(ApplicationContext applicationContext) {
try {
checkedAccept(applicationContext);
} catch (Exception e) {
throw new RuntimeException(e);
}
}
//We're going to find beans by name (the most controversial point)
private void checkedAccept(ApplicationContext context) throws ClassNotFoundException {
Class entityClass = Class.forName(entityName);
String name = entityClass.getSimpleName();
//Please do not use this code in production
String repositoryName = org.apache.commons.lang3.StringUtils.uncapitalize(name + "Repository");
String repositorySearchName = org.apache.commons.lang3.StringUtils.uncapitalize(name + "SearchRepository");
JpaRepository repository = (JpaRepository) context.getBean(repositoryName);
ElasticsearchRepository searchRepository = (ElasticsearchRepository) context.getBean(repositorySearchName);
//Doing our work
updateData(repository, searchRepository);
}
//Write your logic here
private void updateData(JpaRepository repository, ElasticsearchRepository searchRepository) {
searchRepository.saveAll(repository.findAll());
}
}
}
Because the beans have not yet been created, we will have to wait for them
#Component
public class DelayedTaskExecutor {
#Autowired
private ApplicationContext context;
#EventListener
//We are waiting for the app to launch
public void onAppReady(ApplicationReadyEvent event) {
Queue<Consumer<ApplicationContext>> localQueue = getQueue();
if(localQueue.size() > 0) {
for (Consumer<ApplicationContext> consumer = localQueue.poll(); consumer != null; consumer = localQueue.poll()) {
consumer.accept(context);
}
}
}
public static void add(Consumer<ApplicationContext> consumer) {
getQueue().add(consumer);
}
public static Queue<Consumer<ApplicationContext>> getQueue() {
return Holder.QUEUE;
}
private static class Holder {
private static final Queue<Consumer<ApplicationContext>> QUEUE = new ConcurrentLinkedQueue();
}
}
An entity example:
#Entity
#Table(name = "test_entity")
#Document(indexName = "testentity")
public class TestEntity implements Serializable {
private static final long serialVersionUID = 1L;
#Id
#Field(type = FieldType.Keyword)
#GeneratedValue(generator = "uuid")
#GenericGenerator(name = "uuid", strategy = "uuid2")
private String id;
#NotNull
#Column(name = "code", nullable = false, unique = true)
private String code;
...
}
Related
I'm trying to do a simple get query on springboot using mongodb as database engine
I have tried with several stuff(sending the data as ObjectId and even changing the repository)
public ResponseEntity<Track> get(String trackId) {
Track find = mongoTemplate.findById(new ObjectId(trackId), Track.class);
Optional<Track> track = tracksRepository.findById(trackId);
if (track.isPresent()) {
return new ResponseEntity<>(track.get(), HttpStatus.OK);
}
return new ResponseEntity<>(HttpStatus.NOT_FOUND);
}
with mongo config
#Configuration
#EnableMongoRepositories(basePackages = "data.store.repositories")
public class MongoConfig extends AbstractMongoClientConfiguration {
private final Logger LOGGER = Logger.getLogger(this.getClass().getSimpleName());
#Primary
#Bean
#Override
public MongoClient mongoClient() {
return MongoClients.create(MongoClientSettings.builder()
.applyToClusterSettings(builder -> builder.hosts(Arrays.asList(new ServerAddress(host, port))))
.build());
}
private MongoCredential mongoCredentials() {
return MongoCredential.createCredential(username, database, password.toCharArray());
}
#Bean
public MongoTemplate mongoTemplate() {
MongoTemplate mongoTemplate = new MongoTemplate(mongoClient(), getDatabaseName());
mongoTemplate.setReadPreference(ReadPreference.secondaryPreferred());
return mongoTemplate;
}
protected String getDatabaseName() {
return database;
}
#Override
public boolean autoIndexCreation() {
return false;
}
}
EDIT: Adding class for context
#Document("track")
public class Track {
#Id
#Field(ATTR_ID)
#JsonProperty(ATTR_ID)
public String id;
public static final String ATTR_ID = "id";
}
and getting always null, with existing keys on my database. could you help me find the issue?
Thanks in advance
I tried this with similar configuration class and found the following worked fine creating/accessing data using MongoTemplate.
The POJO class:
public class Test {
#MongoId(FieldType.OBJECT_ID)
private String id;
private String name;
public Test() {
}
public Test(String s) {
super();
this.name = s;
}
// get, set methods
public String toString( ) {
return id + " - " + name;
}
}
From Spring's CommandLineRunner.run():
// Insert a document into the database
Test t1 = new Test("alpha");
t1 = mt.insert(t1);
System.out.println(t1); // 61e7de9f5aadc2077d9f4a58 - alpha
// Query from the database using the _id
ObjectId id = new ObjectId("61e7de9f5aadc2077d9f4a58");
Test t2 = mt.findById(id, Test.class);
System.out.println(t2);
Note that you need to do this from the class where you are running the code:
#Autowired private MongoTemplate mt;
You can use the #MongoId or #Id annotations in our POJO class to represent MongoDB _id field. The type of the field can be a String or ObjectId. It depends upon how you define.
See this from Spring Data MongoDB documentation on How the _id Field is Handled in the Mapping Layer using:
#MongoId
#Id
Solution is to add to MongoId annotation field type object id
#MongoId(FieldType.OBJECT_ID)
private String id;
I am writing a PUT request API with spring and mongodb. But the save() inserts a new object instead of update the current one.
#Document("Test")
public class Expense {
#Field(name = "name")
private String expenseName;
#Field(name = "category")
private ExpenseCategory expenseCategory;
#Field(name = "amount")
private BigDecimal expenseAmount;
public Expense( String expenseName, ExpenseCategory expenseCategory, BigDecimal expenseAmount) {
this.expenseName = expenseName;
this.expenseCategory = expenseCategory;
this.expenseAmount = expenseAmount;
}
public String getExpenseName() {
return expenseName;
}
public void setExpenseName(String expenseName) {
this.expenseName = expenseName;
}
public ExpenseCategory getExpenseCategory() {
return expenseCategory;
}
public void setExpenseCategory(ExpenseCategory expenseCategory) {
this.expenseCategory = expenseCategory;
}
public BigDecimal getExpenseAmount() {
return expenseAmount;
}
public void setExpenseAmount(BigDecimal expenseAmount) {
this.expenseAmount = expenseAmount;
}
}
This is my reporsitory class
public interface ExpenseRepository extends MongoRepository<Expense, String> {
}
This is my Service class which shows how to update the class.
#Service
public class ExpenseService {
private final ExpenseRepository expenseRepository;
public ExpenseService(ExpenseRepository expenseRepository) {
this.expenseRepository = expenseRepository;
}
public void updateExpense(String id, Expense expense){
Expense savedExpense = expenseRepository.findById(id)
.orElseThrow(() -> new RuntimeException(
String.format("Cannot Find Expense by ID %s", id)));
savedExpense.setExpenseName(expense.getExpenseName());
savedExpense.setExpenseAmount(expense.getExpenseAmount());
savedExpense.setExpenseCategory(expense.getExpenseCategory());
expenseRepository.save(savedExpense);
}
}
This is my controller
#RestController
#RequestMapping("/api/expense")
public class ExpenseController {
private final ExpenseService expenseService;
public ExpenseController(ExpenseService expenseService) {
this.expenseService = expenseService;
}
#PutMapping("/{id}")
public ResponseEntity<Object> updateExpense(#PathVariable String id, #RequestBody Expense expense){
expenseService.updateExpense(id, expense);
return ResponseEntity.ok().build();
}
}
As shown in mongodb compass, mongodb auto generates an _id field for every object. So I do not define a id field or use #id annotation to define a primary for the collection. However, in the service class, expenseRepository.findById(id) retrieves the desired object and update it. Why does save() do the insert instead of update? Many thanks.
JPA Can't find the existing entry as no id field id set. You need to add an id field and set generation type to auto.
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
private int id;
It is interesting point that it sometimes performs insert operation correctly. I do not know why and how this situation can be happens. So, I could not figure out where I made a mistake(s).
Here is my project files.
1) SentFilesDao.java
#Dao
public interface SentFilesDao {
#Query("SELECT id, name, type, status, dateTime FROM sent")
LiveData<List<SentFilesEntity>> getAllSentFiles();
#Insert(onConflict = OnConflictStrategy.IGNORE)
void insert(SentFilesEntity file);
#Query("DELETE FROM sent")
void deleteAllRecords();
}
2) SentFilesRepository.java
public class SentFilesRepository {
private SentFilesDao mSentFilesDao;
private LiveData<List<SentFilesEntity>> mAllSentFiles;
SentFilesRepository(Application application) {
AppDatabase db = AppDatabase.getDatabase(application);
mSentFilesDao = db.mSentFilesDao();
mAllSentFiles = mSentFilesDao.getAllSentFiles();
}
LiveData<List<SentFilesEntity>> getAllSentFiles() { return mAllSentFiles; }
public void insert(SentFilesEntity sentFilesEntity) {
new SentFilesRepository.insertAsyncTask(mSentFilesDao).execute(sentFilesEntity);
}
private static class insertAsyncTask extends AsyncTask<SentFilesEntity, Void, Void> {
private SentFilesDao mAsyncTaskDao;
insertAsyncTask(SentFilesDao dao) {
mAsyncTaskDao = dao;
}
#Override
protected Void doInBackground(final SentFilesEntity... params) {
mAsyncTaskDao.insert(params[0]);
Log.e("doInBackground: ", "reached here!"); // no any log in Logcat about this. Weird...
return null;
}
}
}
3) SentFilesEntity.java
#Entity(tableName = "sent")
public class SentFilesEntity {
#PrimaryKey(autoGenerate = true)
#ColumnInfo(name = "id")
public int s_f_id;
#ColumnInfo(name = "name")
public String s_f_name;
#ColumnInfo(name = "type")
public String s_f_type;
#ColumnInfo(name = "status")
public String s_f_operation_status;
#ColumnInfo(name = "dateTime")
public String s_f_time;
}
4) SentFilesViewModel.java
public class SentFilesViewModel extends AndroidViewModel {
private SentFilesRepository mRepository;
private LiveData<List<SentFilesEntity>> mAllSentFiles;
public SentFilesViewModel(Application application) {
super(application);
mRepository = new SentFilesRepository(application);
mAllSentFiles = mRepository.getAllSentFiles();
}
public LiveData<List<SentFilesEntity>> getAllSentFiles() { return mAllSentFiles; }
public void insert(SentFilesEntity sentFile) { mRepository.insert(sentFile); }
}
What I am trying to do:
My program sends selected files from one device to another one through WiFi Direct. As file received or sent, I want to save some information about file (such as file name, file type (image, video, music, APK and etc.), send/receive date and time) and operation (succeeded or failed). I can handle all these things.
What is the problem?
Well, getting all the properties (those I mentioned above), is not end. Main problem is saving that data to database. To do that I created some required files (I shared their codes above). When file sent, I am trying to insert related information to database. But it does not insert it. That is the my problem.
Well, main codes are too long to post here. However I posted my latest codes to my GitHub project and the main logic starts from here and it connected to FileStatisticsActivity.
p.s: Maybe sharing the project link is not ethic, I am so sorry for that.
Thanks.
In my project, I am having trouble writing a createCriteria query with a composite primary key. My Entity class & DAO method are given below -
#Entity
#Table(name="METRICS")
public class Metrics implements Serializable {
private static final long serialVersionUID = -2580493160757497919L;
#EmbeddedId
protected MetricsID metricsID;
#Column(name="PROJ_PERF")
private String proj_perf;
#Column(name="ANALYSIS")
private String analysis;
public String getProj_perf() {
return proj_perf;
}
public void setProj_perf(String proj_perf) {
this.proj_perf = proj_perf;
}
public String getAnalysis() {
return analysis;
}
public void setAnalysis(String analysis) {
this.analysis = analysis;
}
public MetricsID getMetricsID() {
return metricsID;
}
public void setMetricsID(MetricsID metricsID) {
this.metricsID = metricsID;
}
}
#Embeddable
public class MetricsID implements Serializable {
private static final long serialVersionUID = 4691163770334366543L;
#Column(name="PROJECT_ID")
private String project_id;
#Column(name="METRICS_NO")
private int metrics_no;
public String getProject_id() {
return project_id;
}
public void setProject_id(String project_id) {
this.project_id = project_id;
}
public int getMetrics_n0() {
return metrics_no;
}
public void setMetrics_no(int i) {
this.metrics_no = i;
}
}
#Override
#Transactional
public List<Metrics> viewMetrics(String project_id) throws Exception {
List<Metrics> metrics = (List<Metrics>)sessionFactory.getCurrentSession().
createCriteria(Metrics.class).createAlias("metricsID.project_id", "project_id_alias").
add(Restrictions.eqProperty("project_id_alias.project_id", project_id)).list();
return metrics;
}
The error I am getting is - org.hibernate.QueryException: not an association: metricsID.project_id
I searched for several similar examples, and used alias on the suggestion of one of the search results, but it's my first time using an alias. What am I doing wrong?
Why do you need to use an alias? Have you tried to access directly?
Following this example, this code should work
#Override
#Transactional
public List<Metrics> viewMetrics(String project_id) throws Exception {
List<Metrics> metrics =
(List<Metrics>) sessionFactory.getCurrentSession()
.createCriteria(Metrics.class)
.add(Restrictions.eq("metricsID.project_id", project_id))
.list();
return metrics;
}
Im using the entity manager JPA with eclipse link 2.3 &Derby db and I have model with 10 entities and for each entity I need to store 1000 records ,this process takes about 70 sec. I have test it for the same model with 10 entities but with 100 records the all process with the commit take about 1.2 sec which is great.
the bottle neck is the entityManager.getTransaction().commit(); which I do just one time after i persist all the data, the commit take more 95% from the all process.
when I use the JVM monitor I dive in to the commit and I see that one of the class is responsible for almost all commit time ,the class is org.eclipse.persistence.mappings.ManyToManyMapping
http://www.eclipse.org/eclipselink/api/1.0/org/eclipse/persistence/mappings/ManyToManyMapping.html
my entities and the model doesn't have any many to many relationship or use any many to many annotation what could be the reason for the Exponential behavior ?
I have noticed that when I remove this two entities the time was saved by 85%
what is wrong with this entities
The navigation is from person which have cardinality 1 to title award that have cardinality of N i.e one person can have many awards...
#javax.persistence.Entity
#javax.persistence.Table(name = "a3_Person")
public class Person {
#javax.persistence.Id
#javax.persistence.Column(length = 20)
//// Id;
private String person_id;
public String getPerson_id() { return this.person_id; }
public void setPerson_id(String person_id) { this.person_id = person_id; }
#javax.persistence.Column(length = 30)
//// Name;
private String person_name;
public String getPerson_name() { return this.person_name; }
public void setPerson_name(String person_name) { this.person_name = person_name; }
//// Awards;
private List<TitleAward> person_awards;
public List<TitleAward> getPerson_awards() { return this.person_awards; }
public void setPerson_awards(List<TitleAward> person_awards) { this.person_awards = person_awards; }
}
#javax.persistence.Entity
#javax.persistence.Table(name = "a3_TitleAward")
public class TitleAward {
#javax.persistence.Id
#javax.persistence.Column(length = 20)
//// Id;
private String titleaward_id;
public String getTitleaward_id() { return this.titleaward_id; }
public void setTitleaward_id(String titleaward_id) { this.titleaward_id = titleaward_id; }
#javax.persistence.Column(length = 30)
//// Type;
private String titleaward_type;
public String getTitleaward_type() { return this.titleaward_type; }
public void setTitleaward_type(String titleaward_type) { this.titleaward_type = titleaward_type; }
#javax.persistence.Column(length = 30)
//// Category;
private String Cateeheihbadc;
public String getCateeheihbadc() { return this.Cateeheihbadc; }
public void setCateeheihbadc(String Cateeheihbadc) { this.Cateeheihbadc = Cateeheihbadc; }
#javax.persistence.Column()
//// Year;
private String titleaward_year;
public String getTitleaward_year() { return this.titleaward_year; }
public void setTitleaward_year(String titleaward_year) { this.titleaward_year = titleaward_year; }
#javax.persistence.Column()
//// Won;
private Boolean titleaward_won;
public Boolean getTitleaward_won() { return this.titleaward_won; }
public void setTitleaward_won(Boolean titleaward_won) { this.titleaward_won = titleaward_won; }
//// Person;
private Person Pers_fhfgdcjef;
public Person getPers_fhfgdcjef() { return this.Pers_fhfgdcjef; }
public void setPers_fhfgdcjef(Person Pers_fhfgdcjef) { this.Pers_fhfgdcjef = Pers_fhfgdcjef; }
}
There are a number of performance optimization outlined here,
http://java-persistence-performance.blogspot.com/2011/06/how-to-improve-jpa-performance-by-1825.html
ManyToManyMapping is also used for the #OneToMany annotation that uses a #JoinTable, are you using this? In general correctly profiling and understanding a profile can be difficult, so your profile may not be valid.
Please include your code, and a sample of the SQL log, and profile. You can also enable the EclipseLink PerformanceMonitor,
see,
http://www.eclipse.org/eclipselink/documentation/2.4/concepts/monitoring003.htm
If 100 records only takes 1.2s then you could probably break your process into batches of 100 and get 12s instead of 70s. 70s sounds like you have some sort of n^2 issue going on.