We are using jhipster generator for our new project. to store data it we select Postgres as well as elasticsearch, all search operation will perform using elasticsearch.
when we start the application, It use liquibase to upload csv files and dump data into tables.we added number of csv file and made some change on liquibase configuration files as well, but the problem we found right now that it is only dumping csv data into Postgres only, we are not able to find data dump into elasticsearch.
I do some research and found this.
but still, I am struggling with implementation, any advice will be really helpful.
The JHipster Elasticsearch is indexing on every change over the REST resource. See here. This means that all your data which you are inserting over liquibase is not getting indexed. You can use the generator-jhipster-elasticsearch-reindexer to reindex data which is already in the db.
Related
I have java vertx web application which uses mysql as database also take apache kafka in use. To increase the speed of search query i was thinking to add apache solr in my application and store few tables in it.
But i m not able to find a way to keep those few table stored in mysql data in sync with apache solr.
i try to read DIH documentation but didnt found any solution
My requirement is to get polarion data and store that to our sql server database.
I went through Polarion SDK document and I feel webservice is the way to do that....
Which is the best way to read and store specific data from polarion to SQL server.
The webservice is very slow and depending on your size it will not be practicable to export your data with help of the webservice.
However the data in Polarion is stored in an SVN-repository in form of small .xml-files. So you can read these XML files directly from the repository.
As Polarion is not stored in a database compatible format, you need to setup your own DB-schema and the transform from the XML-files should be straight forward.
You can either checkout a complete Polarion Project or you retrieve the files on demand via http(s) second approach will be slightly slower again.
I'm currently building a Spring Boot Service with a h2 in-memory database.
This Database acts as an cache for a part of the data on a central db2 database with a different database schema.
Now when the Spring boot service starts it needs to populate the h2 database with the latest data from the central database.
How can I do this in the best way performance wise?
I'm currently looking to create an different data-source in my service to first get the data and then save the data to the h2.
This doesn't feel like a good solution and it would take quite a long time to populate the database.
If you want to use H2 instead of your DB2 database ... and if you don't want to re-create the database each time you run your app ...
... then consider using an H2 file, instead of in-memory:
http://www.h2database.com/html/features.html
jdbc:h2:[file:][<path>]<databaseName>
jdbc:h2:~/test
jdbc:h2:file:/data/sample
jdbc:h2:file:C:/data/sample (Windows only)
You can "initialize" the file whenever you want (perhaps just once).
Performance should be excellent.
Per your update:
I still need to access the central db to get the latest data in the
fastest way possible. The central db needs to stay for other services
also accessing this
The "fastest" way to get the very latest data ... is to query the central db directly. Period - no ifs/ands/buts.
But, if for whatever reason, you want to "cache" a subset of "recent" data ... then H2 is an excellent choice.
And if you don't want to "rebuild" each time you start your H2 database, then save H2 to a file instead of making it in-memory.
The performance difference between H2:mem and H2:file is small, compared to the network overhead of querying your central db.
'Hope that helps...
What are the options to index large data from Oracle DB to elastic search cluster? Requirement is to index 300Million records one time into multiple indexes and also incremental updates having around approximate 1 Million changes every day.
I have tried JDBC plugin for elasticsearch river/feeder, both seems to be running inside or require locally running elastic search instance. Please let me know if there is any better option for running elastic search indexer as a standalone job (probably java based). Any suggestions will be very helpful.
Thanks.
We use ES as a reporting db and when new records are written to SQL we take the following action to get them into ES:
Write the primary key into a queue (we use rabbitMQ)
Rabbit picks up the primary key (when it has time) and queries the relation DB to get the info it needs and then writes the data into ES
This process works great because it handles both new data and old data. For old data just write a quick script to write 300M primary keys into rabbit and you're done!
there are many integration options - I've listed out a few to give you some ideas, the solution is really going to depend on your specific resources and requirements though.
Oracle Golden Gate will look at the Oracle DB transaction logs and feed them in real-time to ES.
ETL for example Oracle Data Integrator could run on a schedule and pull data from your DB, transform it and send to ES.
Create triggers in the Oracle DB so that data updates can be written to ES using a stored procedure. Or use the trigger to write flags to a "changes" table that some external process (e.g. a Java application) monitors and uses to extract data from the Oracle DB.
Get the application that writes to the Oracle DB to also feed ES. Ideally your application and Oracle DB should be loosely coupled - do you have an integration platform that can feed the messages to both ES and Oracle?
I am working on a Java program that connects to a databases and serves as a design and data manipulation tool. I eventually want to port my program to an XML database of some kind.
More specifically, I want to be able to use Java to upload XSD (representing a database) that I generate to an actual XML database (like Sedna) and then connect to this database and query/update it.
I have been exploring Sedna and the XQJ API and I don't see how to do this. Can it be done through Java?
Edit I don't have any data I want to map to XML or a database. I have a database structure specified in XSD, and I just want to create the corresponding database on a server somewhere.
There are at least two frameworks worth trying for XML to database mapping:
Castor XML http://www.castor.org
Hibernate http://docs.jboss.org/hibernate/orm/3.3/reference/en/html/xml.html