I have java vertx web application which uses mysql as database also take apache kafka in use. To increase the speed of search query i was thinking to add apache solr in my application and store few tables in it.
But i m not able to find a way to keep those few table stored in mysql data in sync with apache solr.
i try to read DIH documentation but didnt found any solution
Related
I'm currently building a Spring Boot Service with a h2 in-memory database.
This Database acts as an cache for a part of the data on a central db2 database with a different database schema.
Now when the Spring boot service starts it needs to populate the h2 database with the latest data from the central database.
How can I do this in the best way performance wise?
I'm currently looking to create an different data-source in my service to first get the data and then save the data to the h2.
This doesn't feel like a good solution and it would take quite a long time to populate the database.
If you want to use H2 instead of your DB2 database ... and if you don't want to re-create the database each time you run your app ...
... then consider using an H2 file, instead of in-memory:
http://www.h2database.com/html/features.html
jdbc:h2:[file:][<path>]<databaseName>
jdbc:h2:~/test
jdbc:h2:file:/data/sample
jdbc:h2:file:C:/data/sample (Windows only)
You can "initialize" the file whenever you want (perhaps just once).
Performance should be excellent.
Per your update:
I still need to access the central db to get the latest data in the
fastest way possible. The central db needs to stay for other services
also accessing this
The "fastest" way to get the very latest data ... is to query the central db directly. Period - no ifs/ands/buts.
But, if for whatever reason, you want to "cache" a subset of "recent" data ... then H2 is an excellent choice.
And if you don't want to "rebuild" each time you start your H2 database, then save H2 to a file instead of making it in-memory.
The performance difference between H2:mem and H2:file is small, compared to the network overhead of querying your central db.
'Hope that helps...
i have a spring boot Web application with Spring Data JPA and Hibernate and a want to write a Java handler to check in time intervals the fill level of my postgresql database and delete old data accordingly. for example if it reach a max of 10 GB the database has to delete the old data. Is there any Java Library or any suggestion for this purpose? I found only SELECT statements to perform directly on the PostgreSQL Database (https://wiki.postgresql.org/wiki/Disk_Usage)
We are using jhipster generator for our new project. to store data it we select Postgres as well as elasticsearch, all search operation will perform using elasticsearch.
when we start the application, It use liquibase to upload csv files and dump data into tables.we added number of csv file and made some change on liquibase configuration files as well, but the problem we found right now that it is only dumping csv data into Postgres only, we are not able to find data dump into elasticsearch.
I do some research and found this.
but still, I am struggling with implementation, any advice will be really helpful.
The JHipster Elasticsearch is indexing on every change over the REST resource. See here. This means that all your data which you are inserting over liquibase is not getting indexed. You can use the generator-jhipster-elasticsearch-reindexer to reindex data which is already in the db.
What are the options to index large data from Oracle DB to elastic search cluster? Requirement is to index 300Million records one time into multiple indexes and also incremental updates having around approximate 1 Million changes every day.
I have tried JDBC plugin for elasticsearch river/feeder, both seems to be running inside or require locally running elastic search instance. Please let me know if there is any better option for running elastic search indexer as a standalone job (probably java based). Any suggestions will be very helpful.
Thanks.
We use ES as a reporting db and when new records are written to SQL we take the following action to get them into ES:
Write the primary key into a queue (we use rabbitMQ)
Rabbit picks up the primary key (when it has time) and queries the relation DB to get the info it needs and then writes the data into ES
This process works great because it handles both new data and old data. For old data just write a quick script to write 300M primary keys into rabbit and you're done!
there are many integration options - I've listed out a few to give you some ideas, the solution is really going to depend on your specific resources and requirements though.
Oracle Golden Gate will look at the Oracle DB transaction logs and feed them in real-time to ES.
ETL for example Oracle Data Integrator could run on a schedule and pull data from your DB, transform it and send to ES.
Create triggers in the Oracle DB so that data updates can be written to ES using a stored procedure. Or use the trigger to write flags to a "changes" table that some external process (e.g. a Java application) monitors and uses to extract data from the Oracle DB.
Get the application that writes to the Oracle DB to also feed ES. Ideally your application and Oracle DB should be loosely coupled - do you have an integration platform that can feed the messages to both ES and Oracle?
I have created a java application which stores data into MySQL database.
For that I have done the following things:
I have installed MySQL database on my computer.
I have created a database on that MySQL server.
I have created a table in the database with the required schema.
Now I want to deliver this application to various clients but my clients are not technical persons and I don't want to give instructions to each of my client to do the above three steps.
How can I integrate some functionality into my app so that it can do atleast step 2 and step 3 automatically so that the client needs to install only MySQL database.
It would be much better if the code can install the MySQL database automatically from the setup file attached with the application.
How the applications available in the market manage information?
For 2 and 3 you just need two SQL statements to run during installation: CREATE DATABASE and CREATE TABLE.
As an alternative I would suggest you to use SQLite for which your clients wouldn't need to install any database servers.
Personally, I like how Confluence (for example) deals with that:
The Confluence installation includes an embedded HSQLDB database, supplied for evaluation purposes. This is what you get when using the automatic installer on Windows.
As documented, the embedded database is Not Suitable for Production Instances of Confluence so they suggest to use an external database for production and provide detailed Database Setup Guides (installation, schema and user creation) for MySQL, PostgreSQL, Oracle, DB2, SQL Server and generic instructions for others databases.
The application will take care of creating the tables on startup if the schema is empty.
For those who prefer to create the tables manually, they provide a database creation script.
When upgrading to a higher version of Confluence, the Confluence application takes care of the schema update.
you can store data in two ways
using xampp
1st is using xampp/lampp/wampp and go to insert and just insert
using php
php used to insert data with mysql query
For steps 2 and 3 better create an OS shell script (bat or bash) that will execute mysql cli tool to create database and schema from your file
mysql -u root -p superpwd < create_database.sql
the create_database.sql better to create with help of mysqldump cli tool from your own database
Later you can include this script into your MySQL bundled installation.
I would think that you could do a data dump via phpAdmin which should script out the tables of the database along with insert statements for the actual data. I'm not a Java developer, but I think you should be able to use the functionality of the Java libraries that give you database access to turn back around and load your scripted out database as a file, read that file and then execute it against a database that you create via code.
Hope this helps for you.
For reference, here is how to create a mySql database via a command line .