how to index data from database using solr - java

I configured Import Handler for fatching data from MySql data base. In the solr example which i downloaded from net in C:\apache-solr-1.4.0\example\example-DIH\solr\db\lib there is hsqldb-1.8.0.10.jar.
I want to know:
since i am using MySQL do i need to
put jar for MySql.

please be sure to check out the Solr Wiki as it very much describes the process of indexing data from a MySQL database.
As for you question regarding where to put the MySQL connector jar (in my case it is called the "mysql-connector-java-5.1.12-bin.jar")... you will need to place this jar in the C:\apache-solr-3.1.0\example\lib\ folder.
Hope that helps!

Related

HSQLDB data storage as file in Spring Boot Application, unable to visualize

I am trying my hands on spring boot application. I planned to use HSQLDB for the database.
Purpose: Create User Table, Insert, Update, Delete data
I created User Entity, User dao, and saved user data in the user entity.
Everything is working fine.
What I want is to see the data in the table as we can see it for MySQL.
I tried to use razorSQL, Dbeaver but I can't see tables.
application.properties
spring.jpa.hibernate.ddl-auto: update
spring.jpa.hibernate.dialect=org.hibernate.dialect.HSQLDialect
spring.jpa.database: HSQL
spring.jpa.show-sql: true
spring.hsql.console.enabled: true
spring.datasource.url=jdbc:hsqldb:file:data/mydb
spring.datasource.username=sa
spring.datasource.password=
spring.datasource.driverClassName=org.hsqldb.jdbcDriver
I can see the User table data in the browser:
Files created in the data folder:
I have googled a lot but nothing helps.
Questions:
Can we see the data stored in HSQLDB(when running) as we can see for the MySQL in PHPMyAdmin?
In which file data is stored, I have seen that in the script file it saves all the statements (insert, delete etc). Do we have a separate file to store the data?
what is the use of tmp folder created?
Let me know if you need more details. I need to be clear on this. It has taken a lot of time still I am not satisfied
I am able to visualize the data of hsqldb with the help of hsqldb.jar
Assuming
Database folder named "data" is created which contains files with mydb.log, mydb.properties, mydb.script, mydb.tmp
Steps to visualize when using it as fileDb.
1. Download HSQLDB jar.
2. Extract in the folder where we have "data" folder(it contains database files) database files generated ("data" folder).
3. Now we are in the folder, where database folder is created. Run this command "java -cp hsqldb-2.4.1/hsqldb/lib/hsqldb.jar org.hsqldb.util.DatabaseManagerSwing" here "hsqldb-2.4.1" is the downloaded hsqldb folder. It will open up a UI.
4. In this UI, make a new connection, select type as "HSQL Database Engine Standalone" put URL as "jdbc:hsqldb:file:data/mydb" (here data is the folder and mydb is the DB name), give user and password as defined in application properties, then say ok. It should connect. (Maken sure the path to the file DB is relative to the folder from where we opened the UI)
Let me know in case anyone is getting errors
You can run HSQLDB as a Server and connect to it simultaneously from your Spring app and from a database utility such as dBeaver. The connection URL will look like jdbc:hsqldb:hsql://localhost/mydb. This is very similar to the way MySQL is used.
Detailed coverage is here: http://hsqldb.org/doc/guide/listeners-chapt.html but see the introduction to the Guide first. You can also consult various step-by-step HSQLDB tutorials on the web.

Compile Dropwizard Code without migration.xml

I have a finalized database in SQL SERVER containing 50+ tables in it and needed to connect it with Dropwizard Code.
I am new to JAVA so my conception about Migrations.xml is it is used to create the tables in database or if any change in database is needed it will be updated through migrations.xml.
So if i don't need any change in database (as told earlier it is finalized).
Can i skip this migrations.xml file?
Need some experts advice please.
If you are handling your database changes elsewhere, then you have no need for any migration xml files within your dropwizard project. It's an optional module, you don't need to use it. You don't even need to include the dropwizard-migrations dependency if you don't want to include database updates in your dropwizard project. You can still connect to your database fine within dropwizard. The docs provide examples using modules dropwizard-jdbi and dropwizard-hibernate.
To connect to your database, add the appropriate code the your java configuration file and yml config as explained in the docs.
jdbi
http://www.dropwizard.io/0.9.2/docs/manual/jdbi.html
hibernate
http://www.dropwizard.io/0.9.2/docs/manual/hibernate.html

How to see all tables in my h2 database at localhost:8082?

I use JDBC and created h2 database called usaDB from sql script. Then I filled all tables with jdbc.
The problem is that after I connect to usaDB at localhost:8082 I cannot see on the left tree
my tables. There is only INFORMATION_SCHEMA database and rootUser which I specified creating usaDB.
How to view the content of tables in my h2 database?
I tried query SELECT * FROM INFORMATION_SCHEMA.TABLES.
But it returned many table names except those I created. My snapshot:
I had the same issue and the answer seems to be really stupid: when you type your database name you shouldn't add ".h2.db" suffix, for example, if you have db file "D:\somebase.h2.db" your connection string should be like "jdbc:h2:file:/D:/somebase". In other way jdbc creates new empty database file named "somebase.h2.db.h2.db" and you see what you see: only system tables.
You can use the SHOW command:
Using this command, you can lists the schemas, tables, or the columns of a table. e.g.:
SHOW TABLES
This problem drove me around the twist and besides this page I read many (many!) others until I solved it.
My Use Case was to see how a SpringBatch project created in STS using :: Spring Boot :: (v1.3.1.RELEASE) was going to behave with the H2 database; to do the latter, I needed to be able to get the H2 console running as well to query the DB results of the batch run.
This is what I did and found out:
Created an Web project in STS using Spring Boot:
Added the following to the pom.xml of the latter:
Added a Spring configuration file as follows to the project:
This solves the Web project deficiencies in STS. If you run the project now, you can access the H2 console as follows: http://localhost:8080/console
Now create a SpringBatch project in STS as follows (the alternative method creates a different template missing most of the classes for persisting data. This method creates 2 projects: one Complete, and the other an initial. Use the Complete in the following.):
The SpringBatch project created with STS uses an in memory H2 database that it CLOSES once the application run ends; once you run it, you can see this in the logging output.
So what we need is to create a new DataSource that overrides the default that ships with the project (if you are interested, just have a look at the log messages and you will see that it uses a default datasource...this is created from:
o.s.j.d.e.EmbeddedDatabaseFactory with the following parameters:
Starting embedded database: url='jdbc:hsqldb:mem:testdb', username='sa')
So, it starts an in memory, and then closes it. You have no chance of seeing the data with the H2 console; it has come and gone.
So, create a DataSource as follows:
You can of course use a properties file to map the parameters, and profiles for different DataSource instances...but I digress.
Now, make sure you set the bit that the red arrow in the picture is pointing to, to a location on your computer where a file can be persisted.
Running the SpringBatch (Complete project) you should now have a db file in that location after it runs (persisting Person data)
Run the Web project you configured previously in these steps, and you WILL :=) see your data, and all the Batch job and step run data (et voila!):
Painful but rewarding. Hope it helps you to really BOOTSTRAP :=)
I have met exactly this problem.
From what you describe, I suppose that you connect your jdbc with the "real" h2 server, but you are connecting on web application to database by the wrong mode (embedded-in-memory mode, aka h2mem). It means that h2 will create a new database in-memory, instead of using your true database stored elsewhere.
Please make sure that when you connect to this database, you use the mode Generic H2 (Server), NOTGeneric H2 (Embedded). You can refer to the picture below.
Version of jar file and installed h2 database should be same.
If in case you have created and populated H2 database table using maven dependency in spring boot, then please do change the JDBC URL as jdbc:h2:mem:testdb while connecting to H2 using web console.
It is an old question, but I came across the same problem. Eventually I found out that the default JDBC URL is pointing a test server rather than my application. After correcting it, I could access the right DB.
I tried with both Generic H2 (Embedded) and the Generic H2 (Server) options, both worked as long as the JDBC URL: is provided correctly.
In grails 4.0.1 the jdbc URL for development is jdbc:h2:mem:devDb. Check your application.yml file for the exact URL.
For the people who are using H2 in embedded(persistent mode) and want to "connect" to it from IntelliJ(other IDEs probably apply too).
Using for example jdbc url as follows: jdbc:h2:./database.h2
Note, that H2 does not allow implicit relative paths, and requires adding explicit ./
Relative paths are relative to current workdir
When you run your application, your workdir is most likely set to your project's root dir
On the other hand, IDE's workdir is most likely not your project's root
Hence, in IDE when "connecting" to your database you need to use absolute path like: jdbc:h2:/Users/me/projects/MyAwesomeProject/database.h2
For some reason IntelliJ by default also adds ;MV_STORE=false. It disables MVStore engine which in fact is currently used by default in H2.
So make sure that both your application and your IDE use the same store engine, as MVStore and PageStore have different file layouts.
Note that you cannot "connect" to your database if your application is using it because of locking. The other way around applies too.
In my case the issue was caused by the fact that I didn't set the h2 username, password in java. Unfortunatelly, Spring didn't display any errors to me, so it was not easy to figure out. Adding this lines to dataSource method helped me fix the issue:
dataSource.setUsername("sa");
dataSource.setPassword("");
Also, I should have specified the schema when creating tables in schema.sql
Selecting Generic H2 (Server) solved for me. We tempted to use default Generic H2 (Embedded) which is wrong.

How to index data in single solr app using tomcat server

I am working on single solr app. I downloded solr exampple code for net, which is working fine while running on jetty server.It is having data which are to be indexed in C:\apache-solr-1.4.0\example\exampledocs and the indexes are stored in C:\apache-solr-1.4.0\example\solr\data, using jetty server indexes are created using command java -jar post.jar *.xml. Now i want to know how can i achieve this using Tomcat. do i need to change the configuration to change the path for indexe storage and for xml files storage. how data will b indexed so that i would able to search it.
If I understand your question correctly, you'll want to use the -Durl flag when running post.jar, e.g.:
java -jar -Durl=http://localhost:8080/solr/update post.jar solr.xml monitor.xml
In solrconfig.xml you can mention the path that has to hold the index
<dataDir>${solr.data.dir:}</dataDir>
I think you just have to read more from SOLR documentation, and click through what you have in the package.
There is an tomcat deployment doc in solr wiki:
http://wiki.apache.org/solr/SolrTomcat
And the war file is in the dist folder you've downloaded.
How to search it? There is no simple answer. I suggest you read more on the solr wiki. Find out what is a handler, what is the difference between dismax handler and standard handler, how schema.xml defines the database.

Creating a database changelog xml file from an existing database (including stored procs) using Liquibase

Is it possible to create an initial database changelog xml file from the existing state of the database?
I believe I've generated the schema using generateChangeLog, but it doesn't seem to return the stored procedures (or the data).
I'm using SQL Sever 2008
You can return the data using a diffTypes flag that includes "DATA". See http://www.liquibase.org/documentation/diff.html.
Liquibase cannot currently output stored procedures, however. For that you will need to use a different tool and include them in the generated changelog using the tag.
download liquibase.jar , database driver and save to one directory (Ex:/home/mySystem/liquibase), in the command line change the directory to (/home/mySystem/liquibase) and run the below mentioned command
java -jar liquibase.jar --driver=org.postgresql.Driver --classpath=postgresql-42.1.3.jar --changeLogFile=db.changelog.xml --url="jdbc:postgresql://localhost:5432/<databasename>" --username=<username> --password=<password> update
Reference link:
http://www.liquibase.org/documentation/generating_changelogs.html
I just yesterday discovered SQL Power Architect, which seems to be able to generate Liquibase configurations:
http://www.sqlpower.ca/page/architect
For more info on this combination see this blog post:
http://blog.mgm-tp.com/2010/11/data-modeling-part2/

Categories