I've been working with Java JDBC (mainly in desktop apps) for a long time and I'm realizing that many operations that I perform could be improved and simplified. For example:
Hardcoding SQL statements directly into Java is not very practical.
Replacing variables with "?" in JDBC is nice but it would be much better to use real variable names like "USER-NAME" or something like that.
Executing multiple update statements at once would be very cool.
In an effort to improve JDBC, I decided to write my own tool but before I reinvent the wheel I would like to know if there is any Java utility that is able to:
Read and execute .sql scripts, preferably stored inside the JAR of my application.
Define variables in these scripts, preferably with real names and not with the "?" character.
Run query (SELECT) and update (CREATE, INSERT, DELETE, ...) statements from these scripts.
Execute multiple update statements in one single method call. This could let me, for example, run DDL and DML scripts to initialize databases.
I know about JDBC ScriptRunner but it's not complete enough. Is there something better out there? I honestly think such a tool like this would be very useful.
Note: I don't mind to import a lib.
I use Warework just for that. It's a big JAR but I think it does what you're looking for. I'll show you how it works:
1- Create this directory structure in the source folder of your project:
/META-INF/system/statement/sql
2- In the "/META-INF/system" directory, create a file named "pool-service.xml" with this content:
<?xml version="1.0" encoding="UTF-8"?>
<proxy-service xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://repository.warework.com/xsd/proxyservice-1.0.0.xsd">
<clients>
<client name="c3p0-client" connector="com.warework.service.pool.client.connector.C3P0Connector">
<parameter name="driver-class" value="com.mysql.jdbc.Driver" />
<parameter name="jdbc-url" value="jdbc:mysql://host:port/database-name" />
<parameter name="user" value="the-user-name" />
<parameter name="password" value="the-password" />
<parameter name="connect-on-create" value="true" />
</client>
</clients>
</proxy-service>
In this file, replace the values of the parameters with the ones you need to connect with your database. Keep "connect-on-create" equals to "true".
3- Write your .sql scripts in the "/META-INF/system/statement/sql" directory. You can write SELECT statements like this (one statement per file):
find-user.sql
SELECT * FROM HOME_USERS A WHERE A.ID = ${USER_ID}
and UPDATE statements like this (one or multiple statements per file):
create-user.sql
INSERT INTO HOME_USERS (ID, NAME) VALUES (${USER_ID}, ${USER_NAME});
INSERT INTO ACTIVE_USERS (ID) VALUES (${USER_ID});
4- To connect with the database, perform this:
// "Test.class" must be any class of your project (the same project where /META-INF/system directory exists).
// Do not change "full" and "relational-database" strings.
// If you change "system" for "test", then the directory will be /META-INF/test.
RDBMSView ddbb = (RDBMSView) ScopeFactory.createTemplate(Test.class, "full", "system").getObject("relational-database");
// Connect with the database.
ddbb.connect();
5- Run a SELECT statement from a .sql file like this:
// Values for variables in the SELECT statement.
Hashtable values = new Hashtable();
// Set variables to filter the query.
values.put("USER_ID", new Integer(8375));
// Read '/META-INF/system/statement/sql/find-user.sql', replace variables and run.
// -1 values are for pagination (first -1 is the page, second -1 is the max rows per page).
ResultSet result = (ResultSet) ddbb.executeQueryByName("find-user", values, -1, -1);
6- Run UPDATE statements from a .sql file like this:
// Values for variables in the UPDATE statements.
Hashtable values = new Hashtable();
// Set variables for the update statement.
values.put("USER_ID", new Integer(3));
values.put("USER_NAME", "Oompa Loompa");
// Read '/META-INF/system/statement/sql/create-user.sql', replace variables and run.
// ';' is the character that separates each statement.
ddbb.executeUpdateByName("create-user", values, new Character(';'));
The RDBMSView class provides these methods plus connect/disconnect, commit, rollback, ... You can also run statements directly from String objects.
About named parameters here are a couple of solutions I found around
http://www.javaworld.com/javaworld/jw-04-2007/jw-04-jdbc.html?page=1
Named parameters in JDBC
Extracting statements from a script is a quite simple task (but not trivial) if you limit yourself to SELECT/UPDATE/INSERT/DELETE statements. Just split at any ";" character wich isn't between two "'" characters. It becomes a more complex parsing problem if you admit also CREATE TRIGGERS or CREATE PROCEDURE etc. to your script since they have have ";" as part of their sintax.
I personally think it's safer to keep DDL out of the equation and manage database creation with some external tool like Liquibase
Related
I am new to JOOQ and love learning and using it.
I am at the point where I want to do some testing, but instead of testing
on the 'real' database I want to use a 'copy' of that database.
When using jdbc all it really took was changing the database name in the
create statement and using that name when connecting to the database.
I quickly discovered that anything I tried to write to my test database
was ending up in the production database
In the JOOQ documentation it looked like I could solve the problem with some mapping.
I added a Settings class and got the DSLContext using the settings, as shown below
My base database name is 'kpi'.
Connection conn = JooqUtil.getConnection("user", "pw", kpitest, hostip, sb);
Settings settings = new Settings()
.withRenderMapping(new RenderMapping()
.withSchemata(
new MappedSchema().withInput(kpi)
.withOutput(kpitest)));
// Add the settings to the DSLContext
if (sb.length() == 0) {
dsl = DSL.using(conn, SQLDialect.MYSQL, settings);
}
The above resulted in seeing the testName database being used,
BUT access to the tables and fields was using the productionName.
Going back to the documentation it looks like there is a way to map the tables
also but it looks like a lot of work.
Further reading, I found some settings in the configuration file using schemata
My current JOOQ configuration xml file contains
<inputSchema>kpi</inputSchema>
It looks like I should change it to something like
<inputSchema></inputSchema> << maybe drop altogether
<schemata>
<schema>
<inputSchema>kpi</inputSchema>
<outputSchema>kpi</outputSchema>
</schema>
<schema>
<inputSchema>kpi</inputSchema>
<outputSchema>kpitest</outputSchema>
</schema>
</schemata>
In reading the manual I was not sure if DEV and PRODUCTION names had literal significance or not.
What would help is a (real world) example where there was a production and test database,
or one develpment database per developer, and how the tables and fields were accessed
if there was a change in syntax. A change in syntax would be difficult for doing unit type testing though.
Thanks for any quidance and links to examples
I am using an APACHE DERBY database, and basing my database interactions on EntityManager, and I don't want to use JDBC class to build a query to change my tables' names (i just need to put a prefix to each new user to the application, but have the same structure of tables), such as:
//em stands for EntityManager object
Query tableNamesQuery= em.createNamedQuery("RENAME TABLE SCHEMA.EMP_ACT TO EMPLOYEE_ACT");
em.executeUpdate();
// ... rest of the function's work
// The command works from the database command prompt but i don't know how to use it in a program
//Or as i know you can't change system tables data, but here's the code
Query tableNamesQuery= em.createNamedQuery("UPDATE SYS.SYSTABLES SET TABLENAME='NEW_TABLE_NAME' WHERE TABLETYPE='T'");
em.executeUpdate();
// ... rest of the function's work
My questions are :
This syntax is correct?
Will it work?
Is there any other alternative?
Should I just use the SYS.SYSTABLES and find all the tables that has 'T' as tabletype and alter their name their, will it change the access name ?
I think you're looking for the RENAME TABLE statement: http://db.apache.org/derby/docs/10.10/ref/rrefsqljrenametablestatement.html
Don't just issue update statements against the system catalogs, you will corrupt your database.
I want to insert exchange.body to a database table for one of the condition of my route.
Is there any example/tutorial of camel-jdbc component to insert message body?
Can I import the SQL statement itself and pass exchange.body to it?
I looked at http://camel.apache.org/jdbc.html example, but could not understand it.
Here Spring example is confusing for me. I didn't get why is it setting the body as SQL query and again importing some query from the class path. (There is no insert query example mentioned here.)
If you want to insert using the same statement (changing the parameters only) - use SQL component.
If you want to insert using arbitrary SQL statement into the component - use JDBC component.
SQL component usage:
from("direct:start").to("sql:insert into table foo (c1, c1) values ('#','#')");
com.google.common.collect.Lists;
producerTemplate.sendBody("direct:start", Lists.newArrayList("value1","value2"));
JDBC component usage:
from("direct:start").to("jdbc:dataSource");
producerTemplate.sendBody("direct:start", "insert into table foo (c1, c1) values ('value1','value2')");
You probably need to do some restructure of your payload before inserting it anyway, so there should probably be no issue to do a transformation using whatever method in Camel to set the body to the appropriate INSERT statement.
The important thing is what kind of payload structure your incoming message have. In the basic case - it's a string - it should be fairly simple
// In a Java bean/processor before the JDBC endpoint.
// Update: make sure to sanitize the payload from SQL injections if it contains user inputs or external data not generated by trusted sources.
exchange.getIn().setBody("INSERT INTO MYTABLE VALUES('" + exchange.getIn().getBody(String.class) + "', 'fixedValue', 1.0, 42)");
In case your message contains complex data structures, this code will of course be more complex, but it's pretty much the same way regular application will generate SQL queries.
The classpath example you are refering to
<jdbc:embedded-database id="testdb" type="DERBY">
<jdbc:script location="classpath:sql/init.sql"/>
</jdbc:embedded-database>
Simply shows how to test the JDBC component by starting a Database server embedded (Apache Derby) and populate it with some initial data (the sql/init.sql file). This part is not really part of the core jdbc component, but simply in the documentation to get up and running a sample without needing to configure a DB server and setup the JDBC connection properties.
That said, you might want to use the SQL component for more complex scenarios.
I have a basic setup where the database is read by multiple web applications, and periodically I have a batch application which does a lot of writing. During the writing, the peroformance of the web-apps degrade heavily (their database reads are very slow).
The env. is MySQL db using MYISAM engine, the batch application is a Java SE app, using spring-batch and SimpleJDBCTemplate to issue SQL commands via JDBC. I found that MySQL has a parameter that lowers the priority of write operations on MYISAM engine: low_priority_updates. To quote the docs, amongs others, you can "SET LOW_PRIORITY_UPDATES=1 to change the priority in one thread". I opted for this because it's easiest from the config standpoint of my application. What I've done is configured my DataSource such that it exectutes that "SET ..." for each connection it opens, like so:
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<!-- other props omitted -->
<property name="connectionInitSqls">
<list>
<value>SET low_priority_updates="ON"</value>
</list>
</property>
</bean>
Now my question is, how do I actually check that an SQL issued via this datasource does actually run with low priority? If I do SHOW FULL PROCESSLIST in MySQL while the inserts are happening it just tell me what SQL they're executing, nothing about the priority:
If I check the server variables low_priority_updates is "OFF" but that's just the server variable, it sais nothing about the thread-local value.
So again, is there any actual way to check if per query/thread low_priority_updates's values are taken into account?
By issuing SET LOW_PRIORITY_UPDATES=1 command, you are affecting the variable value for the session. Therefore it is possible to see this by checking the value of the variable in the session.
I know of two ways to do it:
1- SHOW SESSION VARIABLES LIKE 'low_priority_dapdates'
this shows ON/OFF
2- select ##session.low_priority_updates
this gives 0/1
Important: the above statements/calls will show you the values of the variables in the session where they run.
Therefore, you will need to run them using the connections themselves in order to see the values. I don't know of a way in MySQL where you can select values for variables that belong to another session.
If you would like to see them as a list, you might need to do a work around by creating a table and logging that info yourself. for example:
CREATE TABLE `mydb`.`my_low_priority_updates` (
`connection_id` INT ,
`low_priority_updates_value` INT NOT NULL
)
ENGINE = MyISAM;
then you need a statement that inserts the connection id and the value into the table:
insert into my_low_priority_updates(connection_id,low_priority_updates_value)
select connection_id(),##session.low_priority_updates
from dual
where not exists (select 1 from my_low_priority_updates where connection_id=connection_id())
you can put this statement in a procedure and make sure its called, or add it in a trigger on a table that you know gets updated/inserted into.
after that, querying the my_low_priority_updates table later will show you the values of the variable in each connection.
Can anyone suggest a good way of detecting if a database is empty from Java (needs to support at least Microsoft SQL Server, Derby and Oracle)?
By empty I mean in the state it would be if the database were freshly created with a new create database statement, though the check need not be 100% perfect if covers 99% of cases.
My first thought was to do something like this...
tables = metadata.getTables(null, null, null, null);
Boolean isEmpty = !tables.next();
return isEmpty;
...but unfortunately that gives me a bunch of underlying system tables (at least in Microsoft SQL Server).
There are some cross-database SQL-92 schema query standards - mileage for this of course varies according to vendor
SELECT COUNT(*) FROM [INFORMATION_SCHEMA].[TABLES] WHERE [TABLE_TYPE] = <tabletype>
Support for these varies by vendor, as does the content of the columns for the Tables view. SQL implementation of Information Schema docs found here:
http://msdn.microsoft.com/en-us/library/aa933204(SQL.80).aspx
More specifically in SQL Server, sysobjects metadata predates the SQL92 standards initiative.
SELECT COUNT(*) FROM [sysobjects] WHERE [type] = 'U'
Query above returns the count of User tables in the database. More information about the sysobjects table here:
http://msdn.microsoft.com/en-us/library/aa260447(SQL.80).aspx
I don't know if this is a complete solution ... but you can determine if a table is a system table by reading the table_type column of the ResultSet returned by getTables:
int nonSystemTableCount = 0;
tables = metadata.getTables(null, null, null, null);
while( tables.next () ) {
if( !"SYSTEM TABLE".equals( tables.getString( "table_type" ) ) ) {
nonSystemTableCount++;
}
}
boolean isEmpty = nonSystemTableCount == 0;
return isEmpty;
In practice ... I think you might have to work pretty hard to get a really reliable, truly generic solution.
Are you always checking databases created in the same way? If so you might be able to simply select from a subset of tables that you are familiar with to look for data.
You also might need to be concerned about static data perhaps added to a lookup table that looks like 'data' from a cursory glance, but might in fact not really be 'data' in an interesting sense of the term.
Can you provide any more information about the specific problem you are trying to tackle? I wonder if with more data a simpler and more reliable answer might be provided.
Are you creating these databases?
Are you creating them with roughly the same constructor each time?
What kind of process leaves these guys hanging around, and can that constructor destruct?
There is certainly a meta data process to loop through tables, just through something a little more custom might exist.
In Oracle, at least, you can select from USER_TABLES to exclude any system tables.
I could not find a standard generic solution, so each database needs its own tests set.
For Oracle for instance, I used to check tables, sequences and indexes:
select count(*) from user_tables
select count(*) from user_sequences
select count(*) from user_indexes
For SqlServer I used to check tables, views and stored procedures:
SELECT * FROM sys.all_objects where type_desc in ('USER_TABLE', 'SQL_STORED_PROCEDURE', 'VIEW')
The best generic (and intuitive) solution I got, is by using ANT SQL task - all I needed to do is passing different parameters for each type of database.
i.e. The ANT build file looks like this:
<project name="run_sql_query" basedir="." default="main">
<!-- run_sql_query: -->
<target name="run_sql_query">
<echo message="=== running sql query from file ${database.src.file}; check the result in ${database.out.file} ==="/>
<sql classpath="${jdbc.jar.file}"
driver="${database.driver.class}"
url="${database.url}"
userid="${database.user}"
password="${database.password}"
src="${database.src.file}"
output="${database.out.file}"
print="yes"/>
</target>
<!-- Main: -->
<target name="main" depends="run_sql_query"/>
</project>
For more details, please refer to ANT:
https://ant.apache.org/manual/Tasks/sql.html