Description of code
Database connection
I try to to store Java object locally database without use external database.
For this I use JDBC with H2 via Hibernate :
/**
* #param connection the connection to set
*/
public static void setConnectionHibernate() {
Properties connectionProps = new Properties();
connectionProps.put("user", "sa");
try {
Class.forName("org.h2.Driver");
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
url = "jdbc:h2:mem:db1;DB_CLOSE_DELAY=-1;MODE=MySQL;";
}
Query
I store the PROCEDURE in String with this code :
static final String CREATE_PROCEDURE_INITPSEUDOS = "CREATE OR REPLACE PROCEDURE init_pseudos (MaxPseudo INT) BEGIN WHILE MaxPseudo >= 0 DO"
+
" INSERT INTO Pseudos (indexPseudo)" +
" VALUES (MaxPseudo);" +
" SET MaxPseudo = MaxPseudo - 1;" +
" END WHILE;" +
" END init_pseudos;";
Query execution
And I execute the statement with this code :
public static void initBaseDonneePseudos() {
try (Connection connection = DriverManager.getConnection(url, connectionProps);
Statement stmt = connection.createStatement()) {
stmt.execute(RequetesSQL.CREATE_TABLE_PSEUDOS);
stmt.execute(RequetesSQL.CREATE_PROCEDURE_INITPSEUDOS);
stmt.execute(RequetesSQL.CREATE_FUNCTION_RECUPEREPSEUDO);
stmt.execute(RequetesSQL.INIT_TABLE_PSEUDOS);
} catch (SQLException e) {
e.printStackTrace();
}
}
Problem
Test
I execute this test to test statement :
#Nested
class BaseDonneeInteractionTest {
#BeforeEach
public void setUp() {
BaseDonnee.setConnectionHibernate();
}
#Test
void testInitBaseDonnee() {
assertDoesNotThrow(() -> BaseDonnee.initBaseDonneePseudos());
}
}
Error
But I obtain this error
I didn't find the problem of the query, anybody have the solution to solve this ?
The "MySQL Compatibility Mode" doesn't make H2 100% compatible with MySQL. It just changes a few things. The documentation lists them:
Creating indexes in the CREATE TABLE statement is allowed using INDEX(..) or KEY(..). Example: create table test(id int primary key, name varchar(255), key idx_name(name));
When converting a floating point number to an integer, the fractional digits are not truncated, but the value is rounded.
ON DUPLICATE KEY UPDATE is supported in INSERT statements, due to this feature VALUES has special non-standard meaning is some contexts.
INSERT IGNORE is partially supported and may be used to skip rows with duplicate keys if ON DUPLICATE KEY UPDATE is not specified.
REPLACE INTO is partially supported.
Spaces are trimmed from the right side of CHAR values.
REGEXP_REPLACE() uses \ for back-references.
Datetime value functions return the same value within a command.
0x literals are parsed as binary string literals.
Unrelated expressions in ORDER BY clause of DISTINCT queries are allowed.
Some MySQL-specific ALTER TABLE commands are partially supported.
TRUNCATE TABLE restarts next values of generated columns.
If value of an identity column was manually specified, its sequence is updated to generate values after inserted.
NULL value works like DEFAULT value is assignments to identity columns.
Referential constraints don't require an existing primary key or unique constraint on referenced columns and create a unique constraint automatically if such constraint doesn't exist.
LIMIT / OFFSET clauses are supported.
AUTO_INCREMENT clause can be used.
YEAR data type is treated like SMALLINT data type.
GROUP BY clause can contain 1-based positions of expressions from the SELECT list.
Unsafe comparison operators between numeric and boolean values are allowed.
That's all. There is nothing about procedures. As #jccampanero pointed out in the other answer, you must use the syntax specific to H2 if you want to create stored procedures.
The problem is that in H2 there are not explicit procedures or functions as you are trying defining.
For that purpose, H2 allows you to create used defined functions instead. Please, consider reed the appropriate documentation.
Basically, you create a user defined function by declaring an ALIAS for a bunch of Java code.
For example, in your use case, your CREATE_PROCEDURE_INITPSEUDOS could look similar to this:
CREATE ALIAS INIT_PSEUDOS AS $$
import java.sql.Connection;
import java.sql.Statement;
import java.sql.SQLException;
#CODE
void init_pseudos(final Connection conn, final int maxPseudo) throws SQLException {
try (Statement stmt = conn.createStatement()) {
while (maxPseudo >= 0) do {
stmt.execute("INSERT INTO Pseudos (indexPseudo) VALUES (MaxPseudo);");
maxPseudo = maxPseudo - 1;
}
}
}
$$;
Note the following:
As I said, you define a user defined function as Java code. That Java code should be enclosed between two $$ delimiters.
Although I included explicitly some imports, you can use any class in the java.util or java.sql packages in your code. If you want to included explicitly some imports, or if you require classes from other packages than the mentioned, the corresponding imports should be provided right after the first $$ token. In addition, you need to include #CODE the signal H2 where your imports end and your actual Java method starts.
If you need a reference to a Connection to the database in your code, it should be the first argument of your method.
Prefer to raise and not hide exceptions: it will allow your transactions to be committed or rollbacked as a whole appropriately.
You can invoke such a function as usual:
CALL INIT_PSEUDOS (5);
Please, provide the appropriate value for the maxPseudo argument.
Please, consider the provided code as just an example of use: you can improve the code in different ways, like using PreparedStatements instead of Statements for efficiency purposes, checking parameters for nullability, etcetera.
Using H2 as a test database product
Historically, H2 has been used a lot as a test replacement for the actual production database product. But this means you're limiting yourself to the least common denominator between H2 and MySQL, in your case. Including, as others pointed out, the lack of support for stored procedures. You could re-implement all the stored procedures in H2 (very tedious and error prone), or, you could just use testcontainers and run actual integration tests on MySQL directly.
I really think that using H2 as an integration test database is an outdated concept as I've shown in this blog post (unless you're also using H2 in production). You'll be much happier developing and testing everything against MySQL directly!
Using recursive SQL instead
You don't really need that procedure, I think? You probably wrote it to avoid too many round trips to the database for that loop. But you could batch the inserts or generate the following recursive SQL:
INSERT INTO Pseudos (indexPseudo)
WITH RECURSIVE
p (i) AS (
SELECT MaxPseudo
UNION ALL
SELECT i - 1 FROM p WHERE i >= 0
)
SELECT i FROM p;
Now there's no more need to place this in a procedure, you can run the query directly.
Translating your procedure to H2
Just to be complete, for translation of simple procedural logic, jOOQ would offer the feature transparently. You can try it online, here. It's probably overkill, I recommend the other approaches, but perhaps worth a try.
Disclaimer: I work for the company behind jOOQ.
Related
I am trying to setup a unit test environment for a large codebase. For these unit tests I use H2 database instead of the IBM DB2 database, which is used in production and I already implemented a few UDF's to map IBM DB2 related functions to the H2 database.
Some more Details about the project:
Java 8 JDK 321 64 Bit
DB2 12, DSN 12015
H2 version 2.1.212 with <MODE=DB2;DEFAULT_NULL_ORDERING=HIGH;
I am trying to implement the IBM Db2 function: strip() Reference Doc - IBM. This function is called as part of a larger select statement with the clause: STRIP(T2.ITEM_TYPE_NAME_GER, B, ' '). While I can map the first and last input parameter to a Java function and call this function as an ALIAS in H2, I was not able to manage to get the 2nd parameter interpreted in the correct way as a String or Expression. The JDBC/H2 engine always tries to map it to a table column:
org.h2.jdbc.JdbcSQLSyntaxErrorException: Feld "B" nicht gefunden
Column "B" not found; SQL statement:
SELECT T1.ITEM_TYPE_KEY,T1.SUPER_ITM_TYPE_KEY,T2.ITM_TYPE_KEY_TRANS,T2.ITEM_TYPE_NAME,T2.COMPLEX_FLAG,T2.ITEM_CATEGORY,T2.HEADER_FLAG,T2.HEADER_NO ,T2.LEVEL_NO,strip(T2.ITEM_TYPE_NAME_GER, B, ' ') ,T2.LEVEL1_DISPLAY FROM public.AA752T T1, public.AA743T T2 WHERE T1.ITEM_TYPE_KEY NOT IN ('F4CO', 'F4CB', 'F4RB', 'F4SO', 'F4SB', 'F4RO') AND T1.ITEM_TYPE_KEY = T2.ITEM_TYPE_KEY ORDER BY T2.HEADER_NO,T2.HEADER_FLAG DESC,T2.LEVEL_NO,T1.SUPER_ITM_TYPE_KEY,T2.LEVEL_PRIORITY [42122-212]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:502)
at org.h2.message.DbException.getJdbcSQLException(DbException.java:477)
at org.h2.message.DbException.get(DbException.java:223)
at org.h2.message.DbException.get(DbException.java:199)
at org.h2.expression.ExpressionColumn.getColumnException(ExpressionColumn.java:244)
at org.h2.expression.ExpressionColumn.optimizeOther(ExpressionColumn.java:226)
at org.h2.expression.ExpressionColumn.optimize(ExpressionColumn.java:213)
at org.h2.expression.function.JavaFunction.optimize(JavaFunction.java:59)
at org.h2.command.query.Select.prepareExpressions(Select.java:1170)
at org.h2.command.query.Query.prepare(Query.java:218)
at org.h2.command.Parser.prepareCommand(Parser.java:574)
at org.h2.engine.SessionLocal.prepareLocal(SessionLocal.java:631)
at org.h2.engine.SessionLocal.prepareCommand(SessionLocal.java:554)
at org.h2.jdbc.JdbcConnection.prepareCommand(JdbcConnection.java:1116)
at org.h2.jdbc.JdbcPreparedStatement.(JdbcPreparedStatement.java:92)
at org.h2.jdbc.JdbcConnection.prepareStatement(JdbcConnection.java:288)
at com.db.cib.gbd.gps.pbs.pricing.StaticItemDetails.retriveItemDisplayDetails(StaticItemDetails.java:920)
This is my Java UDF:
public static String strip(String s, Expression loc, String trimConstant) {
if (loc.toLowerCase() == "b" || loc.toLowerCase() == "both") {
s = s.replaceAll("^[" + trimConstant + "]+|[ \t]+$", "");
} else if (loc.toLowerCase() == "l" || loc.toLowerCase() == "leading") {
s = s.replaceAll("^[" + trimConstant + "]+", "");
} else if (loc.toLowerCase() == "t" || loc.toLowerCase() == "trailing") {
s = s.replaceAll("[" + trimConstant + "]+$", "");
}
return s;
}
Is there a possibility to get the mapping of the column in the correct way, or can you suggest either a SQL function, which is usable as UDF alias (how is this usable?) or a way to solve this error?
To avoid this question: I cannot change the existing sql statement. I have to find an alias for this function.
It isn't possible to create a user-defined function with special arguments in H2 and most likely in all or almost all other database systems. User-defined functions accept only plain comma-separated arguments with literals or expressions in them.
(You also cannot declare a parameter as org.h2.expression.Expression.)
The proper solution here is to use the TRIM function from the SQL Standard:
https://h2database.com/html/functions.html#trim
https://www.ibm.com/docs/en/db2-for-zos/11?topic=functions-trim
TRIM(BOTH ' ' FROM T2.ITEM_TYPE_NAME_GER)
Please note that B, L, and T acronyms is a DB2-specific extension, in H2 you can use only standard BOTH, LEADING, and TRAILING.
If you cannot change your query you can only modify sources of H2 and compile its own version with support of the STRIP. But actually you can run into some other issue immediately. When you want to use multiple database systems at once you need to realize that they all are very different from each other. H2 provides compatibility modes for others, but even in these modes there is a very limited compatibility. It means you need to avoid usage of vendor-specific functions and other grammar elements and in some cases different SQL for different systems may be required.
You also can try to create constants B, BOTH, L, LEADING, T, and TRAILING with some values
CREATE CONSTANT B VALUE 1;
CREATE CONSTANT BOTH VALUE 1;
CREATE CONSTANT L VALUE 2;
…
and create a function with three arguments, second argument will be of type int (or other, if you'll decide to choose values of some other data type). But names of constants may conflict with column names, so this workaround is far from being perfect and may not work at all in some queries.
For example, I have the code below:
import java.sql.*; ...
public void main (string[] args){
try {
Class.forName („COM.ibm.db2.jdbc.net.DB2Driver“);} catch (ClassNotFoundException e) {//error handling}
try {
String url = "jdbc:db2://host:6789/myDB2"
Connection con = DriverManager.getConnection(url, "login", "password");
PreparedStatement pStmt = con.prepareStatement("UPDATE PERS SET Salary=Salary*2.0 WHERE PNR=?"
pStmt.setInt (1, 35);
pStmt.executeUpdate();
pStmt.setString (1, args[0]);
pStmt.executeUpdate();
con.close();
} catch (SQLException e) { //error handling}
}
Presumably we have the table like:
+--------+----------+-----------+
|PNR |Name |Salary |
+--------+----------+-----------+
|34 |Tim |20000 |
+--------+----------+-----------+
|35 |John |45000 |
+--------+----------+-----------+
I have a difficult time predicting what will happen if:
args[0]="35 OR Salary<100000"
Doesn't the setString command replace args[0] with 35 OR Salary < 100000 and then all the salary record gets doubled?
Server-Side Prepared Statements
SQL parameters help to avoid SQL injection because the value of the parameter is not combined with the SQL query at all. The SQL query with parameter placeholders is sent to the MySQL server, where it is parsed and analyzed. It does things like check that you wrote valid SQL syntax, that the tables and columns you reference exist, and you have the right privileges to access those tables and columns.
This is why parameters can't be used for table names or column names or other syntax. Because the validation occurs when the parameters are still left as placeholders. The value of the parameters is sent later, so the validation must assume a parameter must replace only a single scalar value in your SQL query.
After this point, the query is stored internally in the MySQL server as non-textual data structures. It is no longer in SQL, it's just a number of internal objects in the MySQL code. The places where you had used ? become query elements that MySQL knows need to be supplied with values before it can execute the query.
When you run pStmt.executeUpdate() the values of the variables you bound to the parameters are sent to the MySQL server. They are filled into the placeholders in the non-textual representation of the query.
This way, the parameter values are not combined until after the parsing is done, therefore there's no way for the content of the parameter to change the SQL syntax. It affects the SQL query only like a single string would, as if there were a type of quote delimiter that could not be broken by unmatched quote characters in the parameter content.
Query parameters are a reliable way to protect against SQL injection.
Emulated Prepared Statements
Some drivers implement an "emulated" prepared statement. This means it does nothing with the SQL query you pass to prepareStatement(), except save the SQL string in the JDBC driver (on the client-side). It does not send the SQL query to the server at this time.
Then when you run executeUpdate() your variables are interpolated into the parameter placeholders in the SQL string, and the full string is sent to the server. Then the MySQL server parses the combined SQL query, with parameter values and all. MySQL Server can't even tell which values were literal values in the original SQL query versus which were combined as parameters. They all appear as literal values to the parser.
In this case, you have to trust that the JDBC driver does correct escaping, so quotes and other characters inside your parameter content can't mix up the SQL parser. The driver should be well-tested to handle all cases, like special character sets, and hex-encoded quote characters and other ways to trick it.
That won't cause SQL injection issues. It will translate to:
UPDATE PERS SET Salary=Salary*2.0 WHERE PNR='35 OR Salary<100000'
The inserted quotes will save you from SQL injection. I am simplifying a bit. JDBC implementation determines how exactly PreparedStatement translates to a real SQL query. It doesn't necessarily actually translate it to the above SQL. But this is one way it can prevent attacks.
Careful, though. If you use user input to create your SQL, you sill still be susceptible to SQL injection. As long as you use user inputs only to call .setXYZ() params, you'll be safe from it.
We have an Oracle database with the following charset settings
SELECT parameter, value FROM nls_database_parameters WHERE parameter like 'NLS%CHARACTERSET'
NLS_NCHAR_CHARACTERSET: AL16UTF16
NLS_CHARACTERSET: WE8ISO8859P15
In this database we have a table with a CLOB field, which has a record that starts with the following string, stored obviously in ISO-8859-15: X²ARB (here correctly converted to unicode, in particular that 2-superscript is important and correct).
Then we have the following trivial piece of code to get the value out, which is supposed to automatically convert the charset to unicode via globalization support in Oracle:
private static final String STATEMENT = "SELECT data FROM datatable d WHERE d.id=2562456";
public static void main(String[] args) throws Exception {
Class.forName("oracle.jdbc.driver.OracleDriver");
try (Connection conn = DriverManager.getConnection(DB_URL);
ResultSet rs = conn.createStatement().executeQuery(STATEMENT))
{
if (rs.next()) {
System.out.println(rs.getString(1).substring(0, 5));
}
}
}
Running the code prints:
with ojdbc8.jar and orai18n.jar: X�ARB -- incorrect
with ojdbc7.jar and orai18n.jar: X�ARB -- incorrect
with ojdbc-6.jar: X²ARB -- correct
By using UNISTR and changing the statement to SELECT UNISTR(data) FROM datatable d WHERE d.id=2562456 I can bring ojdbc7.jar and ojdbc8.jar to return the correct value, but this would require an unknown number of changes to the code as this is probably not the only place where the problem occurs.
Is there anything I can do to the client or server configurations to make all queries return correctly encoded values without statement modifications?
It definitely looks like a bug in the JDBC thin driver (I assume you're using thin). It could be related to LOB prefetch where the CLOB's length, character set id and the first part of the LOB data is sent inband. This feature was introduced in 11.2. As a workaround, you can disable lob prefetch by setting the connection property
oracle.jdbc.defaultLobPrefetchSize
to "-1". Meanwhile I'll follow up on this bug to make sure that it gets fixed.
Please have a look at Database JDBC Developer's Guide - Globalization Support
The basic Java Archive (JAR) file ojdbc7.jar, contains all the
necessary classes to provide complete globalization support for:
CHAR or VARCHAR data members of object and collection for the character sets US7ASCII, WE8DEC, WE8ISO8859P1, WE8MSWIN1252, and UTF8.
To use any other character sets in CHAR or VARCHAR data members of
objects or collections, you must include orai18n.jar in the CLASSPATH
environment variable:
ORACLE_HOME/jlib/orai18n.jar
I am attempting to create a test database (based off of my production db) at runtime, but rather than have to maintain an exact duplicate test db i'd like to copy the entire data structure of my production db at runtime and then when I close the test database, drop the entire database.
I assume I will be using statements such as:
CREATE DATABASE test //to create the test db
CREATE TABLE test.sampleTable LIKE production.sampleTable //to create each table
And when I am finished with the test db, calling a close method will run something like:
DROP DATABASE test //delete the database and all its tables
But how do I go about automatically finding all the tables within the production database without having to manually write them out. The idea is that I can manipulate my production db without having to be concerned with maintaining the structure identically within the test db.
Would a stored procedure be necessary in this case? Some sample code on how to achieve something like this would be appreciated.
If the database driver you are using supports it, you can use DatabaseMetaData#getTables to get the list of tables for a schema. You can get access to DatabaseMetaData from Connection#getMetaData.
In your scripting language, you call "SHOW TABLES" on the database you want to copy. Reading that result set a row at a time, your program puts the name of the table into a variable (let's call it $tablename) and can generate the sql: "CREATE TABLE test.$tablename LIKE production.$tablename". Iterate through the result set and you're done.
(You won't get foreign key constraints that way, but maybe you don't need those. If you do, you can run "SHOW CREATE TABLE $tablename" and parse the results to pick out the constraints.)
I don't have a code snippet for java, but here is one for perl that you could treat as pseudo-code:
$ref = $dbh->selectall_arrayref("SHOW TABLES");
unless(defined ($ref)){
print "Nothing found\n";
} else {
foreach my $row_ref (#{$ref}){
push(#tables, $row_ref->[0]);
}
}
The foreach statement iterates over the result set in an array reference returned by the database interface library. The push statement puts the first element of the current row of the result set into an array variable #tables. You'd be using the database library appropriate for your language of choice.
I would use mysqldump : http://dev.mysql.com/doc/refman/5.1/en/mysqldump.html
It will produce a file containing all the sql commands needed to replicate the prod database
The solutions was as follows:
private static final String SQL_CREATE_TEST_DB = "CREATE DATABASE test";
private static final String SQL_PROD_TABLES = "SHOW TABLES IN production";
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
jdbcTemplate.execute(SQL_CREATE_TEST_DB);
SqlRowSet result = jdbcTemplate.queryForRowSet(SQL_PROD_TABLES);
while(result.next()) {
String tableName = result.getString(result.getMetaData().getColumnName(1)); //Retrieves table name from column 1
jdbcTemplate.execute("CREATE TABLE test2." + tableName + " LIKE production." + tableName); //Create new table in test2 based on production structure
}
This is using Spring to simplify the database connection etc, but the real magic is in the SQL statements. As mentioned by D Mac, this will not copy foreign key constraints, but that can be achieved by running another SQL statement and parsing the results.
I want to build an SQL string to do database manipulation (updates, deletes, inserts, selects, that sort of thing) - instead of the awful string concat method using millions of "+"'s and quotes which is unreadable at best - there must be a better way.
I did think of using MessageFormat - but its supposed to be used for user messages, although I think it would do a reasonable job - but I guess there should be something more aligned to SQL type operations in the java sql libraries.
Would Groovy be any good?
First of all consider using query parameters in prepared statements:
PreparedStatement stm = c.prepareStatement("UPDATE user_table SET name=? WHERE id=?");
stm.setString(1, "the name");
stm.setInt(2, 345);
stm.executeUpdate();
The other thing that can be done is to keep all queries in properties file. For example
in a queries.properties file can place the above query:
update_query=UPDATE user_table SET name=? WHERE id=?
Then with the help of a simple utility class:
public class Queries {
private static final String propFileName = "queries.properties";
private static Properties props;
public static Properties getQueries() throws SQLException {
InputStream is =
Queries.class.getResourceAsStream("/" + propFileName);
if (is == null){
throw new SQLException("Unable to load property file: " + propFileName);
}
//singleton
if(props == null){
props = new Properties();
try {
props.load(is);
} catch (IOException e) {
throw new SQLException("Unable to load property file: " + propFileName + "\n" + e.getMessage());
}
}
return props;
}
public static String getQuery(String query) throws SQLException{
return getQueries().getProperty(query);
}
}
you might use your queries as follows:
PreparedStatement stm = c.prepareStatement(Queries.getQuery("update_query"));
This is a rather simple solution, but works well.
For arbitrary SQL, use jOOQ. jOOQ currently supports SELECT, INSERT, UPDATE, DELETE, TRUNCATE, and MERGE. You can create SQL like this:
String sql1 = DSL.using(SQLDialect.MYSQL)
.select(A, B, C)
.from(MY_TABLE)
.where(A.equal(5))
.and(B.greaterThan(8))
.getSQL();
String sql2 = DSL.using(SQLDialect.MYSQL)
.insertInto(MY_TABLE)
.values(A, 1)
.values(B, 2)
.getSQL();
String sql3 = DSL.using(SQLDialect.MYSQL)
.update(MY_TABLE)
.set(A, 1)
.set(B, 2)
.where(C.greaterThan(5))
.getSQL();
Instead of obtaining the SQL string, you could also just execute it, using jOOQ. See
http://www.jooq.org
(Disclaimer: I work for the company behind jOOQ)
One technology you should consider is SQLJ - a way to embed SQL statements directly in Java. As a simple example, you might have the following in a file called TestQueries.sqlj:
public class TestQueries
{
public String getUsername(int id)
{
String username;
#sql
{
select username into :username
from users
where pkey = :id
};
return username;
}
}
There is an additional precompile step which takes your .sqlj files and translates them into pure Java - in short, it looks for the special blocks delimited with
#sql
{
...
}
and turns them into JDBC calls. There are several key benefits to using SQLJ:
completely abstracts away the JDBC layer - programmers only need to think about Java and SQL
the translator can be made to check your queries for syntax etc. against the database at compile time
ability to directly bind Java variables in queries using the ":" prefix
There are implementations of the translator around for most of the major database vendors, so you should be able to find everything you need easily.
I am wondering if you are after something like Squiggle (GitHub). Also something very useful is jDBI. It won't help you with the queries though.
I would have a look at Spring JDBC. I use it whenever I need to execute SQLs programatically. Example:
int countOfActorsNamedJoe
= jdbcTemplate.queryForInt("select count(0) from t_actors where first_name = ?", new Object[]{"Joe"});
It's really great for any kind of sql execution, especially querying; it will help you map resultsets to objects, without adding the complexity of a complete ORM.
I tend to use Spring's Named JDBC Parameters so I can write a standard string like "select * from blah where colX=':someValue'"; I think that's pretty readable.
An alternative would be to supply the string in a separate .sql file and read the contents in using a utility method.
Oh, also worth having a look at Squill: https://squill.dev.java.net/docs/tutorial.html
I second the recommendations for using an ORM like Hibernate. However, there are certainly situations where that doesn't work, so I'll take this opportunity to tout some stuff that i've helped to write: SqlBuilder is a java library for dynamically building sql statements using the "builder" style. it's fairly powerful and fairly flexible.
I have been working on a Java servlet application that needs to construct very dynamic SQL statements for adhoc reporting purposes. The basic function of the app is to feed a bunch of named HTTP request parameters into a pre-coded query, and generate a nicely formatted table of output. I used Spring MVC and the dependency injection framework to store all of my SQL queries in XML files and load them into the reporting application, along with the table formatting information. Eventually, the reporting requirements became more complicated than the capabilities of the existing parameter mapping frameworks and I had to write my own. It was an interesting exercise in development and produced a framework for parameter mapping much more robust than anything else I could find.
The new parameter mappings looked as such:
select app.name as "App",
${optional(" app.owner as "Owner", "):showOwner}
sv.name as "Server", sum(act.trans_ct) as "Trans"
from activity_records act, servers sv, applications app
where act.server_id = sv.id
and act.app_id = app.id
and sv.id = ${integer(0,50):serverId}
and app.id in ${integerList(50):appId}
group by app.name, ${optional(" app.owner, "):showOwner} sv.name
order by app.name, sv.name
The beauty of the resulting framework was that it could process HTTP request parameters directly into the query with proper type checking and limit checking. No extra mappings required for input validation. In the example query above, the parameter named serverId
would be checked to make sure it could cast to an integer and was in the range of 0-50. The parameter appId would be processed as an array of integers, with a length limit of 50. If the field showOwner is present and set to "true", the bits of SQL in the quotes will be added to the generated query for the optional field mappings. field Several more parameter type mappings are available including optional segments of SQL with further parameter mappings. It allows for as complex of a query mapping as the developer can come up with. It even has controls in the report configuration to determine whether a given query will have the final mappings via a PreparedStatement or simply ran as a pre-built query.
For the sample Http request values:
showOwner: true
serverId: 20
appId: 1,2,3,5,7,11,13
It would produce the following SQL:
select app.name as "App",
app.owner as "Owner",
sv.name as "Server", sum(act.trans_ct) as "Trans"
from activity_records act, servers sv, applications app
where act.server_id = sv.id
and act.app_id = app.id
and sv.id = 20
and app.id in (1,2,3,5,7,11,13)
group by app.name, app.owner, sv.name
order by app.name, sv.name
I really think that Spring or Hibernate or one of those frameworks should offer a more robust mapping mechanism that verifies types, allows for complex data types like arrays and other such features. I wrote my engine for only my purposes, it isn't quite read for general release. It only works with Oracle queries at the moment and all of the code belongs to a big corporation. Someday I may take my ideas and build a new open source framework, but I'm hoping one of the existing big players will take up the challenge.
Why do you want to generate all the sql by hand? Have you looked at an ORM like Hibernate Depending on your project it will probably do at least 95% of what you need, do it in a cleaner way then raw SQL, and if you need to get the last bit of performance you can create the SQL queries that need to be hand tuned.
You can also have a look at MyBatis (www.mybatis.org) . It helps you write SQL statements outside your java code and maps the sql results into your java objects among other things.
Google provides a library called the Room Persitence Library which provides a very clean way of writing SQL for Android Apps, basically an abstraction layer over underlying SQLite Database. Bellow is short code snippet from the official website:
#Dao
public interface UserDao {
#Query("SELECT * FROM user")
List<User> getAll();
#Query("SELECT * FROM user WHERE uid IN (:userIds)")
List<User> loadAllByIds(int[] userIds);
#Query("SELECT * FROM user WHERE first_name LIKE :first AND "
+ "last_name LIKE :last LIMIT 1")
User findByName(String first, String last);
#Insert
void insertAll(User... users);
#Delete
void delete(User user);
}
There are more examples and better documentation in the official docs for the library.
There is also one called MentaBean which is a Java ORM. It has nice features and seems to be pretty simple way of writing SQL.
Read an XML file.
You can read it from an XML file. Its easy to maintain and work with.
There are standard STaX, DOM, SAX parsers available out there to make it few lines of code in java.
Do more with attributes
You can have some semantic information with attributes on the tag to help do more with the SQL. This can be the method name or query type or anything that helps you code less.
Maintaince
You can put the xml outside the jar and easily maintain it. Same benefits as a properties file.
Conversion
XML is extensible and easily convertible to other formats.
Use Case
Metamug uses xml to configure REST resource files with sql.
If you put the SQL strings in a properties file and then read that in you can keep the SQL strings in a plain text file.
That doesn't solve the SQL type issues, but at least it makes copying&pasting from TOAD or sqlplus much easier.
How do you get string concatenation, aside from long SQL strings in PreparedStatements (that you could easily provide in a text file and load as a resource anyway) that you break over several lines?
You aren't creating SQL strings directly are you? That's the biggest no-no in programming. Please use PreparedStatements, and supply the data as parameters. It reduces the chance of SQL Injection vastly.