I'm trying to generate a Database using JOOQ
i do create a Table with this code:
CreateTableAsStep<Record> table = create.createTable("TestTable");
CreateTableColumnStep step = table.column("testColumn", SQLDataType.Integer);
step.execute();
this works fine, but when it comes to inserting data, i run into a problem
the doc includes the following example:
create.insertInto(AUTHOR)
.set(AUTHOR.ID, 100)
.set(AUTHOR.FIRST_NAME, "Hermann")
.set(AUTHOR.LAST_NAME, "Hesse")
.newRecord()
.set(AUTHOR.ID, 101)
.set(AUTHOR.FIRST_NAME, "Alfred")
.set(AUTHOR.LAST_NAME, "Döblin")
.execute();
here AUTHOR is not a simple String it expects a org.jooq.Table<R extends Record>
i thought there might be a return type when creating the table, but i did not find it. Google did not help as Table is not the best word to search for ;-)
Question: how can i get to an instance of a Table - i do have its name as String?
You can always create Table references via DSL.table(String) or DSL.table(Name). For example:
// Assuming this:
import static org.jooq.impl.DSL.*;
create.insertInto(table(name("TestTable")))
.set(field(name("testColumn")), 1)
.execute();
Notice also my usage of DSL.field(Name).
Plain SQL vs. Name references
It's worth reading up on the difference between creating dynamic table / field objects at runtime either with plain SQL strings (as in DSL.table(String)) or with name references (as in DSL.table(Name)). Essentially:
Plain SQL strings are case-insensitive and subject to SQL injection
Name references are case-sensitive by default
In your case, as you probably created case sensitive table/column names, you should prefer the latter. More info can be found here:
http://www.jooq.org/doc/latest/manual/sql-building/plain-sql
http://www.jooq.org/doc/latest/manual/sql-building/names
Related
Description of code
Database connection
I try to to store Java object locally database without use external database.
For this I use JDBC with H2 via Hibernate :
/**
* #param connection the connection to set
*/
public static void setConnectionHibernate() {
Properties connectionProps = new Properties();
connectionProps.put("user", "sa");
try {
Class.forName("org.h2.Driver");
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
url = "jdbc:h2:mem:db1;DB_CLOSE_DELAY=-1;MODE=MySQL;";
}
Query
I store the PROCEDURE in String with this code :
static final String CREATE_PROCEDURE_INITPSEUDOS = "CREATE OR REPLACE PROCEDURE init_pseudos (MaxPseudo INT) BEGIN WHILE MaxPseudo >= 0 DO"
+
" INSERT INTO Pseudos (indexPseudo)" +
" VALUES (MaxPseudo);" +
" SET MaxPseudo = MaxPseudo - 1;" +
" END WHILE;" +
" END init_pseudos;";
Query execution
And I execute the statement with this code :
public static void initBaseDonneePseudos() {
try (Connection connection = DriverManager.getConnection(url, connectionProps);
Statement stmt = connection.createStatement()) {
stmt.execute(RequetesSQL.CREATE_TABLE_PSEUDOS);
stmt.execute(RequetesSQL.CREATE_PROCEDURE_INITPSEUDOS);
stmt.execute(RequetesSQL.CREATE_FUNCTION_RECUPEREPSEUDO);
stmt.execute(RequetesSQL.INIT_TABLE_PSEUDOS);
} catch (SQLException e) {
e.printStackTrace();
}
}
Problem
Test
I execute this test to test statement :
#Nested
class BaseDonneeInteractionTest {
#BeforeEach
public void setUp() {
BaseDonnee.setConnectionHibernate();
}
#Test
void testInitBaseDonnee() {
assertDoesNotThrow(() -> BaseDonnee.initBaseDonneePseudos());
}
}
Error
But I obtain this error
I didn't find the problem of the query, anybody have the solution to solve this ?
The "MySQL Compatibility Mode" doesn't make H2 100% compatible with MySQL. It just changes a few things. The documentation lists them:
Creating indexes in the CREATE TABLE statement is allowed using INDEX(..) or KEY(..). Example: create table test(id int primary key, name varchar(255), key idx_name(name));
When converting a floating point number to an integer, the fractional digits are not truncated, but the value is rounded.
ON DUPLICATE KEY UPDATE is supported in INSERT statements, due to this feature VALUES has special non-standard meaning is some contexts.
INSERT IGNORE is partially supported and may be used to skip rows with duplicate keys if ON DUPLICATE KEY UPDATE is not specified.
REPLACE INTO is partially supported.
Spaces are trimmed from the right side of CHAR values.
REGEXP_REPLACE() uses \ for back-references.
Datetime value functions return the same value within a command.
0x literals are parsed as binary string literals.
Unrelated expressions in ORDER BY clause of DISTINCT queries are allowed.
Some MySQL-specific ALTER TABLE commands are partially supported.
TRUNCATE TABLE restarts next values of generated columns.
If value of an identity column was manually specified, its sequence is updated to generate values after inserted.
NULL value works like DEFAULT value is assignments to identity columns.
Referential constraints don't require an existing primary key or unique constraint on referenced columns and create a unique constraint automatically if such constraint doesn't exist.
LIMIT / OFFSET clauses are supported.
AUTO_INCREMENT clause can be used.
YEAR data type is treated like SMALLINT data type.
GROUP BY clause can contain 1-based positions of expressions from the SELECT list.
Unsafe comparison operators between numeric and boolean values are allowed.
That's all. There is nothing about procedures. As #jccampanero pointed out in the other answer, you must use the syntax specific to H2 if you want to create stored procedures.
The problem is that in H2 there are not explicit procedures or functions as you are trying defining.
For that purpose, H2 allows you to create used defined functions instead. Please, consider reed the appropriate documentation.
Basically, you create a user defined function by declaring an ALIAS for a bunch of Java code.
For example, in your use case, your CREATE_PROCEDURE_INITPSEUDOS could look similar to this:
CREATE ALIAS INIT_PSEUDOS AS $$
import java.sql.Connection;
import java.sql.Statement;
import java.sql.SQLException;
#CODE
void init_pseudos(final Connection conn, final int maxPseudo) throws SQLException {
try (Statement stmt = conn.createStatement()) {
while (maxPseudo >= 0) do {
stmt.execute("INSERT INTO Pseudos (indexPseudo) VALUES (MaxPseudo);");
maxPseudo = maxPseudo - 1;
}
}
}
$$;
Note the following:
As I said, you define a user defined function as Java code. That Java code should be enclosed between two $$ delimiters.
Although I included explicitly some imports, you can use any class in the java.util or java.sql packages in your code. If you want to included explicitly some imports, or if you require classes from other packages than the mentioned, the corresponding imports should be provided right after the first $$ token. In addition, you need to include #CODE the signal H2 where your imports end and your actual Java method starts.
If you need a reference to a Connection to the database in your code, it should be the first argument of your method.
Prefer to raise and not hide exceptions: it will allow your transactions to be committed or rollbacked as a whole appropriately.
You can invoke such a function as usual:
CALL INIT_PSEUDOS (5);
Please, provide the appropriate value for the maxPseudo argument.
Please, consider the provided code as just an example of use: you can improve the code in different ways, like using PreparedStatements instead of Statements for efficiency purposes, checking parameters for nullability, etcetera.
Using H2 as a test database product
Historically, H2 has been used a lot as a test replacement for the actual production database product. But this means you're limiting yourself to the least common denominator between H2 and MySQL, in your case. Including, as others pointed out, the lack of support for stored procedures. You could re-implement all the stored procedures in H2 (very tedious and error prone), or, you could just use testcontainers and run actual integration tests on MySQL directly.
I really think that using H2 as an integration test database is an outdated concept as I've shown in this blog post (unless you're also using H2 in production). You'll be much happier developing and testing everything against MySQL directly!
Using recursive SQL instead
You don't really need that procedure, I think? You probably wrote it to avoid too many round trips to the database for that loop. But you could batch the inserts or generate the following recursive SQL:
INSERT INTO Pseudos (indexPseudo)
WITH RECURSIVE
p (i) AS (
SELECT MaxPseudo
UNION ALL
SELECT i - 1 FROM p WHERE i >= 0
)
SELECT i FROM p;
Now there's no more need to place this in a procedure, you can run the query directly.
Translating your procedure to H2
Just to be complete, for translation of simple procedural logic, jOOQ would offer the feature transparently. You can try it online, here. It's probably overkill, I recommend the other approaches, but perhaps worth a try.
Disclaimer: I work for the company behind jOOQ.
I am trying to write some tests for a Java Spark-Sql application. One operation I need to test renames a column, and I ran into some difficulty comparing the actual value of the renamed column with my expected value. After some experimentation, I was able to write the following two tests to demonstrate the problem:
First, as a sanity check, I tried this (df is a spark sql DataFrame, generated by reading some sample data from a json file I'm testing against):
#Test
public void testColumnEquality() throws Exception {
Column val1 = df.col("col2");
Column val2 = df.col("col2");
Assert.assertEquals(val1, val2);
}
Which passes, as one would expect. Then I tried this:
#Test
public void testReanmeColumnEquality() throws Exception {
Column val1 = df.col("col2").as("col2");
Column val2 = df.col("col2").as("col2");
Assert.assertEquals(val1, val2);
}
which fails with the error java.lang.AssertionError: expected:<col2 AS col2#4L> but was:<col2 AS col2#5L>
Digging around in the scala code (full disclosure - I know very little scala) it looks like this has to do with the NamedExpression unique id.
Is there any way to sensibly check that these two columns represent the same operations with the same alias?
(I'm working in spark 1.6, and would ideally like a solution for that version line, but if this is fixed in 2.0 that would also be good information.)
Thanks you.
I wrote a blog post about how to resolve this:
The trick is: check whether the Expression has the Alias trait:
`column.expr() instanceof Alias`
If it does, unpack the child expression and the name using the Extractor pattern:
alias = (Alias) column.expr()
Option<Tuple2<Expression, String>> aliasTuple = Alias$.MODULE$.unapply(alias);
I did some digging and it looks like the information about the child of a Column with an alias is lost in the process of instantiating the new Column. Maybe there is a state to query somewhere, but I didn't find it.
So it's not an answer, but hopefully it is useful or of interest to somebody.
more info
The definition of the as method on a Column object refers to the name function (see Column.scala), which just call the Alias case class defined here. The Alias (with its child), is not exposed. It is directly given to the Column class withExpr function which instantiate a new column based on the Alias named expression.
So you either compare directly the result of toString on the columns (loosing the information on where the column comes from, i.e. which dataframe), or you actually parse the string printed by the explain(true) method...but it doesn't seem sensible to me...
I am using jdbc PreparedStatement for data insertion.
Statement stmt = conn.prepareStatement(
"INESRT INTO" + tablename+ "("+columnString+") VALUES (?,?,?)");
tablename and columnString are something that is dynamically generated.
I've tried to parameterise tablename and columnString but they will just resolve to something like 'tablename' which will violate the syntax.
I've found somewhere online that suggest me to lookup the database to check for valid tablename/columnString, and cache it somewhere(a Hashset perhaps) for another query, but I'm looking for better performance/ quick hack that will solve the issue, perhaps a string validator/ regex that will do the trick.
Have anyone came across this issue and how do you solve it?
I am not a java-guy, so, only a theory.
You can either format dynamically added identifiers or white-list them.
Second option is way better. Because
most developers aren't familiar enough with identifiers to format them correctly. Say, to quote an identifier, which is offered in the first comment, won't make it protected at all.
there could be another attack vector, not entirely an injection, but similar: imagine there is a column in your table, an ordinary user isn't allowed to - say, called "admin". With dynamically built columnString using data coming from the client side, it's piece of cake to forge a privilege escalation.
Thus, to list all the possible (and allowed) variants in your code beforehand, and then to verify entered value against it, would be the best.
As of columnString - is consists of separate column names. Thus, to protect it, one have to verify each separate column name against a white list, and then assemble a final columnString from them.
Create a method that generates the sql string for you:
private static final String template = "insert into %s (%s) values (%s)";
private String buildStmt(String tblName, String ... colNames) {
StringJoiner colNamesJoiner = new StringJoiner(",");
StringJoiner paramsJoiner = new StringJoiner(",");
Arrays.stream(colNames).forEach(colName -> {
colNamesJoiner.add(colName);
paramsJoiner.add("?");
});
return String.format(template, tblName, colNamesJoiner.toString(), paramsJoiner.toString());
}
Then use it...
Statement stmt = conn.prepareStatement(buildStmt(tablename, [your column names]));
As an elaboration on #Anders' answer, don't use the input parameter as the name directly, but keep a properties file (or database table) that maps a set of allowed inputs to actual table names.
That way any invalid name will not lead to valid SQL (and can be caught before any SQL is generated) AND the actual names are never known outside the application, thus making it far harder to guess what would be valid SQL statements.
I think, the best approach is to get table and columns names from database or other non user input, and use parameters in prepared statement for the rest.
There are multiple solutions we can apply.
1) White List Input Validation
String tableName;
switch(PARAM):
case "Value1": tableName = "fooTable";
break;
case "Value2": tableName = "barTable";
break;
...
default : throw new InputValidationException("unexpected value provided for table name");
By doing this input validation on tableName, will allows only specified tables in the query, so it will prevents sql injection attack.
2) Bind your dynamic columnName(s) or tableName(s) with special characters as shown below
eg:
For Mysql : use back codes (`)
Select `columnName ` from `tableName `;
For MSSQL : Use double codes(" or [ ] )
select "columnName" from "tableName"; or
select [columnName] from [tableName];
Note: Before doing this you should sanitize your data with this special characters ( `, " , [ , ] )
I am attempting to create a test database (based off of my production db) at runtime, but rather than have to maintain an exact duplicate test db i'd like to copy the entire data structure of my production db at runtime and then when I close the test database, drop the entire database.
I assume I will be using statements such as:
CREATE DATABASE test //to create the test db
CREATE TABLE test.sampleTable LIKE production.sampleTable //to create each table
And when I am finished with the test db, calling a close method will run something like:
DROP DATABASE test //delete the database and all its tables
But how do I go about automatically finding all the tables within the production database without having to manually write them out. The idea is that I can manipulate my production db without having to be concerned with maintaining the structure identically within the test db.
Would a stored procedure be necessary in this case? Some sample code on how to achieve something like this would be appreciated.
If the database driver you are using supports it, you can use DatabaseMetaData#getTables to get the list of tables for a schema. You can get access to DatabaseMetaData from Connection#getMetaData.
In your scripting language, you call "SHOW TABLES" on the database you want to copy. Reading that result set a row at a time, your program puts the name of the table into a variable (let's call it $tablename) and can generate the sql: "CREATE TABLE test.$tablename LIKE production.$tablename". Iterate through the result set and you're done.
(You won't get foreign key constraints that way, but maybe you don't need those. If you do, you can run "SHOW CREATE TABLE $tablename" and parse the results to pick out the constraints.)
I don't have a code snippet for java, but here is one for perl that you could treat as pseudo-code:
$ref = $dbh->selectall_arrayref("SHOW TABLES");
unless(defined ($ref)){
print "Nothing found\n";
} else {
foreach my $row_ref (#{$ref}){
push(#tables, $row_ref->[0]);
}
}
The foreach statement iterates over the result set in an array reference returned by the database interface library. The push statement puts the first element of the current row of the result set into an array variable #tables. You'd be using the database library appropriate for your language of choice.
I would use mysqldump : http://dev.mysql.com/doc/refman/5.1/en/mysqldump.html
It will produce a file containing all the sql commands needed to replicate the prod database
The solutions was as follows:
private static final String SQL_CREATE_TEST_DB = "CREATE DATABASE test";
private static final String SQL_PROD_TABLES = "SHOW TABLES IN production";
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
jdbcTemplate.execute(SQL_CREATE_TEST_DB);
SqlRowSet result = jdbcTemplate.queryForRowSet(SQL_PROD_TABLES);
while(result.next()) {
String tableName = result.getString(result.getMetaData().getColumnName(1)); //Retrieves table name from column 1
jdbcTemplate.execute("CREATE TABLE test2." + tableName + " LIKE production." + tableName); //Create new table in test2 based on production structure
}
This is using Spring to simplify the database connection etc, but the real magic is in the SQL statements. As mentioned by D Mac, this will not copy foreign key constraints, but that can be achieved by running another SQL statement and parsing the results.
my code is written in java, and I am really new to java, so i hope my explanations are correct:
i have a java written web service that works with a data base.
the data base types can be PostgreSQL and mysql.
so my webservice works with the JDBC connection for both data bases.
one of my data base tables is table urls,
for postgressql it is created like this:
CREATE TABLE urls (
id serial NOT NULL primary key,
url text not null,
type integer not null);
for mysql it is creates like this:
CREATE TABLE IF NOT EXISTS URLS (
id INTEGER primary key NOT NULL AUTO_INCREMENT,
url varchar (1600) NOT NULL,
type INTEGER NOT NULL );
when I try inserting data to this table I use an entity called urls:
this entity has:
private BigDecimal id;
private String url;
private BigInteger type;
when I try to insert values to the urls table I assign values to the urls entity, while leaving the id as NULL since it is AUTO_INCREMENT for mysql and serial for postgres.
the query works for my sql, but fails for postgress.
in the postgres server log I can see the following error:
null value in column "id" violates not-null constraint
cause I sends NULL as the id value.
I found out that in order for the query to work I should use this query:
INSERT INTO URLS(ID, TYPE, URL) VALUES(DEFAULT, 1, 'DED'); or this one:
INSERT INTO URLS(TYPE, URL) VALUES(1, 'DED'); or this one:
instead of this one, that I use:
INSERT INTO URLS(ID, TYPE, URL) VALUES(NULL, 1, 'DED');
so my question is,
how do I assign the DEFAULT value to a BigDecimal value in java ?
is removing the id from my entity is the right way to go ?
how can I make sure that any changes I do to my code wont harm the mysql or any other data base that I will use ?
If you specify the column name in the insert query then postgres does not take the default value. So you should use your second insert query.
INSERT INTO URLS(TYPE, URL) VALUES(1, 'DED');
This syntax is correct for both postgres and MySQL.
This should resolve your question (1) and (3). For (2) DO NOT delete the id field from your entity. This id is going to be your link to the database row for a specific object of the entity.
1 - I think it is proper to use Long or long types instead of BigDecimal for id fields.
2 - Yes it generally helps, but it lowers portability. BTW, using an ORM framework like Hibernate may be a good choice.
3 - Integration testing usually helps and you may want to adopt TDD style development.
When using this statement:
INSERT INTO URLS(ID, TYPE, URL) VALUES(NULL, 1, 'DED');
you are telling the database that you want to insert a NULL value into the column ID and Postgres will do just that. Unlike MySQL, PostgreSQL will never implicitely replace a value that you supply with something totally different (actually all DBMS except MySQL work that way - unless there is some trigger involved of course).
So the only solution to is to actually use an INSERT that does not supply a value for the ID column. That should work on MySQL as well.