I am attempting to create a test database (based off of my production db) at runtime, but rather than have to maintain an exact duplicate test db i'd like to copy the entire data structure of my production db at runtime and then when I close the test database, drop the entire database.
I assume I will be using statements such as:
CREATE DATABASE test //to create the test db
CREATE TABLE test.sampleTable LIKE production.sampleTable //to create each table
And when I am finished with the test db, calling a close method will run something like:
DROP DATABASE test //delete the database and all its tables
But how do I go about automatically finding all the tables within the production database without having to manually write them out. The idea is that I can manipulate my production db without having to be concerned with maintaining the structure identically within the test db.
Would a stored procedure be necessary in this case? Some sample code on how to achieve something like this would be appreciated.
If the database driver you are using supports it, you can use DatabaseMetaData#getTables to get the list of tables for a schema. You can get access to DatabaseMetaData from Connection#getMetaData.
In your scripting language, you call "SHOW TABLES" on the database you want to copy. Reading that result set a row at a time, your program puts the name of the table into a variable (let's call it $tablename) and can generate the sql: "CREATE TABLE test.$tablename LIKE production.$tablename". Iterate through the result set and you're done.
(You won't get foreign key constraints that way, but maybe you don't need those. If you do, you can run "SHOW CREATE TABLE $tablename" and parse the results to pick out the constraints.)
I don't have a code snippet for java, but here is one for perl that you could treat as pseudo-code:
$ref = $dbh->selectall_arrayref("SHOW TABLES");
unless(defined ($ref)){
print "Nothing found\n";
} else {
foreach my $row_ref (#{$ref}){
push(#tables, $row_ref->[0]);
}
}
The foreach statement iterates over the result set in an array reference returned by the database interface library. The push statement puts the first element of the current row of the result set into an array variable #tables. You'd be using the database library appropriate for your language of choice.
I would use mysqldump : http://dev.mysql.com/doc/refman/5.1/en/mysqldump.html
It will produce a file containing all the sql commands needed to replicate the prod database
The solutions was as follows:
private static final String SQL_CREATE_TEST_DB = "CREATE DATABASE test";
private static final String SQL_PROD_TABLES = "SHOW TABLES IN production";
JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
jdbcTemplate.execute(SQL_CREATE_TEST_DB);
SqlRowSet result = jdbcTemplate.queryForRowSet(SQL_PROD_TABLES);
while(result.next()) {
String tableName = result.getString(result.getMetaData().getColumnName(1)); //Retrieves table name from column 1
jdbcTemplate.execute("CREATE TABLE test2." + tableName + " LIKE production." + tableName); //Create new table in test2 based on production structure
}
This is using Spring to simplify the database connection etc, but the real magic is in the SQL statements. As mentioned by D Mac, this will not copy foreign key constraints, but that can be achieved by running another SQL statement and parsing the results.
Related
Description of code
Database connection
I try to to store Java object locally database without use external database.
For this I use JDBC with H2 via Hibernate :
/**
* #param connection the connection to set
*/
public static void setConnectionHibernate() {
Properties connectionProps = new Properties();
connectionProps.put("user", "sa");
try {
Class.forName("org.h2.Driver");
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
url = "jdbc:h2:mem:db1;DB_CLOSE_DELAY=-1;MODE=MySQL;";
}
Query
I store the PROCEDURE in String with this code :
static final String CREATE_PROCEDURE_INITPSEUDOS = "CREATE OR REPLACE PROCEDURE init_pseudos (MaxPseudo INT) BEGIN WHILE MaxPseudo >= 0 DO"
+
" INSERT INTO Pseudos (indexPseudo)" +
" VALUES (MaxPseudo);" +
" SET MaxPseudo = MaxPseudo - 1;" +
" END WHILE;" +
" END init_pseudos;";
Query execution
And I execute the statement with this code :
public static void initBaseDonneePseudos() {
try (Connection connection = DriverManager.getConnection(url, connectionProps);
Statement stmt = connection.createStatement()) {
stmt.execute(RequetesSQL.CREATE_TABLE_PSEUDOS);
stmt.execute(RequetesSQL.CREATE_PROCEDURE_INITPSEUDOS);
stmt.execute(RequetesSQL.CREATE_FUNCTION_RECUPEREPSEUDO);
stmt.execute(RequetesSQL.INIT_TABLE_PSEUDOS);
} catch (SQLException e) {
e.printStackTrace();
}
}
Problem
Test
I execute this test to test statement :
#Nested
class BaseDonneeInteractionTest {
#BeforeEach
public void setUp() {
BaseDonnee.setConnectionHibernate();
}
#Test
void testInitBaseDonnee() {
assertDoesNotThrow(() -> BaseDonnee.initBaseDonneePseudos());
}
}
Error
But I obtain this error
I didn't find the problem of the query, anybody have the solution to solve this ?
The "MySQL Compatibility Mode" doesn't make H2 100% compatible with MySQL. It just changes a few things. The documentation lists them:
Creating indexes in the CREATE TABLE statement is allowed using INDEX(..) or KEY(..). Example: create table test(id int primary key, name varchar(255), key idx_name(name));
When converting a floating point number to an integer, the fractional digits are not truncated, but the value is rounded.
ON DUPLICATE KEY UPDATE is supported in INSERT statements, due to this feature VALUES has special non-standard meaning is some contexts.
INSERT IGNORE is partially supported and may be used to skip rows with duplicate keys if ON DUPLICATE KEY UPDATE is not specified.
REPLACE INTO is partially supported.
Spaces are trimmed from the right side of CHAR values.
REGEXP_REPLACE() uses \ for back-references.
Datetime value functions return the same value within a command.
0x literals are parsed as binary string literals.
Unrelated expressions in ORDER BY clause of DISTINCT queries are allowed.
Some MySQL-specific ALTER TABLE commands are partially supported.
TRUNCATE TABLE restarts next values of generated columns.
If value of an identity column was manually specified, its sequence is updated to generate values after inserted.
NULL value works like DEFAULT value is assignments to identity columns.
Referential constraints don't require an existing primary key or unique constraint on referenced columns and create a unique constraint automatically if such constraint doesn't exist.
LIMIT / OFFSET clauses are supported.
AUTO_INCREMENT clause can be used.
YEAR data type is treated like SMALLINT data type.
GROUP BY clause can contain 1-based positions of expressions from the SELECT list.
Unsafe comparison operators between numeric and boolean values are allowed.
That's all. There is nothing about procedures. As #jccampanero pointed out in the other answer, you must use the syntax specific to H2 if you want to create stored procedures.
The problem is that in H2 there are not explicit procedures or functions as you are trying defining.
For that purpose, H2 allows you to create used defined functions instead. Please, consider reed the appropriate documentation.
Basically, you create a user defined function by declaring an ALIAS for a bunch of Java code.
For example, in your use case, your CREATE_PROCEDURE_INITPSEUDOS could look similar to this:
CREATE ALIAS INIT_PSEUDOS AS $$
import java.sql.Connection;
import java.sql.Statement;
import java.sql.SQLException;
#CODE
void init_pseudos(final Connection conn, final int maxPseudo) throws SQLException {
try (Statement stmt = conn.createStatement()) {
while (maxPseudo >= 0) do {
stmt.execute("INSERT INTO Pseudos (indexPseudo) VALUES (MaxPseudo);");
maxPseudo = maxPseudo - 1;
}
}
}
$$;
Note the following:
As I said, you define a user defined function as Java code. That Java code should be enclosed between two $$ delimiters.
Although I included explicitly some imports, you can use any class in the java.util or java.sql packages in your code. If you want to included explicitly some imports, or if you require classes from other packages than the mentioned, the corresponding imports should be provided right after the first $$ token. In addition, you need to include #CODE the signal H2 where your imports end and your actual Java method starts.
If you need a reference to a Connection to the database in your code, it should be the first argument of your method.
Prefer to raise and not hide exceptions: it will allow your transactions to be committed or rollbacked as a whole appropriately.
You can invoke such a function as usual:
CALL INIT_PSEUDOS (5);
Please, provide the appropriate value for the maxPseudo argument.
Please, consider the provided code as just an example of use: you can improve the code in different ways, like using PreparedStatements instead of Statements for efficiency purposes, checking parameters for nullability, etcetera.
Using H2 as a test database product
Historically, H2 has been used a lot as a test replacement for the actual production database product. But this means you're limiting yourself to the least common denominator between H2 and MySQL, in your case. Including, as others pointed out, the lack of support for stored procedures. You could re-implement all the stored procedures in H2 (very tedious and error prone), or, you could just use testcontainers and run actual integration tests on MySQL directly.
I really think that using H2 as an integration test database is an outdated concept as I've shown in this blog post (unless you're also using H2 in production). You'll be much happier developing and testing everything against MySQL directly!
Using recursive SQL instead
You don't really need that procedure, I think? You probably wrote it to avoid too many round trips to the database for that loop. But you could batch the inserts or generate the following recursive SQL:
INSERT INTO Pseudos (indexPseudo)
WITH RECURSIVE
p (i) AS (
SELECT MaxPseudo
UNION ALL
SELECT i - 1 FROM p WHERE i >= 0
)
SELECT i FROM p;
Now there's no more need to place this in a procedure, you can run the query directly.
Translating your procedure to H2
Just to be complete, for translation of simple procedural logic, jOOQ would offer the feature transparently. You can try it online, here. It's probably overkill, I recommend the other approaches, but perhaps worth a try.
Disclaimer: I work for the company behind jOOQ.
I have a table PERSON with more than 5 millions rows and I need to update field NICKNAME on each one of them based on the field NAME inside the same table.
ResultSet rs = statement.executeQuery("select NAME from PERSON");
while(rs.next())
{
// some parsing function like:
// Nickname = myparsingfunction(rs.getString("NAME"));
rs.updateString( "NICKNAME", Nickname );
rs.updateRow();
}
But I got this error:
not implemented by SQLite JDBC driver
I'm using sqlite-jdbc-3.8.11.2.jar downloaded at https://bitbucket.org/xerial/sqlite-jdbc/downloads.
I know I could use the following SQL query:
statement.executeUpdate("update PERSONS set NICKNAME = Nickname where ID = Id");
But that would take forever and I understand updating ResultSet would be faster. So what options do I have to update the table on the fastest way? Any other driver available? Should I move out of Java?
UPDATE
I was able to find a fast solution using below syntax. The block between CASE and END was a concatenated string that I built before executing the SQL query, so I could send all updates at once.
update PERSON
set NICKNAME= case ID
when 173567 then 'blabla'
when 173568 then 'bleble'
...
when 173569 then 'blublu'
end
where ID in (173567, 173568, 173569)
As you have encountered, the SQLite JDBC driver does not currently support the updateString operation. This can be seen in the source code for this driver.
I can think of three options:
As you stated in your question, you can select the name and ID of the person and then update the person by its ID. Those updates could be done in a batch (using PreparedStatement.addBatch()) to improve performance (tutorial).
Implement the method myparsingfunction in pure SQL so that the query could become UPDATE PERSONS SET NICKNAME = some_function(NAME).
Create an user-defined function (using org.sqlite.Function), implemented in Java, and call it inside the SQL. Example, taken from this answer:
Function.create(db.getConnection(), "getNickName", new Function() {
protected void xFunc() throws SQLException {
String name = value_text(0);
String nickName = ...; // implement myparsingfunction here
result(nickName);
}
});
and use it like this: UPDATE PERSONS SET NICKNAME = getNickName(NAME);
SQLite does not support stored procedures so that option is out of the table.
I'm not sure which of these options would provide the best performance (certainly using pure SQL would be faster but that may not be a viable solution). You should benchmark each solution to find the one that fits you.
I am using an APACHE DERBY database, and basing my database interactions on EntityManager, and I don't want to use JDBC class to build a query to change my tables' names (i just need to put a prefix to each new user to the application, but have the same structure of tables), such as:
//em stands for EntityManager object
Query tableNamesQuery= em.createNamedQuery("RENAME TABLE SCHEMA.EMP_ACT TO EMPLOYEE_ACT");
em.executeUpdate();
// ... rest of the function's work
// The command works from the database command prompt but i don't know how to use it in a program
//Or as i know you can't change system tables data, but here's the code
Query tableNamesQuery= em.createNamedQuery("UPDATE SYS.SYSTABLES SET TABLENAME='NEW_TABLE_NAME' WHERE TABLETYPE='T'");
em.executeUpdate();
// ... rest of the function's work
My questions are :
This syntax is correct?
Will it work?
Is there any other alternative?
Should I just use the SYS.SYSTABLES and find all the tables that has 'T' as tabletype and alter their name their, will it change the access name ?
I think you're looking for the RENAME TABLE statement: http://db.apache.org/derby/docs/10.10/ref/rrefsqljrenametablestatement.html
Don't just issue update statements against the system catalogs, you will corrupt your database.
I want to display the data of a postgresql database using the CRUD in the play framework; I looked for any examples to get idea which I didn't find after a long time of searching in google. Someone help me with this if you can or post a valid link regarding this. Thanks in advance!
I use, Play 1.2.5, java and postgresql.
I assume you want to do this in your application code in runtime.
You can execute query using DB plugin to search DB metadata using native PostgreSQL queries.
Here's an example how to get column names of my system DOCUMENT table:
List<String> columns = new ArrayList<>();
ResultSet result = DB.executeQuery("select column_name from INFORMATION_SCHEMA.COLUMNS where table_name = 'DOCUMENT';");
while(result.next()) {
String column = result.getString(1);
columns.add(column);
}
This note that code is somewhat simplified and you should use prepared statements if anything in this query will be inserted from data that user or any other system entered.
Use DB.getConnection().prepareStatement() to get PreparedStatement instance.
I am performing a call to a function which is part of a DB package. This package is deployed in two locations. One local and another remote (across the Atlantic).
I am retrieving the data via the Spring JDBC template.
There is one function which returns approximately 1000 rows (not all that much) and this is taking about 1.5 seconds when getting the data locally but it's taking in the region of 12 seconds when getting the data remotely.
In all sample code, names have been changed and code has been simplified a little.
Please see an example of the current Java code:
SimpleJdbcCall simpleJdbcCall = new SimpleJdbcCall(getDataSource())
.withSchemaName(MY_SCHEMA_NAME)
.withCatalogName("REFCURSOR_PKG")
.withFunctionName("GET_DATA")
.returningResultSet("RESULT_SET", new DataEntryMapper());
SqlParameterSource params = new MapSqlParameterSource()
.addValue("the_name", name)
.addValue("the_rev", rev);
Map resultSet = simpleJdbcCall.execute(params);
ArrayList list = (ArrayList) resultSet.get("RESULT_SET");
The RowMapper class looks something like this:
class RouteDataEntryMapper implements RowMapper {
public RouteDataEntry mapRow(ResultSet resultSet, int rowNum) throws SQLException {
return new DataEntry(resultSet.getString("name"),
Integer.parseInt(resultSet.getString("rev"));
}
}
SQL package spec snippet:
TYPE REF_CURSOR IS REF CURSOR;
SQL function:
FUNCTION GET_ROUTE_DATA(the_name VARCHAR2, the_rev VARCHAR2) RETURN REF_CURSOR AS
RESULT_SET REF_CURSOR;
BEGIN
OPEN RESULT_SET FOR
select *
from table_name tn
where tn.name = the_name
and tn.rev = the_rev;
RETURN RESULT_SET;
CLOSE RESULT_SET;
EXCEPTION WHEN OTHERS THEN
RAISE;
END GET_ROUTE_DATA;
I have tried using regular boiler plate JDBC also (create connection, prepare statement, execute statement, retrieve data from RESULT_SET, etc) and I found that the vast majority of time was spent looping over the RESULT_SET and extracting the data out of it and into some pojos. In the case of the Spring code above, most of the time was spent during the execute() method but this is probably because it creates the objects using the RowMapper at that time.
So, the common area between them is the performing of actions such as:
rs.getString("name")
and I'm guessing that this is where the problem lies but I could be wrong.
As I said, locally the delay is fine but remotely it's taking way too long. Is this because it's going to the DB on every rs.get... ? Is there a better way to do this?
Thanks in advance.
rs.getString("name")
ResultSet.get*(String columnName) can be replaced with ResultSet.get*(int columnNaumber) which is slightly faster but I doubt that the main problem here.
Is this because it's going to the DB on every rs.get... ?
While it really depends the driver I suspect it won't. For a cached result-set it might go to ther server when your scroll through the cursor but it would still fetch a bunch of rows in every roundtrip.
Two more suggestions I have are:
Use a network sniffing utility to see the data being transferred
Check your driver for any option to pre-fetch and such like.
add this line :-
.withoutProcedureColumnMetaDataAccess
in the following code lines
SimpleJdbcCall simpleJdbcCall = new SimpleJdbcCall(getDataSource())
.withSchemaName(MY_SCHEMA_NAME)
.withCatalogName("REFCURSOR_PKG")
.withFunctionName("GET_DATA")
.withoutProcedureColumnMetaDataAccess // to avoid fetching meta data info from database