We have a need to find a way to copy certain data from production into our dev regions so that we can debug/fix any issue.
Sometimes single user related data gets impacted. We have to replicate the same scenario in dev and find a solution.
Presently we follow two approaches:-
1. Check the audit history and try to recreate the similar scenario
in dev. <50% sucess rate in recreating the exact same scenario.
2. Restore+Encrypt the "whole" production into dev and then continue
on the work. It is an overkill if issue impacts only a single user.
So I am trying to find a way to just select a single user data from production and insert it into dev region.
We just have Java and Oracle. Can't use any external tools. Because we
dont have license and cannot download freeware due to security issues.
I tried the follwing:-
Write a java code which will query the informaition schema tables to find the relationships between the tables and create select statements like below:-
select 'insert into TABLE1(C1,C2,C3,C4) values ('||''''||C1||''''||','||coalesce(to_char(C2),'null')||','||''''||C3||''''||','||coalesce(to_char(C4),'null'));'
from TABLE1 where ID='1006' union all
select 'insert into TABLE2(C1,C2,C3,C4) values ('||''''||C1||''''||','||coalesce(to_char(C2),'null')||','||''''||C3||''''||','||coalesce(to_char(C4),'null'));'
from TABLE2 WHERE TABLE1ID in ( select ID FROM TABLE1 where ID='1006') union all
select 'insert into TABLE3(C1,C2,C3,C4) values ('||''''||C1||''''||','||coalesce(to_char(C2),'null')||','||''''||C3||''''||','||coalesce(to_char(C4),'null'));'
from TABLE3 WHERE TABLE2ID in ( select ID FROM TABLE2 WHERE TABLE1ID in ( select ID FROM TABLE1 where ID='1006'));
2. Use this set of selects in production, so that you get a set of insert statements as output.
3. Use the insert statements in dev.
Problem:-
The select queries are becoming huge. Around 25 MB in total :(
We cannot even execute that big query in production.
Could you suggest any better approach for this usecase?
Does oracle itself allow selective data exports? Or any other way I should write my java code?
We use something like this to move records from one database to another:
copy from username/password#database1 to username/password#database2 insert target_table using select * from source_table where where_clause_goes_here;
Use datapump to move data for the tables you need and with the whereclause you want. Straight forward and standard functionality of the database.
If both the DBs are Oracle, you can create a DBLINK in your local
database for the remote DB and Create a job in your local DB that queries
all the data from remote DB using the DBLINK, and update the tables in your
local database.
Or there are plenty of data migration API are availabe you can give a try to one of them.
Below are the some link,have a look in to them,may be it will solve your problem
http://code.google.com/p/c5-db-migration/
http://flywaydb.org/documentation/migration/java.html
http://migrate4j.sourceforge.net/
http://flywaydb.org/ --- its better to use
http://www.operatornew.com/2012/11/automatic-db-migration-for-java-web.html
Related
With SQL I can copy data from one table to another mirror table. (e.g. insert into TABLE_EXAMPLE_COPY select * from TABLE_EXAMPLE; .
How can I do the same thing using Hibernate org.hibernate.Criteria or org.hibernate.Query or org.hibernate.SQLQuery?
If you want to perform that action from within the boundaries of JPA or Hibernate, the best way to accomplish that is to use a Native SQL statement.
session
.createNativeQuery( "INSERT INTO table_copy SELECT * FROM table" ).
.executeUpdate();
The other options involve reading the source table into a POJO and then transforming that into the POJO representation for the copy table and saving those rows. The problem with these is that you also introduce network latency and JVM overhead just to create an in-memory object, transform it, and then push it back over the network to the database.
The presented above solution avoids all those drawbacks and allows the database to handle all that in the best performing way it knows how.
I don't really ask a lot of questions but This time, it's too much. Here's the problem.
I have these two database (Sybase) servers and there's this database with over 90 tables but i need to archive only 20 tables.
These tables are however quite large and can contain up to 90million records. So here's the deal. Currently, what I do is that
For the big tables (alot of records), I create a temp table and copy from the temp table to the destination but running an insert for each statement.
After the copying is done, i drop the temp table created.
Now, I've tried other methods like for instance.
Up to now, the multi threading is just okay but the speed of archival is not good enough. for instance, it can archive up to 1.6M records within one hour. That is not good enough for my Boss.
Kindly advice on any other solution, approach or thought you'd think can help. Please not that all solutions are welcome.
Thanks in advance.
Do not copy such amount of data by yourself. Create database jobs to copy/archive tables. And monitor the output/logs of those jobs in your application. It will be much faster.
Generate SQL executive script and pass it to database. It mean fetch all records from select statement and create insert/update statements:
String query = "UPDATE OR INSERT INTO TABLE (ID, VALUE) VALUES (9, 2) MATCHING (IDPRODUCT, COUNT); "+
"UPDATE OR INSERT INTO TABLE (ID, VALUE) VALUES (10, 1) MATCHING (IDPRODUCT, COUNT); "+
"COMMIT WORK;";
If Sybase can connect to other Sybase instance create procedure for execute previous clause. For FierebirdSQL it possible through ON EXTERNAL and EXECUTE PROCEDURE with procedure name as parameter instructions.
The users need to monitor and know how the tool is running. For each commited table update user interface.
Please help me in finding a solution for
merging multiple tables with similar names in MySQL [duplicate] from different Databases
Our application was deployed in Multiple Schema based on the Locations.
Now we are going to merge the data of different databases to single database of MySql
As I am new to Database Migration , I really likes to take advice of experts.
1) Any good tools can be used for this process.
2) What are the precautions to be done for this.
The application is in Spring,Hibernate, Java
Database is MySql
you should use table name with there database name..
For example if you want to merge as union or join the use table name as:
Select * from db_name1.table_name
union
select * from db_name2.table_name...
Otherwise clarify your question.
I have 2 DBs, Database A and Database B.
What I want to achieve:
build records from Database A and insert them to Database B
Process those records in my java app
What I'm currently doing:
I use two separate queries:
For (1) I use INSERT INTO ... SELECT ...
For (2) I perform another SELECT.
My solution works but it isn't optimal since I'm getting the records from Database A twice (instead of just one time).
Is there a way to execute the INSERT INTO ... SELECT ... and get the inner select result as a ResultSet?
I know I can perform only a SELECT and then insert the records in a batch, but thats a bit cumbersome and I want to find out if there's a cleaner solution.
Your cleaner solution look more cumbersome than simple read and write operation.
As you have to manipulate data in database B. You simply do this
Read Data from A to your app
Process data
Write data to B from your app
Then you have singe read single write and is simple.
You can not gain the result of INSERT INTO as Result set as this is INSERT statement
Sadly, I do not think that this is possible. What you are trying to achieve are two distinct operations i.e. an INSERT and a SELECT. However you cut it you are still going have to do at least one INSERT and one SELECT.
use this for two database
INSERT INTO Database2 (field1,field2,field3){
SELECT * FROM Database1;);
Both the database have the same field name.
I need to insert a record to table if the record doesn't exist, and to update a record if the record exists in the table.
Of course, I can write:
p-code:
SELECT * FROM table1 WHERE id='abc' by JDBC
if(exists)
UPDATE table1 SET ... WHERE id='abc' by JDBC;
else
INSERT INTO table1... by JDBC;
However, I don't think the code is elegant.
Alternatively, I can also write it in this way:
p-code:
int row = Statement.executeUpdate("INSERT INTO table1...", 2);
if(row==0)
update table1 SET ... WHERE id='abc' by JDBC;
Do you think the latter way is better and faster? Thanks!
EDIT: in MYSQL
It depends on what type of database your are using and whether or not you can take advantage of database specific features. MySQL for instance lets you do the following:
INSERT INTO territories (code, territory) VALUES ('NO', 'Norway')
ON DUPLICATE KEY UPDATE territory = 'Norway'
However, the above is not standard (SQL-92) compliant. That is, it will most likely not work on all databases. In other words, you would have to stick with the code as you have written it. It might not look that elegant, but it is probably the most safe solution to go with.
You might want to look at using the DBMS to do the check within a single statement i.e. use the SQL EXISTS condition: WHERE EXISTS or WHERE NOT EXISTS
Maybe the database you are using has an insert or update feature which solves this automatically for you. In DB2 you can use MERGE INTO for example. See here
This is probably the reason to switch to one of popular ORM solutions (Hibernate, Toplink, iBatis). These tools "know" various SQL dialects and optimise your queries accrodingly.