I have an Oracle (11) database with a table that has a self-referencing constraint.
I have some REALLY HORRIBLE java code, that has been happily running for a long time, that creates a BATCH of two inserts and runs them. This part of our UNIT TEST code and has been happily running for years.
The strange thing is that the insert of the CHILD is added to the batch before the insert of the PARENT.
I'd have thought that would never work. But it does
Well, it works on my original database, but not on a new "copy".
So my table is like this:
create table my_table (primary_key_id INT not null, related_id INT null);
alter table my_table add primary key (primary_key_id);
create index fk_my_table on my_table (related_id);
ALTER TABLE my_table ADD CONSTRAINT fkc_my_table FOREIGN KEY (related_id) REFERENCES my_table (primary_key_id);
The JAVA code is like this:
public void wossupDoc(Connection con) throws Exception {
String sql = "INSERT INTO my_table (primary_key_id, related_id) values (?,?)";
try(PreparedStatement ps = con.prepareStatement(sql)){
ps.setLong(1, 100);
ps.setLong(2, 101);
ps.addBatch();
ps.setLong(1, 101);
ps.setNull(2, Types.NUMERIC);
ps.addBatch();
ps.executeBatch();
}
(I created that table and put this java code into a unit test exactly like this so I could run it repeatedly against the different database)
So this code WORKS when run against DATABASE "A", but doesn't work when run against DATABASE "B".
So, that's the same code, the same JDBC drivers, but a different URL in the database connection properties... Yes it's always autocommit=false.
I've confirmed that, in the scenario where it works, there are two records in the database table.
The databases were created by two different DBAs a number of years apart, and they are nearly identical ... But are obviously different somehow.
Database versions:
Oracle Database 11g Release 11.2.0.3.0 - 64bit Production
Oracle Database 11g Release 11.2.0.4.0 - 64bit Production
(Yes I know those are REALLY OLD versions, I'm trying to migrate our code to an Oracle 19 database, and that's how I've discovered this scenario)
Does anyone know WHY this ever worked? Is there a database/connection parameter I could tweak to turn it on or off??
My true problem is that I'm wondering whether I've got any "batching" code in the main app that might try to do this, and so has been working accidentally for the past few years; and I won't know until we go live with the new database and the users start doing things...
As I investigate further, this looks like its a behavioural change in Oracle between 11.2.0.3.0 and 11.2.0.4.0. Checking more recent versions of Oracle the behaviour is consistent - the entries in the batch have to be valid based upon the sequence they are added to the batch. Looks like I have to accept this behaviour and fix the code/data that's failing. Makes sense, but more work for me.
Given 11.2 is so old a version of Oracle, no one else should really be getting similar problems. Thanks to everyone who looked at the question...
Related
I have a ton of raw html files that I'm parsing and inserting to a MySQL database via a connection in Java.
I'm using "REPLACE INTO" statements and this method:
public void migrate(SomeThread thread) throws Exception{
PreparedStatement threadStatement = SQL.prepareStatement(threadQuery);
thread.prepareThreadStatement(threadStatement);
threadStatement.executeUpdate();
threadStatement.close();
for(SomeThread.Post P : thread.threadPosts){
PreparedStatement postStatement = SQL.prepareStatement(postQuery);
P.preparePostStatement(postStatement);
postStatement.executeUpdate();
postStatement.close();
}
}
I am running 3 separate instances of my program each in its own command prompt, with their own separate directory of htmls to parse and commit.
I'm using HeidiSQL to monitor the database and a funny thing is happening where I'll see that I have 500,000 rows in a table at one point for example, then I'll close HeidiSQL and check back later to find that I now have 440,000 rows. The same thing occurs for the two tables that I'm using.
Both of my tables use a primary key called "id", each of their ID's have their own domain but it's possible their values overlap and are overwriting each other? I'm not sure if this could be an issue because I'd think SQL would differentiate between the table's "local" id values.
Otherwise I was thinking it could be that since I'm running 3 separate instances that each have their connection to the DB, some kind of magic is happening where right as one row is being committed, the execution swaps to another commit statement, displaces the table, then back to the first commit and then some more magic that causes the database to roll back the number of rows collected.
I'm pretty new to SQL so I'm not too sure where to start, if somebody has an idea about what the heck is going on and could point me in the right direction I'd really appreciate it.
Thanks
You might want to use INSERT INTO instead of REPLACE INTO.
Data doesn't disappear.
Here are some tips:
Do you have another thread running that actually deletes entries?
Do other people have access to the database?
Not sure what HeidiSQL may do. To exclude that possibility maybe use MySQL Workbench instead.
Yeah now that I run a COUNT(*) query against my tables I see that all my rows are in fact there.
Most likely the heidiSQL summary page is just a very rough estimate.
Thanks for the suggestion to use workbench pete, I will try it and see if it is better than Heidi as Heidi is freezing up on me on a regular basis.
I have an application that logs a lot of data to a MySQL database. The in-production version already runs insert statements in batches to improve performance. We're changing the db schema a bit so that some of the extraneous data is sent to a different table that we can join on lookup.
However, I'm trying to properly design the queries to work with our batch system. I wanted to use the mysql LAST_QUERY_ID so I wouldn't have to worry about getting the generated keys and matching them up (seems like a very difficult task).
However, I can't seem to find a way to add different insert statements to a batch, so how can resolve this? I assume I need to build a second batch and add all detail queries to that, but that means that the LAST_QUERY_ID loses meaning.
s = conn.prepareStatement("INSERT INTO mytable (stuff) VALUES (?)");
while (!queue.isEmpty()){
s.setLong(1, System.currentTimeMillis() / 1000L);
// ... set other data
s.addBatch();
// Add insert query for extra data if needed
if( a.getData() != null && !a.getData().isEmpty() ){
s = conn.prepareStatement("INSERT INTO mytable_details (stuff_id,morestuff)
VALUES (LAST_INSERT_ID(),?)");
s.setString(1, a.getData());
s.addBatch();
}
}
This is not how batching works. Batching only works within one Statement, and for a PreparedStatement that means that you can only add batches of parameters for one and the same statement. Your code also neglects to execute the statements.
For what you want to do, you should use setAutoCommit(false), execute both statement and then commit() (or rollback if an error occurred).
Also I'd suggest you look into the JDBC standard method of retrieving generated keys, as that will make your code less MySQL specific. See also Retrieving AUTO_INCREMENT Column Values through JDBC.
I've fixed it for now though I wish there was a better way. I built an arraylist of extra data values that I can associates with the generatedKeys returned from the batch inserts. After the first query batch executes, I build a second batch with the right ids/data.
I've been trying to understand how an update made from a different session will affect the session held by a runnning program, using JDBC with Oracle Driver (ojdbc6.jar, db version 11.2). I'm confused by the documentation over here, especially table 17-1: http://docs.oracle.com/cd/E11882_01/java.112/e16548/resltset.htm#JJDBC28628
Assuming the following code:
String SQL_QUERY = "select duration, cost from sample_table where name = ? and day = ? and type = ?";
PreparedStatement ps = connection.prepareStatement(SQL_QUERY);
ps.setString(1,name);
ps.setInt(2,day);
ps.setInt(3,type);
ResultSet rs = ps.executeQuery();
if (rs.next()) {
SampleObject so = new SampleObject(); //default constructor
so.setDuration(rs.getInt(1));
so.setRate(rs.getDouble(2));
}
rs.close();
//and so on...
Suppose I have ran this query several times, for several different parameters, and my program has been running for a couple of hours.
Through a different session (in my case a sqldeveloper connection), I update this table (or delete a row from it) and commit the update.
Right after that, I query the table through the code but I do not see the updated results, I keep seeing the old values. I repeat my test and the results keep coming back with the old values.
I stop the running program, start it again with no changes and after repeating the test, I see the updated values.
What's going on? I haven't set any explicit cache mechanisms, the code to query the database is as simple as it can be. Is there any default caching going on? How can I guarantee I would always see the most updated values, regardless of which session has changed them?
EDIT: One thing that I did not mention at first: when this showed up my first thought was obviouslly "oh, I forgot to commit", then commited again. No changes. Run the test with the code on my local machine, saw the new values. Ran in the server, did not see.
Changes only occur when you COMMIT them. Before you do that, you can't see the changes. Afterward you can see the change.
I have a table, say example1 and I'm using a jdbc statement to delete one of its rows. I have tried various methods, from delete from example1 where id = 1 to statement.addbatch(sql) but it does not delete the row. If I execute the same sql statement in Toad for Mysql it's able to delete the row just fine.
Weird thing is that using jdbc I am able to delete rows from other tables just fine; it's just this one particular table giving me unexpected results.
There is nothing special about this table. It has a primary key and no constraints/foreign key relationships.
Also, this delete is a part of a transaction so auto-commit is set to false and once all records get updated/inserted/deleted then the commit is done. This does not seem to have any problem with any other table and all the updates/deletes/inserts are done just fine.
Permission-wise this table has same permission for the db user that any other table in the db.
Any ideas or pointers will be greatly appreciated!
Turning on general logging on the database or profiling in the JDBC driver would show you what's actually going to the database:
http://dev.mysql.com/doc/refman/5.0/en/connector-j-reference-configuration-properties.html
Enable profiling of queries for Connector/J by adding this to your connection string: profileSQL=true
General Logging documentation:
http://dev.mysql.com/doc/refman/5.1/en/query-log.html
There's also mk-query-digest for sniffing your network traffic and analyzing the results:
http://www.maatkit.org/doc/mk-query-digest.html
I have come across the same situation
I remember there was a defnitely mistake in the query
Try to execute the query in the mysql sqlyog or any GUI and check if it works, i am 100% sure it wont work
then correct the query and check it
I need to insert a record to table if the record doesn't exist, and to update a record if the record exists in the table.
Of course, I can write:
p-code:
SELECT * FROM table1 WHERE id='abc' by JDBC
if(exists)
UPDATE table1 SET ... WHERE id='abc' by JDBC;
else
INSERT INTO table1... by JDBC;
However, I don't think the code is elegant.
Alternatively, I can also write it in this way:
p-code:
int row = Statement.executeUpdate("INSERT INTO table1...", 2);
if(row==0)
update table1 SET ... WHERE id='abc' by JDBC;
Do you think the latter way is better and faster? Thanks!
EDIT: in MYSQL
It depends on what type of database your are using and whether or not you can take advantage of database specific features. MySQL for instance lets you do the following:
INSERT INTO territories (code, territory) VALUES ('NO', 'Norway')
ON DUPLICATE KEY UPDATE territory = 'Norway'
However, the above is not standard (SQL-92) compliant. That is, it will most likely not work on all databases. In other words, you would have to stick with the code as you have written it. It might not look that elegant, but it is probably the most safe solution to go with.
You might want to look at using the DBMS to do the check within a single statement i.e. use the SQL EXISTS condition: WHERE EXISTS or WHERE NOT EXISTS
Maybe the database you are using has an insert or update feature which solves this automatically for you. In DB2 you can use MERGE INTO for example. See here
This is probably the reason to switch to one of popular ORM solutions (Hibernate, Toplink, iBatis). These tools "know" various SQL dialects and optimise your queries accrodingly.