We Keep Coding

Compare very large tables in java

I am not able to find any satisfying solution so asking here.
I need to compare data of two large tables(~50M) with the same schema definition in JAVA.
I can not use order by clause while getting the resultset object and records might be not in order in both of the tables.
Can anyone help me what can be the right way to do it?

You could extract the data of the first DB table into a text file, and create a while loop on the resultSet for the 2nd table. As you iterate through the ResultSet do a search/verify against the text file. This solution works if memory is of concern to you.
If not, then just use a HashMap to hold the data for the first table and do the while loop and look up the records of the 2nd table from the HashMap.

This really depends on what you mean by 'compare'? Are you trying to see if they both contain the exact same data? Find rows in one not in the other? Find rows with the same primary keys that have differing values?
Also, why do you have to do this in Java? Regardless of what exactly you are trying to do, it's probably easier to do with SQL.
In Java, you'll want to create an class that represents the primary key for the tables, and a second classthat represents the rest of the data, which also includes the primary key class. If you only have a single column as the primary key, then this is easier.
We'll call P the primary key class, and D the rest.
Map map = new HashMap();
Select all of the rows from the first table, and insert them into the hash map.
Query all of the rows in the second table.
For each row, create a P object.
Use that to see what data was in the first table with the same Key.
Now you know if both tables contained the same row, and you can compare the non-key values from both both.
Like I said, this is much much easier to do in straight SQL.
You basically do a full outer join between the two tables. How exactly that join looks depends on exactly what you are trying to do.

How to execute a SQL statement in Java with many values in a single variable in where in clause

I have to execute below query through JDBC call
select primaryid from data where name in ("abc", adc", "anx");
Issue is inside in clause I have to pass 11000 strings. Can I use prepared statement here? Or any other solution any one can suggest. I dont want to execute the query for each record, as it is consuming time. I need to run this query in very less time.
I am reading the strings from an XML file using DOMParser. and I am using sql server db.

I'm just wondering why you would need to have a manual set of 11,000 items where you need to specify each item. It sounds like you need to bring the data into a staging table
(surely it's not been selected from the UI..?), then join to that to get your desired resultset.

Using an IN clause with 11k literal values is a really bad idea - off the top of my head, I know one major RDBMS (Oracle) that doesn't support more than 1k values in the IN list.
What you can do instead:
create some kind of (temporary) table T_NAMES to hold your names; if your RDBMS doesn't support "real" (session-specific) temporary tables, you'll have to add some kind of session ID
fill this table with the names you're looking for
modify your query to use the temporary table instead of the IN list: select primaryid from data where name in (select name from T_NAMES where session_id = ?session_id) or (probably even better) select primaryid from data join t_names on data.name = t_names.name and t_names.session_id = ?session_id (here, ?session_id denotes the bind variable used to pass your session id)

A prepared statement will need to know the number of arguments in advance - something along the lines of :
PreparedStatement stmt = conn.prepareStatement(
"select id, name from users where id in (?, ?, ?)");
stmt.setInt(1);
stmt.setInt(2);
stmt.setInt(3);
11,000 is a large number of parameters. It may be easiest to use a 'batch' approach as described here (in summary - looping over your parameters, using a prepared statement
each time)
Note - if your 11,000 strings are the result of an earlier database select, then the best approach is to write a stored procedure to do the whole calculation in the database (avoiding passing the 11,000 strings back and forth with your code)

You can merge all your parameter strings into one bitg string separating by ';' char
bigStrParameter=";abc;adc;anx;"
And use LOCATE to find substring.
select primaryid from data where LOCATE(concat(';',name,';'),?)>=0;

How to retrieve previously auto-generated PK ID value using JDBC and HSQLDB

I'm working with JDBC and HSQLDB 2.2.9. What's the most efficient and accurate way to insert a new row into a DB and, subsequently, retain its id (PK set to autoincrement) value? The reason I need to do this is probably pretty obvious, but I'll illustrate with an example for discussion:
Say there's a Customer table that has a PersonId field with a FK constraint referring to a row from a Person table. I want to create a new Customer, but to do this I need to first create a new Person and use the new Person.id value to set Customer.PersonId.
I've seen four ways to approach this:
Insert the Person row setting the id field to null. HSQLDB generates the next id value automatically. Then perform a query on the Person table to get the id value just created and use it to create the new Customer row.
This seems expensive just to retrieve a single integer value.
Get the next id value in the Person table and use it in the INSERT statement to set the Person.id value manually. Use the same id value to set Customer.PersonId. No subsequent read from the DB is needed.
Inconsistencies may arise if an id value is obtained, but another connection performs an INSERT in the table before my INSERT INTO Person... statement is executed.
Execute the INSERT statement, as in option 1 above, setting id=null to allow auto-generation. Then use the getGeneratedKeys method to retrieve keys generated in last statement.
I thought this sounded like a good option, but I couldn't get it to work. Here's a snippet of my code:
// PreparedStatement prepared previously...
preparedStatement.executeUpdate();
ResultSet genKeys = preparedStatement.getGeneratedKeys();
int id;
if (genKeys.next()) {
id = genKeys.getInt(1);
}
// Finish up method...
This code was returning an empty ResultSet for genKeys. Am I using the getGeneratedKeys method incorrectly? If I could get this to work, this might be the way to go.
Again, execute the INSERT statement allowing for auto-generated id. Then immediately execute CALL IDENTITY() to retrieve the last id value generated by the connection (as explained here and mentioned in this SO question).
This also seems like a reasonable option, even though I must perform an additional executeQuery. On the positive side, I was actually able to get it to work with the following code:
// INSERT statement executed here...
statement = connection.createStatement();
ResultSet rs = statement.executeQuery("CALL IDENTITY();");
int id;
if (rs.next()) id = rs.getInt(1);
// Finish up method...
So, in summary, the first two options I'm not crazy about. The second two seem ok, but I could only get option 4 to work. Which option is preferred and why? If option 3 is the best, what am I doing wrong? Also, is there a better way that I haven't mentioned? I know words like 'better' can be subjective, but I'm working with a simple DB and want the most direct solution that doesn't open up the DB to possible inconsistencies and doesn't increase the transaction failure rate (due to trying to create a record with an id that already exists).
This seems like a basic question (and essential), but I couldn't find much guidance on the best way to do it. Thanks.
EDIT:
I just found this question that discusses my option 3. According to the accepted answer, it appears I was leaving out the Statement.RETURN_GENERATED_KEYS parameter needed to enable that functionality. I didn't show the prepareStatement method in my code snippet, but I was using the single parameter version. I need to retry using the overloaded, two-parameter version.
There are also a few other SO questions which show up with that question that are closly related to my question. So, I guess mine could be considered a duplicate (not sure how I missed those other questions before). But I'd still like any guidance on whether one solution is considered better than the others. For now, if I get option 3 to work, I'll probably go with that.

I don't have enough reputation to comment on neizan's answer, but here's how I solved the same problem:
The column looked like an ID column, but it wasn't defined as IDENTITY;
As said above, you need to specify RETURN_GENERATED_KEYS.
It looks like if you execute 2 INSERT in sequence, the second one won't return the generated keys. Use "CALL IDENTITY()" instead.
Example using HSQLDB 2.2.9:
CREATE TABLE MY_TABLE (
ID INTEGER IDENTITY,
NAME VARCHAR(30)
)
Then in Java:
PreparedStatement result = cnx.prepareStatement(
"INSERT INTO MY_TABLE(ID, NAME) VALUES(NULL, 'TOM');",
RETURN_GENERATED_KEYS);
int updated = result.executeUpdate();
if (updated == 1) {
ResultSet generatedKeys = result.getGeneratedKeys();
if (generatedKeys.next()) {
int key = generatedKeys.getInt(1);
}
}

Not much action here, so I'll go ahead and answer to bring closure to this question. After playing around with the different options, and after see this question, I was able to get my option 3 to work. Like I mentioned in the edit to my question, I'm going to use option 3. Option 4 also worked fine, but since the accepted answer to the linked question is given by a reputable source, I am sticking with that. I wish I'd have seen that question/answer before starting this one, I'd have saved some time!

insert data into a lot of columns in java JDBC

I have a table with 50 columns and I want to insert all items in a HashMap variable into it (HashMap keys and table column names are the same).
How can I do that without writing 50 lines of code?

Get the key set for the HashMap. Iterate that key set to build a String containing your insert statement. Use the resulting String to create a PreparedStatement. Then iterate that key set again to set parameters by name using the Objects you retrieve from the HashMap.
You might have to write a few extra lines of special-case code if any of your values are of a Class that the JDBC driver isn't sure how to map.

I'd suggest you bite the dust and simply write a method that will do the dirty work for you containing 50 lines of parameter setting code. This isn't so bad, and you only have to write it once. I hope you aren't that lazy ;-)
And by the way, isn't 50 columns in a table a bit much? Perhaps a normalization process could help and lower complexity of your database and the code that will manipulate it.
Another way to go is to use an ORM like Hibernate, or a more lightweight approach like Spring JDBC template.

Call map.keySet() to get the name of all columns.
Create an INSERT statement by iterating the key set.
The column is from an item (a key) in the key set.
The data is from map.get(key).

how to convert the changes that I have done in a resultset to sql queries in java?

I am comparing 2 resultsets, and I have to update one resultset according to the data in another. I can do this easily using updateRow (or insertrow, if required). But I also need to generate sql query (preferably with oracle syntax) and add into an sql file, which gives me the option of updating that later. Can anyone tell me an elegant way of doing this?

Not exactly sure what you are looking to do, but I believe you are looking to update one table with data from another, or insert missing data into the first table when it is in the second. The easiest way to do this would be to use a merge:
merge into t1
using t2
on (t2.id = t1.id)
when matched
then
update
set t1.col1 = t2.col1,
t1.col2 = t2.col2,
t1.col3 = t2.col3
when not matched
then
insert
(t1.id, t1.col1, t1.col2, t1.col3)
values
(t2.id, t2.col1, t2.col2, t2.col3);
It should be noted that t2 can be replaced with a select statement, in case your new data isn't already stored in a table in the correct format.
If this is not what you are looking for, can you please clarify what you mean (an example of current data and desired output would be ideal).