I'm working with latitude and longitude as index and compiled the r*tree module into the sqlite3 database to increase the performance. Further I loaded the tables into an in-memory database. The surprising result was that the ResultSet next method slows down to 25-30 ms instead of 1 ms if the data are coming from hard disk. Normally I expect 250 entry for the result set.
First a connection is set up to the memory.
Class.forName("org.sqlite.JDBC");
Connection connection = DriverManager.getConnection("jdbc:sqlite::memory:");
Then the tables are copied from the hard disk into the memory:
Statement s = connection.createStatement();
s.execute("ATTACH 'myDB.db' AS fs");
s.executeUpdate("CREATE VIRTUAL TABLE coordinates USING rtree(id, min_longitude, max_longitude, min_latitude, max_latitude)");
s.executeUpdate("INSERT INTO coordinates (id, min_longitude, max_longitude, min_latitude, max_latitude)
SELECT id, min_longitude, max_longitude, min_latitude, max_latitude FROM fs.coordinates");
s.executeUpdate("CREATE TABLE locations AS SELECT * from fs.locations");
s.execute("DETACH DATABASE fs");
The last step is query the database and copy the result into an object.
final String sql = "SELECT * FROM locations, coordinates WHERE (locations.id = coordinates.id) AND ((min_latitude >= ? AND max_latitude <= ?) AND (min_longitude >= ? AND max_longitude <= ?))";
PreparedStatement ps = connection.prepareStatement(sql);
// calculate bounding rec and fullfil sql statement.
Result rs = ps.executeQuery();
while (rs.next()) {
// copy the stuff here
}
I tried some stuff like the connection should be "read only" or using TYPE_FORWARD_ONLY and CONCUR_READ_ONLY or increasing the fetch size to perform the result set but rs.next() stays unimpressed.
Does anybody has an idea what's happening in the memory and why rs.next() is so slow? How could I increase the query?
The system uses the driver sqlite-jdbc 3.7.2 from org.xerial and tomcat6.
Update: Added the database schema
CREATE VIRTUAL TABLE coordinates USING rtree(
id,
min_latitude,
max_latitude,
min_longitude,
max_longitude);
CREATE TABLE locations(
id INTEGER UNIQUE NOT NULL,
number_of_locations INTEGER DEFAULT 1,
city_en TEXT DEFAULT NULL,
zip TEXT DEFAULT NULL,
street TEXT DEFAULT NULL,
number TEXT DEFAULT NULL,
loc_email TEXT DEFAULT NULL,
loc_phone TEXT DEFAULT NULL,
loc_fax TEXT DEFAULT NULL,
loc_url TEXT DEFAULT NULL);
In the original database, the query first looks up coordinates with the R-tree index, and then looks up matching locations with the index (implied by UNIQUE) on the id column, as shown by this EXPLAIN QUERY PLAN output:
0|0|1|SCAN TABLE coordinates VIRTUAL TABLE INDEX 2:DaBbDcBd
0|1|0|SEARCH TABLE locations USING INDEX sqlite_autoindex_locations_1 (id=?)
In the in-memory database, the table schema has changed because:
A table created using CREATE TABLE AS has no PRIMARY KEY and no constraints of any kind.
This particular table now looks as follows:
CREATE TABLE locations(
id INT,
number_of_locations INT,
city_en TEXT,
zip TEXT,
street TEXT,
number TEXT,
loc_email TEXT,
loc_phone TEXT,
loc_fax TEXT,
loc_url TEXT
);
It is no longer possible to look up locations by ID efficiently, so the database does full table scans:
0|0|0|SCAN TABLE locations
0|1|1|SCAN TABLE coordinates VIRTUAL TABLE INDEX 1:
When copying databases, you should always use all the original CREATE TABLE/INDEX statements.
Related
Is there a way to enter recurring data into a set database fields while using the prepared statement (pst)? I’m using the Derby DB.
I have an Excel spreadsheet that I get data from. Each line has 5 entry fields: Line item description, cost, quantity, extended cost, and price. I want to enter that into the derby db using just the five entries but with multiple lines each having five entries off the spreadsheet. The database has the following fields:
MATLINEITEM varchar 200
MATLINEITEMCOST varchar 7
MATLINEITEMQTY varchar 7
MATLINEITEMXCOST varchar 7
MATLINEITEMPRICE varchar 7
Using the standard INSERT with “ ?” approach, I would need to have an entry for each line and each spreadsheet field. If I have 10 lines, then I would need 50 “prepared statement” lines. I want to just use a loop to read each line and the five entries assigned to that line and enter it into the database.
The data from the spreadsheet is first converted to String (using POI) and assigned to a TextField.
addSQL = "INSERT INTO MATERIALTBL (MATLINEITEM, MATLINEITEMCOST,
MATLINEITEMQTY, MATLINEITEMXCOST, MATLINEITEMPRICE) VALUES (?,?,?,?,?) ";
pst = conn.prepareStatement(addSQL);
pst.setString(1,jtxtfld_Item.getText());
pst.setString(2,jtxtfld_Cost.getText());
pst.setString(3,jtxtfld_Qty.getText());
pst.setString(4,jtxtfld_XCost.getText());
pst.setString(5,jtxtfld_Price.getText());
pst.executeUpdate();
Any help would be appreciated. Or point me in the right direction as to what I can be using to get this result. Thank you.
If you want to insert multiple INSERT statements, it is possible.
use addBatch() and executeBatch() methods of PrepearedStamement.
You create a PrepearedStatement, and add batches to it, until it is ready to be inserted into the database.
Example
String addSQL = "INSERT INTO MATERIALTBL (MATLINEITEM, MATLINEITEMCOST,
MATLINEITEMQTY, MATLINEITEMXCOST, MATLINEITEMPRICE) VALUES (?,?,?,?,?) ";
PreparedStatemen pst = conn.prepareStatement(addSQL);
//Set auto-commit to false
conn.setAutoCommit(false);
for (int i = 0; i < amountOfIterations; i++){
// Set the variables
pst.setString(1,jtxtfld_Item.getText());
pst.setString(2,jtxtfld_Cost.getText());
pst.setString(3,jtxtfld_Qty.getText());
pst.setString(4,jtxtfld_XCost.getText());
pst.setString(5,jtxtfld_Price.getText());
// Add it to the batch
pst.addBatch();
}
//Create an int[] to hold returned values
int[] count = stmt.executeBatch();
//Explicitly commit statements to apply changes
conn.commit();
Why Use Batches?
This PrepearedStatement will be sent only once to the database, and will insert all your rows at once, saving you unnecessary calls to the database.
We have multiple tables and all are related with first table's primary key (example: id). Id is configured as a sequence and while inserting data into to first table we are using sequence.nextval in the insert query.
Now while inserting data to other tables, how to get current sequence value or current Id.
We have tried below options:
sequence.currval, directly in the insert statement
2.select sequence.currval from dual
Above two options throwing error while using getJdbcTemplate().update().
Could anyone please suggest how to get current sequence value to pass to other tables after inserting data into first table??
If you want to insert the same id (which comes from a sequence) to different tables, simple get it form the first insert and use it in the other inserts.
PrepearedStatement stmt1 = conn.prepareStatement("INSERT INTO TABLE1 (id) VALUES(yoursequence.nextval)", Statemet.RETURN_GENERATED_KEYS);
stmt1.executeUpdate();
ResultSet rs = stmt1.getGeneratedKeys();
rs.next();
long id = rs.getLong(1);
PrepearedStatement stmt2 = conn.prepareStatement("INSERT INTO TABLE2 (id) VALUES(?)");
stmt2.setLong(1,id);
stmt2.executeUpdate();
I have a (sql server) column which have a System Type but no Data Type (It is actually more then one). Using the Microsoft jdbc4 drivers and trying to list all the columns using metadata for this table this column with the missing Data type doesn't show up:
ResultSet rs = connection.getMetaData().getColumns(null,"schema","tableName", null);
while (rs.next()) {
String column = rs.getString("COLUMN_NAME");
System.out.println("column:" + column); // This will not print the column
}
If you use the odbc drivers, in for example C#, it does show up (The column with the missing Data Type). And if this column is a constraint it will show up when using the the jdbc api for listing all constraints:
ResultSet rs = connection.getMetaData().getIndexInfo(null,
"schema",
"tableName",
true/*list unique*/,
true);
while(rs.next()){
String column = rs.getString("COLUMN_NAME");
System.out.println("column:" + column); // This will print the column, if it is a unique constraint.
}
This behavior is the same regardless if you use jdbc4 drivers or the jtds drivers.
So my question is, if this is a bug or is it something that I'm missing? And Is it possible to list the meta data in an other way to get all the columns for table?
To create a Table which doesn't display a Data Type. This demands that you create a user which haven't read access to the user defined Data Types. This is how can reproduce it:
I found this code here
CREATE TYPE TEST_TYPE2 FROM [int] NOT NULL
GO
CREATE TABLE Customer
(
CustomerID INT IDENTITY(1,1) NOT NULL,
LastName VARCHAR(100) NOT NULL,
FirstName VARCHAR(100) NOT NULL,
ZIP TEST_TYPE2 NOT NULL
)
GO
-- Create Table without UDDT
CREATE TABLE Customer_Orig
(
CustomerID INT IDENTITY(1,1) NOT NULL,
LastName VARCHAR(100) NOT NULL,
FirstName VARCHAR(100) NOT NULL,
ZIP INT NOT NULL
)
GO
-- Create User with db_datareader to database
USE [master]
GO
CREATE LOGIN testUser WITH PASSWORD=N'THE_PASSWORD',
DEFAULT_DATABASE=[master],
CHECK_EXPIRATION=OFF, CHECK_POLICY=OFF
GO
USE TEST_DATABASE
GO
CREATE USER testUser FOR LOGIN testUser
GO
USE TEST_DATABASE
GO
EXEC sp_addrolemember N'db_datareader', N'THE_PASSWORD'
GO
I understand why it doesn't show up, but I way does it work for the odbc drivers?
I have set up 2 two tables - table userid and table data in phpmyadmin. The userid table consists of a single column - id and the table data consists of the following columns- id|name|price. I have added an index in the column id of table userid to point to id of the table data. Now i have a user who makes certain selections inside an android application. I want to insert this data into the table data. I know i have to use jdbc and know how to enter data for table without any index. But i am confused as to how to go about doing it in the case of related tables. The userid is obtained from this link http://android-developers.blogspot.in/2011/03/identifying-app-installations.html. Would someone please tell me how to enter the data using java. (The confusion is how to enter the userid and the corresponding data ).
As long as you're aware of the user id when you're inserting a new record into the data table that's all you need. Basically you'll have:
Statement stmt = conn.createStatement();
stmt.executeUpdate( "INSERT INTO data (id, name, price) VALUES ('id from userid table', 'a name', 'a price')");
Obviously the params aren't escaped properly (they really should be) and there's no testing for errors, but that would get you started.
Then to, for example, select all the data related to a given userid, you would do something like:
Statement stmt = conn.createStatement();
stmt.executeQuery( "SELECT * FROM userid LEFT JOIN data WHERE userid.id = data.id" );
Hi I am trying to insert values from excel sheet into SQL Database in java. SQL database has already some rows inserted by some other techniques. Now I need to insert new rows from excel sheet and should eliminate the duplicate values which are existed in the database as well as in the excel sheet. For that I write a query like this.
First I inserted the records from excelsheet into SQL database by using insert query
Statement.executeUpdate(("INSERT INTO dbo.Company(CName,DateTimeCreated) values
('"+Cname”' ,'"+ts+"');
Later I deleted the duplicate values using delete query.
String comprows="delete from dbo.Company where Id not in"
+ "(select min(Id) from dbo.Company "
+ "group by CName having count(*)>=1)";
statement3.executeUpdate(comprows);
where Id is autoincremented integer.
but it is not good to do insert and then delete.
How do I know the values are already exist? If it is exist how do I remove during insertion???
You can simply fire a SELECT for the CName first. If a record is found, update else insert a new record.
Edited to add code snippet:
ResultSet rs = Statement.query("SELECT Id from dbo.Company where CNAME = '" +Cname + "'");
if(rs.next()) {
// retrieve ID from rs
// fire an update for this ID
} else {
// insert a new record.
}
Alternatively, if you think that there are already duplicates on your table and you want to remove them as well..
ResultSet rs = Statement.query("SELECT Id from dbo.Company where CNAME = '"+Cname + "'");
List idList = new ArrayList();
while(rs.next()) {
// collect IDs from rs in a collection say idList
}
if(!isList.isempty()) {
//convert the list to a comma seperated string say idsStr
Statement.executeUpdate("DELETE FROM dbo.Company where id in ("+ idsStr + ")");
}
// insert a new record.
Statement.executeUpdate(("INSERT INTO dbo.Company(CName,DateTimeCreated) values('"+Cname”' ,'"+ts+"');
Of course good practice is to use PreparedStatement as it would improve performance.
PS: Excuse me for any syntax errors.
One option would be to create a temp table and dump your Excel data there. Then you can write an insert that joins the temp table with the dbo.Company table and only insert the records that aren't already there.
You could do a lookup on each record you want to insert but if you are dealing with large volumes that's not a super efficient way to do it since you will have to do a select and an insert for each record in you excel spreadsheet.
Merge statements are pretty effective in these types of situations as well. I don't think all databases support them (I know Oracle does for sure). A merge statement is basically a combo insert and update so you can do the look up to the final table and insert if not found and update if found. The nice thing about this is you get the efficiency of doing all of this as a set rather than one record at a time.
If you can control the DB schema, you might consider putting a unique contraint for whatever column(s) to avoid duplicating. When you do your inserts, it'll throw when it tries to add the dup data. Catch it before it tosses you all the way out.
It's usually good to enforce constraints like this on the DB itself; that means no one querying the database has to worry about invalid duplicates. Also, optimistically trying the insert first (without doing a separate select first) might be faster.