I get this error when i was inserting some rows to database.
Code
public class InsertRowData {
public static void main(String[] args)throws ClassNotFoundException, SQLException{
Connection con = (Connection) DriverManager
.getConnection("jdbc:oracle:thin:#localhost:1521:ORCL" , "system" , "system");
Statement statement = con.createStatement();
String dmlinsert = "insert into student(111,manoj,rawatrulz09)";
int rowseffected = statement.executeUpdate(dmlinsert);
System.out.println("no of rows effected" + rowseffected);
}
}
After searching in google I found the solution to remove single quotes from the column but still getting the same error.
This is a bad approach to insert any data into the Table.
String dmlinsert="insert into student(111,manoj,rawatrulz09)"
first of all, your query missing a syntax Values.
when columns are not mentioned while inserting, values must contain same amount of data and respectively.
But when inserting a data the best practice is to mention columns, so that if in later time you make any change to the table in database, your insert query stays stable.
for e.g:
insert into student(id, name, email) values(111,'manoj','rawatrulz09').
Now, if in future you will add a column to this table, your insert query will still work as it has mentioned columns, but if you won't mention any columns then your code will start giving error when executed no of values don't match given to the table etc... Good luck!
anyway, answer for your insert query:
insert into student values(111,'manoj','rawatrulz09')
I'm trying to find the faster way to do batch insert.
I tried to insert several batches with jdbcTemplate.update(String sql), where
sql was builded by StringBuilder and looks like:
INSERT INTO TABLE(x, y, i) VALUES(1,2,3), (1,2,3), ... , (1,2,3)
Batch size was exactly 1000. I inserted nearly 100 batches.
I checked the time using StopWatch and found out insert time:
min[38ms], avg[50ms], max[190ms] per batch
I was glad but I wanted to make my code better.
After that, I tried to use jdbcTemplate.batchUpdate in way like:
jdbcTemplate.batchUpdate(sql, new BatchPreparedStatementSetter() {
#Override
public void setValues(PreparedStatement ps, int i) throws SQLException {
// ...
}
#Override
public int getBatchSize() {
return 1000;
}
});
where sql was look like
INSERT INTO TABLE(x, y, i) VALUES(1,2,3);
and I was disappointed! jdbcTemplate executed every single insert of 1000 lines batch in separated way. I loked at mysql_log and found there a thousand inserts.
I checked the time using StopWatch and found out insert time:
min[900ms], avg[1100ms], max[2000ms] per Batch
So, can anybody explain to me, why jdbcTemplate doing separated inserts in this method? Why method's name is batchUpdate?
Or may be I am using this method in wrong way?
These parameters in the JDBC connection URL can make a big difference in the speed of batched statements --- in my experience, they speed things up:
?useServerPrepStmts=false&rewriteBatchedStatements=true
See: JDBC batch insert performance
I found a major improvement setting the argTypes array in the call.
In my case, with Spring 4.1.4 and Oracle 12c, for insertion of 5000 rows with 35 fields:
jdbcTemplate.batchUpdate(insert, parameters); // Take 7 seconds
jdbcTemplate.batchUpdate(insert, parameters, argTypes); // Take 0.08 seconds!!!
The argTypes param is an int array where you set each field in this way:
int[] argTypes = new int[35];
argTypes[0] = Types.VARCHAR;
argTypes[1] = Types.VARCHAR;
argTypes[2] = Types.VARCHAR;
argTypes[3] = Types.DECIMAL;
argTypes[4] = Types.TIMESTAMP;
.....
I debugged org\springframework\jdbc\core\JdbcTemplate.java and found that most of the time was consumed trying to know the nature of each field, and this was made for each record.
Hope this helps !
I have also faced the same issue with Spring JDBC template. Probably with Spring Batch the statement was executed and committed on every insert or on chunks, that slowed things down.
I have replaced the jdbcTemplate.batchUpdate() code with original JDBC batch insertion code and found the Major performance improvement.
DataSource ds = jdbcTemplate.getDataSource();
Connection connection = ds.getConnection();
connection.setAutoCommit(false);
String sql = "insert into employee (name, city, phone) values (?, ?, ?)";
PreparedStatement ps = connection.prepareStatement(sql);
final int batchSize = 1000;
int count = 0;
for (Employee employee: employees) {
ps.setString(1, employee.getName());
ps.setString(2, employee.getCity());
ps.setString(3, employee.getPhone());
ps.addBatch();
++count;
if(count % batchSize == 0 || count == employees.size()) {
ps.executeBatch();
ps.clearBatch();
}
}
connection.commit();
ps.close();
Check this link as well
JDBC batch insert performance
Simply use transaction. Add #Transactional on method.
Be sure to declare the correct TX manager if using several datasources #Transactional("dsTxManager"). I have a case where inserting 60000 records. It takes about 15s. No other tweak:
#Transactional("myDataSourceTxManager")
public void save(...) {
...
jdbcTemplate.batchUpdate(query, new BatchPreparedStatementSetter() {
#Override
public void setValues(PreparedStatement ps, int i) throws SQLException {
...
}
#Override
public int getBatchSize() {
if(data == null){
return 0;
}
return data.size();
}
});
}
Change your sql insert to INSERT INTO TABLE(x, y, i) VALUES(1,2,3). The framework creates a loop for you.
For example:
public void insertBatch(final List<Customer> customers){
String sql = "INSERT INTO CUSTOMER " +
"(CUST_ID, NAME, AGE) VALUES (?, ?, ?)";
getJdbcTemplate().batchUpdate(sql, new BatchPreparedStatementSetter() {
#Override
public void setValues(PreparedStatement ps, int i) throws SQLException {
Customer customer = customers.get(i);
ps.setLong(1, customer.getCustId());
ps.setString(2, customer.getName());
ps.setInt(3, customer.getAge() );
}
#Override
public int getBatchSize() {
return customers.size();
}
});
}
IF you have something like this. Spring will do something like:
for(int i = 0; i < getBatchSize(); i++){
execute the prepared statement with the parameters for the current iteration
}
The framework first creates PreparedStatement from the query (the sql variable) then the setValues method is called and the statement is executed. that is repeated as much times as you specify in the getBatchSize() method. So the right way to write the insert statement is with only one values clause.
You can take a look at http://docs.spring.io/spring/docs/3.0.x/reference/jdbc.html
I had also some bad time with Spring JDBC batch template. In my case, it would be, like, insane to use pure JDBC, so instead I used NamedParameterJdbcTemplate. This was a must have in my project. But it was way slow to insert hundreds os thousands of lines in the database.
To see what was going on, I've sampled it with VisualVM during the batch update and, voilà:
What was slowing the process was that, while setting the parameters, Spring JDBC was querying the database to know the metadata each parameter. And seemed to me that it was querying the database for each parameter for each line every time. So I just taught Spring to ignore the parameter types (as it is warned in the Spring documentation about batch operating a list of objects):
#Bean(name = "named-jdbc-tenant")
public synchronized NamedParameterJdbcTemplate getNamedJdbcTemplate(#Autowired TenantRoutingDataSource tenantDataSource) {
System.setProperty("spring.jdbc.getParameterType.ignore", "true");
return new NamedParameterJdbcTemplate(tenantDataSource);
}
Note: the system property must be set before creating the JDBC Template object. It would be possible to just set in the application.properties, but this solved and I've never after touched this again
I don't know if this will work for you, but here's a Spring-free way that I ended up using. It was significantly faster than the various Spring methods I tried. I even tried using the JDBC template batch update method the other answer describes, but even that was slower than I wanted. I'm not sure what the deal was and the Internets didn't have many answers either. I suspected it had to do with how commits were being handled.
This approach is just straight JDBC using the java.sql packages and PreparedStatement's batch interface. This was the fastest way that I could get 24M records into a MySQL DB.
I more or less just built up collections of "record" objects and then called the below code in a method that batch inserted all the records. The loop that built the collections was responsible for managing the batch size.
I was trying to insert 24M records into a MySQL DB and it was going ~200 records per second using Spring batch. When I switched to this method, it went up to ~2500 records per second. so my 24M record load went from a theoretical 1.5 days to about 2.5 hours.
First create a connection...
Connection conn = null;
try{
Class.forName("com.mysql.jdbc.Driver");
conn = DriverManager.getConnection(connectionUrl, username, password);
}catch(SQLException e){}catch(ClassNotFoundException e){}
Then create a prepared statement and load it with batches of values for insert, and then execute as a single batch insert...
PreparedStatement ps = null;
try{
conn.setAutoCommit(false);
ps = conn.prepareStatement(sql); // INSERT INTO TABLE(x, y, i) VALUES(1,2,3)
for(MyRecord record : records){
try{
ps.setString(1, record.getX());
ps.setString(2, record.getY());
ps.setString(3, record.getI());
ps.addBatch();
} catch (Exception e){
ps.clearParameters();
logger.warn("Skipping record...", e);
}
}
ps.executeBatch();
conn.commit();
} catch (SQLException e){
} finally {
if(null != ps){
try {ps.close();} catch (SQLException e){}
}
}
Obviously I've removed error handling and the query and Record object is notional and whatnot.
Edit:
Since your original question was comparing the insert into foobar values (?,?,?), (?,?,?)...(?,?,?) method to Spring batch, here's a more direct response to that:
It looks like your original method is likely the fastest way to do bulk data loads into MySQL without using something like the "LOAD DATA INFILE" approach. A quote from the MysQL docs (http://dev.mysql.com/doc/refman/5.0/en/insert-speed.html):
If you are inserting many rows from the same client at the same time,
use INSERT statements with multiple VALUES lists to insert several
rows at a time. This is considerably faster (many times faster in some
cases) than using separate single-row INSERT statements.
You could modify the Spring JDBC Template batchUpdate method to do an insert with multiple VALUES specified per 'setValues' call, but you'd have to manually keep track of the index values as you iterate over the set of things being inserted. And you'd run into a nasty edge case at the end when the total number of things being inserted isn't a multiple of the number of VALUES lists you have in your prepared statement.
If you use the approach I outline, you could do the same thing (use a prepared statement with multiple VALUES lists) and then when you get to that edge case at the end, it's a little easier to deal with because you can build and execute one last statement with exactly the right number of VALUES lists. It's a bit hacky, but most optimized things are.
Solution given by #Rakesh worked for me.
Significant improvement in performance. Earlier time was 8 min, with this solution taking less than 2 min.
DataSource ds = jdbcTemplate.getDataSource();
Connection connection = ds.getConnection();
connection.setAutoCommit(false);
String sql = "insert into employee (name, city, phone) values (?, ?, ?)";
PreparedStatement ps = connection.prepareStatement(sql);
final int batchSize = 1000;
int count = 0;
for (Employee employee: employees) {
ps.setString(1, employee.getName());
ps.setString(2, employee.getCity());
ps.setString(3, employee.getPhone());
ps.addBatch();
++count;
if(count % batchSize == 0 || count == employees.size()) {
ps.executeBatch();
ps.clearBatch();
}
}
connection.commit();
ps.close();
Encountered some serious performance issue with JdbcBatchItemWriter.write() (link) from Spring Batch and find out the write logic delegates to JdbcTemplate.batchUpdate() eventually.
Adding a Java system properties of spring.jdbc.getParameterType.ignore=true fixed the performance issue entirely ( from 200 records per second to ~ 5000 ).
The patch was tested working on both Postgresql and MsSql (might not be dialect specific)
... and ironically, Spring documented this behaviour under a "note" section link
In such a scenario, with automatic setting of values on an underlying PreparedStatement, the corresponding JDBC type for each value needs to be derived from the given Java type. While this usually works well, there is a potential for issues (for example, with Map-contained null values). Spring, by default, calls ParameterMetaData.getParameterType in such a case, which can be expensive with your JDBC driver. You should use a recent driver version and consider setting the spring.jdbc.getParameterType.ignore property to true (as a JVM system property or in a spring.properties file in the root of your classpath) if you encounter a performance issue — for example, as reported on Oracle 12c (SPR-16139).
Alternatively, you might consider specifying the corresponding JDBC
types explicitly, either through a 'BatchPreparedStatementSetter' (as
shown earlier), through an explicit type array given to a
'List<Object[]>' based call, through 'registerSqlType' calls on a
custom 'MapSqlParameterSource' instance, or through a
'BeanPropertySqlParameterSource' that derives the SQL type from the
Java-declared property type even for a null value.
So, a solution I created threw this exception: jdbc.SQLServerException: The result set has no current row on the line marked in the below code.
public String get64BitEncodedImageBySiteID(int siteID){
try {
Class.forName("com.microsoft.sqlserver.jdbc.SQLServerDriver");
Connection conn = DriverManager.getConnection(url, userName, password);
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery( "SELECT SitePicture FROM SiteTable WHERE SiteID ="+siteID );
rs.next();
// The above line has since been moved to the if statement below where you can see it commented out,
// which prevents the exception from occuring but still doesn't fix the fact that the row is not being found.
if(/*rs.next() &&*/ rs.getBytes("SitePicture")!=null){ // EXCEPTION THROWN HERE!
byte ba[] = rs.getBytes("SitePicture");
return new sun.misc.BASE64Encoder().encodeBuffer(ba);
}
else {return null;}
} catch (Exception ex) {
ex.printStackTrace();
}
return null;
}
The method above, in the instance the exception was thrown, is taking a genuine siteID (22379) from an Entity object pulled directly from the same table. When using System.out.println(siteID); during this method, it declared that number to still be correct, ie still 22379. I've checked directly with the SQL server by running an identical statement in SQL Server, so I know the row exists in the table, but for some reason it is not being found. Image below.
So the problem is, the ResultsSet rs is not finding the row even though I know that it's there. Does anyone have any helpful insights?
Clarification: Just to be clear, I know that the ResultsSet contains no rows and that is why I am getting the exception. I also know that putting the rs.next() into the if statement will prevent the exception (as already stated in the comments). What is puzzling me is that the fact the ResultsSet contains no rows even though a row with the ID being parsed to it verifiably does exists because I have checked it directly with the SQL server.
This turned out to be a local mistake, but I'll post the solution anyway because this situation has some educational value.
As I've learned from #Ralph's comment to this answer, eliminating "the impossible" is a good way for such problems.
After avoiding the risk of siteID being wrong (by hardcoding it), we have a following situation:
the same exact query worked in one environment, but not the other, for only one particular SiteID, 2184
it's impossible that ResultSet just doesn't work for this particular value (I claim it is, because I always assume errors are in my code, not in language libraries)
if so, the databases must differ
Adding result statements inside while loop helped in my case.
while(rs.next) {
rs.getString("your column name");
}
The most likely explanation is that your ResultSet contains no rows. Have you checked that?
If that's the case, rs.next() will return false, but you are not checking the return value any more. Put rs.next() back into the if block, it was OK in there.
You can make sure by:
if (rs.next()) {
if(rs.getBytes("SitePicture")!=null){
byte ba[] = rs.getBytes("SitePicture");
return new sun.misc.BASE64Encoder().encodeBuffer(ba);
}
} else {
System.out.println("No rows returned");
}
EDIT:
what column type is siteID?
Your method takes an int, but your SQL wraps it in quotes, as if it were a string.
EDIT 2:
Using a PreparedStatement might solve your problem.
PreparedStatement ps = conn.prepareStatement("SELECT SitePicture FROM SiteTable WHERE SiteID = ?");
ps.setInt(1, siteId);
ResultSet rs = ps.executeQuery();
You may not get the result without checking whether the result has entries. For that use
while(rs.next()){
rs.getString("column name");
}
and try it. It worked fine for me.
Thanks
I have a stored procedure, I want to call it from JDBC, I got null pointer exception in the line"
while (restuls.next()) {
My code is:
Connection con = Database.getConnection();
CallableStatement callableStatement = null;
try {
String storedProcedure = "{call getAllCustomerAddresses(?,?,?,?,?,?,?)}";
callableStatement = con.prepareCall(storedProcedure);
callableStatement.setInt(1, this.getID());
callableStatement.registerOutParameter(2,
java.sql.Types.INTEGER);
callableStatement.registerOutParameter(3,
java.sql.Types.VARCHAR);
callableStatement.registerOutParameter(4,
java.sql.Types.INTEGER);
callableStatement.registerOutParameter(5,
java.sql.Types.INTEGER);
callableStatement.registerOutParameter(6,
java.sql.Types.INTEGER);
callableStatement.registerOutParameter(7,
java.sql.Types.VARCHAR);
callableStatement.execute();
System.out.println(callableStatement.getInt(2));
System.out.println(callableStatement.getString(3));
System.out.println(callableStatement.getInt(4));
System.out.println(callableStatement.getInt(5));
System.out.println(callableStatement.getInt(6));
System.out.println(callableStatement.getString(7));
ResultSet restuls = callableStatement.getResultSet();
while (restuls.next()) {
int addressID = restuls.getInt(2);
String label = restuls.getString(3);
int regionID = restuls.getInt(4);
int areaID = restuls.getInt(5);
int cityID = restuls.getInt(6);
String description = restuls.getString(7);
this.addresses.add(new CustomerAddressImpl(this, label,
description, RegionImpl.getInstance(regionID),
AreaImpl.getInstance(areaID), CityImpl
.getInstance(cityID), addressID));
}
look at the code, the System.out.println is working , and It is printing the right values from database, so why the results set is null please??
another thing, I must use result set because the stored procedure returns many rows.
I am really confusing why I can print the right values but the result set is null
Thanks in advance
Edit
If you want to give you the stored procedure tell me please
Stored Procedure
ALTER PROCEDURE [dbo].getAllCustomerAddresses(
#customerID INT,
#addressID INT OUTPUT,
#label VARCHAR(200) OUTPUT,
#regionID INT OUTPUT,
#areaID INT OUTPUT,
#cityID INT OUTPUT,
#description TEXT OUTPUT
)
AS
SET NOCOUNT Off;
SELECT #addressID = [ID],
#label = [label],
#regionID = [regionID],
#areaID = [areaID],
#cityID = [cityID],
#description = [description]
FROM Customer_Address
WHERE customerID = #customerID
execute() method of PreparedStatement returns true if result set presents and false otherwise. You do not check the return value of execute(). I think that if you do that you see that it is false.
The reason should be in your stored procedure that IMHO does not return value. So, try to analyze it to understand the problem.
Here are recommendations I can give you:
Use executeQuery() that directly returns ResaultSet instead of execute(). I think this is more convenient.
Avoid using stored procedures that couple your platform independent java code with specific type of database. Try to write all logic in java and use portable SQL statements only.
The last time I saw pure JDBC code was about 10 years ago. There are a lot of tools that help you to avoid writing SQL inside java code. Take a look on JPA, Hibernate, iBatis etc.
Your stored procedure doesn't actually produce a ResultSet because you are using output parameters (not 100% sure, I don't have a SQL Server handy to test).
You may just need to call CallableStatement.getObject(int) or CallableStatement.getObject(String) (or a type specific getter) to get the values instead. If you want to process as a ResultSet, then you should not use the output parameters in your stored procedures, but write the stored procedure as a select without assigning to output parameter. That will create a result set from the stored procedure
Another possibility might by that your stored procedure is first returning one or more update counts before returning the result set. The boolean return value of execute() indicates whether the first result is an update count or a ResultSet. You will need to repeatedly call getMoreResults() and getUpdateCount() to be sure you have processed every result.
Your posted stored procedure contains SET NOCOUNT OFF which signals to SQL Server (or Sybase) that you want update (and I believe select) counts returned as well, you might want to try using SET NOCOUNT ON.
You can also try to process the results of execute() like this to find out if there are indeed multiple update counts etc before the result set:
boolean result = pstmt.execute();
while(true)
if (result) {
ResultSet rs = pstmt.getResultSet();
// Do something with resultset ...
} else {
int updateCount = pstmt.getUpdateCount();
if (updateCount == -1) {
// no more results
break;
}
// Do something with update count ...
}
result = pstmt.getMoreResults();
}
See also Java SQL: Statement.hasResultSet()?
I'm struggling with a homework assignment and am getting hung up on some SQL queries.
My query is interrogating an inventory database for the quantity of some item. The query requests the column with the name quantity_in_stock from the table, given the primary key.
I have initialized some prepared statements. This is the one I'm using here:
stmtFindColumn = Database.getConnection().prepareStatement(String.format("select ? from %s where %s = ?",
INVENTORY_TABLE_NAME, SKU) );
Now a separate method is called. I pass it a static const QTY_IN_STOCK, which is defined as "quantity_in_stock" and the item's SKU number, which is the primary key in the table.
private int getIntegerFromTable(String column, String key) {
int toReturn = 0;
try {
// Complete the prepared statement
stmtFindColumn.setString(1, column);
stmtFindColumn.setString(2, key);
ResultSet result = stmtFindColumn.executeQuery();
toReturn = result.getInt(column);
} catch (SQLException e) {
LOG.error(e.getMessage());
e.printStackTrace();
}
return toReturn;
}
When I run the query I get an sql exception that tells me: Invalid column name quantity_in_stock.
I have tried using a while loop processing result.next() and get the same error. I can't find any examples of how to properly get the results when you know only a single record is being returned.
Help!
EDIT: OK, I've found that part of my problem is I'm not getting a result set, where I should expect one. Any ideas?
UPDATE: I've tested my code, using a garden variety statement and a plain string query instead and it works just fine. So the problem is in my use of the prepared statement. Can someone check if I'm using the ? wildcards correctly? Thanks!
as far as i know, the column name may not be a parameter ...
DarkSquirrel42 is right -- you can't replace the column list of the select using a ? parameter marker. Instead, you can String.format that into place too, for example.
bad:
*select ? from INVENTORY_TABLE_NAME where SKU = ?
good:
select QUANTITY_IN_STOCK from INVENTORY_TABLE_NAME where SKU = ?