I am querying a table in Postgres to see if a row with primary key values match those in a given Map exists. If there are no results, I want to insert a new row. I can do this just fine when I go the standard JDBC route of creating a prepared statement and explicitly setting all the column values, but I want a more generic approach with the help of Groovy's Sql class.
This is my insert statement (the primary key columns are the first three):
insert into MY_TABLE
(contract_month, contract_year, publish_date, open, high, low)
values
(10, 2015, 2015-09-15 00:00:00.0, 46.65, 46.9, 45.98)
The issue is that I'm getting a NullPointerException from the Postgres driver as soon as I insert a row:
Exception in thread "main" java.lang.NullPointerException
at org.postgresql.core.v3.SimpleParameterList.setNull(SimpleParameterList.java:142)
at org.postgresql.jdbc2.AbstractJdbc2Statement.setNull(AbstractJdbc2Statement.java:1259)
at org.postgresql.jdbc3.AbstractJdbc3Statement.setNull(AbstractJdbc3Statement.java:1499)
at org.postgresql.jdbc4.AbstractJdbc4Statement.setNull(AbstractJdbc4Statement.java:85)
at org.postgresql.jdbc2.AbstractJdbc2Statement.setObject(AbstractJdbc2Statement.java:2092)
at org.postgresql.jdbc3g.AbstractJdbc3gStatement.setObject(AbstractJdbc3gStatement.java:38)
at org.postgresql.jdbc4.AbstractJdbc4Statement.setObject(AbstractJdbc4Statement.java:48)
Digging into it, I find that an object called ProtoConnection is null in the org.postgresql.core.v3.SimpleParameterList class. What am I doing wrong?
This is the wrapping code which creates the Sql object:
private Main() {
final def headers = new HashMap<Integer, String>()
final def record = new HashMap<String, Object>()
Sql.withInstance(/* my db */) { final Sql sql ->
sql.withBatch {
new URL(/* my url */).withReader {
it.readLines().each {
if (headers.isEmpty()) {
setHeaders it, headers
}
else {
processRecord it, headers, record
storeRecord sql, record
}
}
}
}
}
}
And this is the database logic. The error occurs at the very last line:
def storeRecord = { final Sql sql, final Map<String, Object> record ->
final def connection = sql.connection
final def metadata = connection.metaData
final def columns = metadata.getColumns(null, null, MY_TABLE, null).with {
final def result = []
while (it.next()) {
result << it.getString(4)
}
result
}
final def primaryKeys = metadata.getPrimaryKeys(null, null, MY_TABLE).with {
final def result = []
while (it.next()) {
result << it.getString(4)
}
result
}
final def whereClause = primaryKeys.collect { final String pk ->
"$pk = ?.$pk"
}
final def select = """
select ${columns.join(', ')}
from MY_TABLE
where ${whereClause.join(' and ')}
"""
final def row = sql.firstRow(select, record)
if (row) {
println "Updating"
// Not implemented yet
}
else {
println "Inserting record: $record"
final def insert = """
insert into MY_TABLE
(${columns.findAll { null != record[it] }.join(", ")})
values
(${columns.findAll { null != record[it] }.collect { record[it] }.join(", ")})
"""
println insert
sql.executeInsert(insert, record)
}
There were a few things I was doing wrong with my code. First of all, I wasn't converting the string objects in the record map into the appropriate JDBC types. This explains why I was able to insert records via the standard prepared statement route, as I originally did the conversion there, and overlooked it when transitioning to Groovy's API.
After fixing this, I then got an error that the hstore extension was not supported. This was simply solved by executing create extension hstore in Postgres.
Finally, I received an index out of bounds error. This was due to a silly bug in my code where I was setting the values in my insert statement, rather than letting the API fill in the parameters. You can see this fix in the final line of the insert statement now:
final def insert = """
insert into MY_TABLE
(${columns.join(', ')})
values
(${columns.collect { "?.$it" }.join(', ')})
"""
sql.executeInsert(insert, record)
Related
I am using Apache camel with sql component for performing sql operations with Postgresql. I have already tried successfully inserting multiple rows in a single table as a batch using batch=true option and providing the Iterator in the message body. To keep the example simple with Student as table name and having 2 columns name & age, below is the code snippet displaying the relevant part:
from("direct:batch_insert_single_table")
...
.process(ex -> {
log.info("batch insert for single table");
final var iterator = IntStream.range(0, 5000).boxed()
.map(x -> {
final var query = new HashMap<String, Object>();
Integer counter = x.intValue();
String name = "abc_" + counter;
query.put("name", name);
query.put("age", counter);
return query;
}).iterator();
ex.getMessage().setBody(iterator);
})
.to("sqlComponent:INSERT INTO student (name, age) VALUES (:#name, :#age);?batch=true")
...
;
This overall takes 10 seconds for 5000 records.
However, when I use the same approach for inserting as a batch on multiple different tables, I get an error:
Here is the code that is not working:
from("direct:batch_insert_multiple_tables")
...
.process(ex -> {
log.info("batch insert for multiple tables");
final var iterator = IntStream.range(0, 3).boxed()
.map(x -> {
final var query = new HashMap<String, Object>();
Integer counter = x.intValue();
String name = "abc_" + counter;
query.put("table", "test" + counter);
query.put("name", "name");
query.put("age", counter);
return query;
}).iterator();
ex.getMessage().setBody(iterator);
})
.to("sqlComponent:INSERT INTO :#table (name,age) VALUES (:#name,:#age);?batch=true")
...
;
The tables test0, test1 & test2 are already existing.
The exception thrown is:
Failed delivery for (MessageId: A0D98C12BAD769F-0000000000000000 on ExchangeId: A0D98C12BAD769F-0000000000000000). Exhausted after delivery attempt: 1 caught: org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCallback; bad SQL grammar []; nested exception is org.postgresql.util.PSQLException: ERROR: syntax error at or near "$1"
Position: 13
Plz suggest if I am doing something wrong or my approach is simply not supported by Apache Camel.
NOTE: I am using the latest version apache camel & Postgre.
Regards,
GSN
You cannot use a parameter for a table name, column name nor any other identifier in PostgreSQL. You either have to use a dynamically generated SQL statement (that is, a statement you construct in your Java code; take special care of SQL injection) or two SQL statements.
Here I have added my code. Issue occurs in try block while I am trying fetch list of tables.
Database is MySql
Exception is : java.lang.IllegalArgumentException: node to traverse cannot be null!
public class DBOptimizationDAO extends HibernateDaoSupport {
private static final Log log = LogFactory.getLog(DBOptimizationDAO.class);
public void optimizeAdapter(String year)
{
List<com.ecw.adapterservice.beans.TransactionInbound> transactionInboundList = null;
StringBuilder queries = new StringBuilder();
try {
transactionInboundList = (List<com.ecw.adapterservice.beans.TransactionInbound>)super.getHibernateTemplate().find("from TransactionInbound where inboundTimestamp < '" + year+ "-01-01'order by 1 desc limit 2");
// Check if archive table exist or not
List<Object> inboundObj = getHibernateTemplate().find("SHOW TABLES LIKE transaction_outbound");
List<Object> outboundObj = getHibernateTemplate().find("SHOW TABLES LIKE 'transaction_outbound_archive'");
The HibernateTemplate::find expects a HQL query in the string parameter and you are passing a native statement. You can do native stuff (queries, statements, etc) using the Session object returned by HibernateTemplate::getSession. To pass a native select query you then have Session::createSQLQuery
BUT do you really want to rely on database specific code to do this? There is a more elegant way to do it by using DatabaseMetaData::getTables. See this answer. And you can get an instance of DatabaseMetaData from a callback method of your HibernateTemplate.
Try this:
Session session = getHibernateTemplate().getSessionFactory().getCurrentSession();
if(!session instaceof SessionImpl){
//handle this, maybe throw an exception
}
else {
Connection con = (SessionImpl)session.connection();
...
}
I'm using JOOQ with plain/raw SQL, so that means i'm not using any code generation or the fluid DSL thingy.
The following code works:
Connection connection = ...;
DSLContext context = DSL.using(connection, ...);
String sql = "select * from mytable t where (t.id = ?)";
String id = ...; //
Result<Record> result = context.fetch(sql, id);
Now let's say i have a query with multiple parameters like this:
String sql = "select * from mytable t where (t.id = ?) " +
"and (t.is_active = ?) and (t.total > ?)";
How do i use a named parameter with these types of queries?
I'm thinking something like :
String sql = "select * from mytable t where (t.id = :id) " +
"and (t.is_active = :is_active) and (t.total > :total)";
ResultQuery<Record> rq = context.resultQuery(sql);
rq.getParam("id").setValue(...);
rq.getParam("is_active").setValue(...);
rq.getParam("total").setValue(...);
Result<Record> result = rq.fetch();
But the above code doesn't work (for obvious reasons). Thanks in advance.
jOOQ currently doesn't support executing SQL with named parameters. You can use jOOQ to render named parameters if you're executing the query with another API, such as Spring JDBC. For more information, consider the manual:
http://www.jooq.org/doc/latest/manual/sql-building/bind-values/named-parameters
But the plain SQL templating API allows for re-using templates, e.g.
String sql = "select * "
+ "from mytable t "
+ "where t.id = {0} or (t.id != {0} and t.name = {1})";
ResultQuery<Record> q = ctx.resultQuery(sql, val(1), val("A"));
This way, you can at least re-use values several times.
Because Lukas said that this feature is not available, I thought I'll code a 'good enough' solution.
The idea is that the variable name like :name has to be replaced with the {0} at all places and the rest is done by JOOQ. I thought this is the easiest way of doing it. (Replacing variables with their proper form, like handling data types is definitely a lot of work.)
I merited some ideas from this other StackOverflow answer and then created this gist in Kotlin (it would have been too long in Java otherwise).
The current gist looks like this now:
import org.jooq.DSLContext
import org.jooq.Record
import org.jooq.ResultQuery
import org.jooq.impl.DSL
object SqlJooqBindVariableOrganizer {
data class Processed(
val statement: String,
val originalStatement: String,
val variables: List<Pair<String, Any>>,
) {
fun toResultQuery(context: DSLContext): ResultQuery<Record> {
return context.resultQuery(
statement,
*variables.map { DSL.`val`(it.second) }.toTypedArray(),
)
}
}
private fun extractBindVariableLocations(
statement: String,
): Map<String, List<IntRange>> {
// https://stackoverflow.com/a/20644736/4420543
// https://gist.github.com/ruseel/e10bd3fee3c2b165044317f5378c7446
// not sure about this regex, I haven't used colon inside string to test it out
return Regex("(?<!')(:[\\w]*)(?!')")
.findAll(statement)
.map { result ->
val variableName = result.value.substringAfter(":")
val range = result.range
variableName to range
}
.groupBy(
{ it.first },
{ it.second }
)
}
fun createStatement(
statement: String,
vararg variables: Pair<String, Any>,
): Processed {
return createStatement(statement, variables.toList())
}
fun createStatement(
statement: String,
variables: List<Pair<String, Any>>,
): Processed {
val locations = extractBindVariableLocations(statement)
val notProvidedKeys = locations.keys.subtract(variables.map { it.first })
if (notProvidedKeys.isNotEmpty()) {
throw RuntimeException("Some variables are not provided:\n"
+ notProvidedKeys.joinToString()
)
}
val relevantVariables = variables
// there may be more variables provided, so filter this
.filter { it.first in locations.keys }
// these locations should have the same order as the variables
// so it is important to know the proper order of the indices
val variableNameToIndex = relevantVariables
.mapIndexed { index, variable -> variable.first to index }
.associateBy({ it.first }, { it.second })
val variableNameReplacements = locations
.flatMap { (variableName, ranges) ->
ranges.map { range -> variableName to range }
}
// the replacements have to be done in a reversed order,
// as the replaced string is not equal length
.sortedByDescending { it.second.first }
// replace :name with {0}
val processedStatement = variableNameReplacements
.fold(statement) { statementSoFar, (variableName, range) ->
// index has to exist, we just checked it
val index = variableNameToIndex[variableName]!!
statementSoFar.replaceRange(range, "{$index}")
}
return Processed(
statement = processedStatement,
originalStatement = statement,
variables = relevantVariables,
)
}
}
I have a MySQL StoredProcedure(EmployeeAbsentReport Procedure)for Given Two Dateranges. which is Successfully running on MySql Commandprompt.
How can I run this Stored Procedure using Hibernate,I found one Example on How to call Stored Procedure Example,But in the Example program
he worked with a single table.
Procedure:
DELIMITER $$
DROP PROCEDURE IF EXISTS `AbsentReportproc`$$
CREATE DEFINER=`root`#`localhost` PROCEDURE `AbsentReportproc`(IN _fromdate DATETIME, IN _todate DATETIME)
BEGIN
CREATE TEMPORARY TABLE daterange25 (dte DATE);
SET #counter := 0;
WHILE (#counter < DATEDIFF(DATE(_todate), DATE(_fromdate))) DO
INSERT INTO daterange25 VALUES (DATE_ADD(_fromdate, INTERVAL #counter:=#counter + 1 DAY));
END WHILE;
SELECT tp.EMPCODE,tp.NAME,tp.DEPARTMENT, Group_Concat(d.dte order by d.dte SEPARATOR '\n')AbsentDate, COUNT(tp.EMPCODE) Totalnoofabsentdates
FROM Master tp
JOIN daterange25 d
LEFT JOIN Transactions tpt ON (tp.EMPCODE = tpt.empcode) AND DATE(S_DateTime) = d.dte
WHERE tpt.empcode IS NULL
GROUP BY tp.EMPCODE;
DROP TABLE daterange25;
END$$
DELIMITER ;
how to call a Stored Procedure using Hibernate with joins(Including a Temporary Table)and writing sql-query in XML mapping file?
Following the common framework method which i have used in one of my project to call the Stored Procedure of MySQL database :
#Transactional(propagation = Propagation.REQUIRED, readOnly = false)
public <T> List<T> executeSP(final String spName,
final String[] parameterNames, final String[] parameterValues,
final String[] resultColumnNames, final int offset, final int size,
final Class<T> returnType) {
validateParams(parameterNames, parameterValues);
return (List<T>) hibernateTemplate
.execute(new HibernateCallback<List<T>>() {
#SuppressWarnings("unchecked")
public List<T> doInHibernate(final Session session) {
SQLQuery query = session.createSQLQuery("call "
+ spName);
if (returnType != null) {
query.addEntity(returnType);
}
setQueryParams(parameterNames, parameterValues, query);
setResultColumnNames(resultColumnNames, query);
if (offset >= 0) {
query.setFirstResult(offset);
}
if (size > 0) {
query.setMaxResults(size);
}
return query.list();
}
});
}
Next, i would suggest that please make the Temporary Table join into the Stored Procedure only, instead of make it join with SP, if execute in SP then it is better from performance point of view also.
I am trying to extract ROWID or the primary key using Spring's NamedParameterJdbcTemplate and GeneratedKeyHolder.
I am trying to do something like this.
MapSqlParameterSource parameters = new MapSqlParameterSource()
.addValue("param1", value1)
.addValue("param2", value2);
KeyHolder keyHolder = new GeneratedKeyHolder();
namedParameterJdbcTemplate.update("INSERT INTO TABLE(ID, col1, col2)"
+ "VALUES(TABLE.TABLE_SEQ.NEXTVAL, :param1, :param2)",
parameters, keyHolder);
After executing above query when I try to do keyHolder.getKey().longValue() it is throwing below exception.
HTTP Status 500 - Request processing failed; nested exception is org.springframework.dao.DataRetrievalFailureException: The generated key is not of a supported numeric type. Unable to cast [oracle.sql.ROWID] to [java.lang.Number]
When I went through this http://docs.oracle.com/cd/B28359_01/java.111/b31224/datacc.htm I understand (i hope i did) that ojdbc is not mapping oracle RowId to java RowId.
Can any one suggest is there any way to extract the key? (Yes it can be done using PreparedStatement but it is making my code bit ugly to read and manipulate on some conditions). Your suggestions are much appreciated.
You have to use this
namedParameterJdbcTemplate.update("INSERT INTO TABLE(ID, col1, col2)"
+ "VALUES(TABLE.TABLE_SEQ.NEXTVAL, :param1, :param2)",
parameters, keyHolder, new String[]{"ID"});
Here is a fully working example: Assuming Database is Oracle and column name which store generated Id is "GENERATED_ID" ( Can be any name)
public Integer insertRecordReturnGeneratedId(final MyObject obj)
{
final String INSERT_QUERY = "INSERT INTO MY_TABLE VALUES(GENERATED_ID_SEQ.NEXTVAL, :param1, :param2)";
try
{
MapSqlParameterSource parameters = new MapSqlParameterSource().addValue( "param1", obj.getField1() ).addValue( "param2", obj.getField1() ) ;
final KeyHolder holder = new GeneratedKeyHolder();
this.namedParameterJdbcTemplate.update( INSERT_QUERY, parameters, holder, new String[] {"GENERATED_ID" } );
Number generatedId = holder.getKey();
// Note: USING holder.getKey("GENERATED_ID") IS ok TOO.
return generatedId.intValue();
}
catch( DataAccessException dataAccessException )
{
}
}