We Keep Coding

Apache Camel with SQL component for inserts with batch=true throws error with multiple tables

I am using Apache camel with sql component for performing sql operations with Postgresql. I have already tried successfully inserting multiple rows in a single table as a batch using batch=true option and providing the Iterator in the message body. To keep the example simple with Student as table name and having 2 columns name & age, below is the code snippet displaying the relevant part:
from("direct:batch_insert_single_table")
...
.process(ex -> {
log.info("batch insert for single table");
final var iterator = IntStream.range(0, 5000).boxed()
.map(x -> {
final var query = new HashMap<String, Object>();
Integer counter = x.intValue();
String name = "abc_" + counter;
query.put("name", name);
query.put("age", counter);
return query;
}).iterator();
ex.getMessage().setBody(iterator);
})
.to("sqlComponent:INSERT INTO student (name, age) VALUES (:#name, :#age);?batch=true")
...
;
This overall takes 10 seconds for 5000 records.
However, when I use the same approach for inserting as a batch on multiple different tables, I get an error:
Here is the code that is not working:
from("direct:batch_insert_multiple_tables")
...
.process(ex -> {
log.info("batch insert for multiple tables");
final var iterator = IntStream.range(0, 3).boxed()
.map(x -> {
final var query = new HashMap<String, Object>();
Integer counter = x.intValue();
String name = "abc_" + counter;
query.put("table", "test" + counter);
query.put("name", "name");
query.put("age", counter);
return query;
}).iterator();
ex.getMessage().setBody(iterator);
})
.to("sqlComponent:INSERT INTO :#table (name,age) VALUES (:#name,:#age);?batch=true")
...
;
The tables test0, test1 & test2 are already existing.
The exception thrown is:
Failed delivery for (MessageId: A0D98C12BAD769F-0000000000000000 on ExchangeId: A0D98C12BAD769F-0000000000000000). Exhausted after delivery attempt: 1 caught: org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCallback; bad SQL grammar []; nested exception is org.postgresql.util.PSQLException: ERROR: syntax error at or near "$1"
Position: 13
Plz suggest if I am doing something wrong or my approach is simply not supported by Apache Camel.
NOTE: I am using the latest version apache camel & Postgre.
Regards,
GSN

You cannot use a parameter for a table name, column name nor any other identifier in PostgreSQL. You either have to use a dynamically generated SQL statement (that is, a statement you construct in your Java code; take special care of SQL injection) or two SQL statements.

Translating HQL Query to executable SQL Query with list parameter

I am writing a util function to get the total record count based on any HQL that I get passed in without loading all data. The passed in HQL might be pretty complex with lots of selects, joins, where conditions, groupings and sortings. For that I want to wrap the query with a SELECT COUNT(*) FROM (<ORIGINAL_QUERY>) x. I found out, that this is not possible with HQL, because Hibernate does not allow subqueries in FROM elements.
Now, I am trying to translate this random HQL query which might have some named parameters (some of them might be simple parameters, some might be lists) to an executable SQL statement without inlining the parameter values. This seems to work with simple parameters, but does not work with list parameters.
The hqlString and the namedParameterMap comes from somewhere outside:
final String hqlString = "...";
final Map<String, Object> namedParametersMap = ...;
Code to translate HQL to SQL (and ParameterTranslations:
final QueryTranslatorFactory translatorFactory = new ASTQueryTranslatorFactory();
final SessionFactoryImplementor factory = getSessionFactory();
final QueryTranslator translator = translatorFactory.createQueryTranslator(hqlString, hqlString, Collections.EMPTY_MAP, factory, null);
translator.compile(Collections.EMPTY_MAP, false);
final String sqlString = translator.getSQLString();
final ParameterTranslations parameterTranslations = translator.getParameterTranslations());
Code to create a SQLQuery, set the Parameters and execute the Query:
SQLQuery sqlQuery = getCurrentSession().createSQLQuery(sqlString);
((Set<?>) parameterTranslations.getNamedParameterNames()).forEach(parameterName -> {
for (int position : parameterTranslations.getNamedParameterSqlLocations(parameterName)) {
sqlQuery.setParameter(position, namedParametersMap.get(parameterName));
}
});
final Long rowCount = ((BigInteger) query.uniqueResult()).longValueExact();
This query works an expected:
final String hqlString = "SELECT p.firstname, p.surname FROM Person p WHERE p.id = :param1";
final Map<String, Object> namedParametersMap = Maps.newHashMap(ImmutableMap.<String, Object>of("param1", Integer.valueOf(1));
This query does not work:
String hqlString = "SELECT p.firstname, p.surname FROM Person p WHERE p.id IN (:param1)";
final Map<String, Object> namedParametersMap = Maps.newHashMap(ImmutableMap.<String, Object>of("param1", Lists.newArrayList(Integer.valueOf(1)));
Exception:
org.hibernate.exception.SQLGrammarException: could not extract ResultSet at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:106) at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:79) at org.hibernate.loader.Loader.getResultSet(Loader.java:2313) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2096) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:2072) at org.hibernate.loader.Loader.doQuery(Loader.java:941) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:352) at org.hibernate.loader.Loader.doList(Loader.java:2813) at org.hibernate.loader.Loader.doList(Loader.java:2796) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2625) at org.hibernate.loader.Loader.list(Loader.java:2620) at org.hibernate.loader.custom.CustomLoader.list(CustomLoader.java:322) at org.hibernate.internal.SessionImpl.listCustomQuery(SessionImpl.java:1996) at org.hibernate.internal.AbstractSessionImpl.list(AbstractSessionImpl.java:322) at org.hibernate.internal.SQLQueryImpl.list(SQLQueryImpl.java:125) at org.hibernate.internal.AbstractQueryImpl.uniqueResult(AbstractQueryImpl.java:966) at HQLQueryUtils.getCount(HQLQueryUtils.java:350)
Caused by: org.postgresql.util.PSQLException: ERROR: operator does not exist: integer = bytea HINT: No operator matches the given name and argument type(s).
You might need to add explicit type casts. Position: 301 at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2433) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2178) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365) at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:155) at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:118) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:70) ... 49 more
Environment:
Java 8
Hibernate 5.1.8
Postgres-JDBC 42.2.2
PostgreSQL Server 10.5

In HQL you can use query parameter and set Collection with setParameterList method.
Query q = session.createQuery("SELECT entity FROM Entity entity WHERE name IN (:names)");
q.setParameterList("names", names);

You can use direct sql
String sql="your query"
Query query = sessionFactory.getCurrentSession().createSQLQuery(sql);
query.setParameter("paramterName", parameterValue);
List<Object[]> resultSet = query.list();
List<YouClass > data= new ArrayList<>();
for (Object[] row : resultSet) {
YouClass yourObject=new YouClass ();
yourObject.setDate((Date) row[0]);
yourObject.setAmount((BigDecimal) row[1]);
data.add(yourObject);
}
Suppose Yourclass has two element date and amount

JDBCTemplate optional parameters

I am using spring JDBCTemplate.
I have a scenario, where the parameters that need to be passed into my query function, are conditional/optional. For example, I have the following code:
List<RealTimeDTO> result = jdbcTemplate.query(sql, new Object[] {custId,
number, requestType, startDate, endDate}, new CCCRowMapper());
In the code, I passed in custId, number, requestType, etc. However, requestType is an optional parameter that may come back as null or empty so I don't want it to be passed into the Object[] if it is either null or empty.
What can I do to handle this type of situation?
I could introduce logic where I only pass in the parameters I want into the Object[], however, I was wondering if there is an already built in functionality that handles this instead of me reinventing the wheel.

One option is to use NamedParameterJdbcTemplate, so the parameter "list" (now a Map) doesn't need to be modified, only the SQL does:
List<RealTimeDTO> query(String name) {
NamedParameterJdbcTemplate jdbcTemplate = new NamedParameterJdbcTemplate(dataSource);
String sql = "SELECT foo, bar" +
" FROM FooBar" +
" WHERE name" + (name == null ? " IS NULL" : "= :name");
Map<String, Object> params = new HashMap<>();
params.put("name", name);
return jdbcTemplate.query(sql, params, new CCCRowMapper());
}
UPDATE
If you have many conditions that may need to be skipped, and all conditions might be eliminated, then use a StringJoiner to build the WHERE clause:
List<RealTimeDTO> query(String name, String phone, int age) {
NamedParameterJdbcTemplate jdbcTemplate = new NamedParameterJdbcTemplate(dataSource);
StringJoiner where = new StringJoiner(" AND ", " WHERE ", "").setEmptyValue("");
if (name != null)
where.add("name = :name");
if (phone != null)
where.add("phone = :phone");
if (age != 0)
where.add("age = :age");
String sql = "SELECT foo, bar" +
" FROM FooBar" +
where;
Map<String, Object> params = new HashMap<>();
params.put("name", name);
params.put("phone", phone);
params.put("age", age);
return jdbcTemplate.query(sql, params, new CCCRowMapper());
}

You can use a static SQL by checking the condition like ? IS NULL OR name = ?. But you have to pass the argument twice AND pass the argument type (at.sql.Types) twice.
String sql = "SELECT foo, bar" +
" FROM FooBar" +
" WHERE (? IS NULL OR name = ?) ";
jdbcTemplate.query(sql, new Object[]{name, name}, new int[]{Types.VARCHAR, Types.VARCHAR}, new CCCRowMapper());
IMO not really better than using conditions.

I'd like to add NamedParameter to #Snozzlebert' answer with the next example.
You have to specify the parameters' type if you want to pass null values otherwise you will get a NullPointerException.
#Autowired
NamedParameterJdbcTemplate namedJdbcTemplate;
MapSqlParameterSource parameters = new MapSqlParameterSource();
parameters.addValue("name", name, Types.VARCHAR);
parameters.addValue("phone", null, Types.VARCHAR);
parameters.addValue("age", null, Types.SMALLINT);
List<RealTimeDTO> list = namedJdbcTemplate.query(
"SELECT foo, bar\n" +
" FROM FooBar\n" +
" WHERE (:name IS NULL OR name = :name) AND \n" +
" (:phone IS NULL OR phone = :phone) AND \n" +
" (:age IS NULL OR age = :age)" +,
parameters,
new RealTimeDTOMapper());

To improve an existing answer, Keep a static query with only bind parameters. Don't 'build' a sql to avoid the pitfall of caching issues and sql injection.
List<RealTimeDTO> query(String name, String phone, int age) {
NamedParameterJdbcTemplate jdbcTemplate = new NamedParameterJdbcTemplate(dataSource);
String sql = "SELECT foo, bar FROM FooBar" +
" WHERE (:name IS NULL OR name = :name)" +
" AND (:phone IS NULL OR phone = :phone)" +
" AND (:age = 0 OR age = :age)";
params.put("name", name);
params.put("phone", phone);
params.put("age", age);
return jdbcTemplate.query(sql, params, new CCCRowMapper());
}

hibernate parameter with that name [<name>] did not exist

I am running the following code.
UserService.java
String alias = "u";
String select = "SELECT u.email";
String where = "u.userId = :id";
Map<String, Object> params = new HashMap<>();
params.put("id", userId);
List<User> users = db.findRecords(User.class, alias, select, where, params);
DB.java
public <T> List<T> findRecords(Class<T> entityClass, String entityAlias, String select, String where, Map<String, Object> params) {
String sql = select + " FROM " + entityClass.getName() + " " + entityAlias;
if (where != null) {
sql = sql + " WHERE " + where;
}
Query query = entityManager.createQuery(sql);
System.out.println(sql);
if (!params.isEmpty()) {
Iterator<Entry<String, Object>> iterator = params.entrySet().iterator();
while (iterator.hasNext()) {
Entry<String, Object> entry = iterator.next();
System.out.println("key: " + entry.getKey() + ", value: " + entry.getValue());
query.setParameter((String) entry.getKey(), (Long) entry.getValue());
}
}
return query.getResultList();
}
I am getting the following error log.
SELECT u.email FROM com.catalog.user.User u WHERE u.userId = :id
key: id, value: 28636907
Caused by: java.lang.IllegalArgumentException: Parameter with that name [id] did not exist
If the parameter is getting printed in the console then what is causing the illegal arguments exception to come up?
Please help!

1) check that it is not a typo in the parameter name :id. It's case sensitive.
2) try to execute query without parameters. It might be a problem in the entity mapping.
3) try to set parameter directly in the query without HashMap.

I think the problem here is caused by the way you are getting the params from the Map, with the Map.entrySet() method the iterated values are only available within the iteration and will be undefined outside of it, that's why the param is correctly printed during the loop and doesn't exist in the query.
If you take a look at the Map.entry documentation it says:
These Map.Entry objects are valid only for the duration of the
iteration;
I suggest that you change the way you store and use the parameters , you can simply use a List<Object> to store the parameters or just pass the id param directly in the method call:
public <T> List<T> findRecords(Class<T> entityClass, String entityAlias, String select, String where, Long id){
Then directly append this value in the query:
query.setParameter("id", id);

SimpleJdbcTemplate and null parameters

I'm using SimpleJdbcTemplate and MapSqlParameterSource in the folowing way:
MapSqlParameterSource parameterSource = new MapSqlParameterSource();
parameterSource.addValue("typeId", typeId, Types.BIGINT);
List<Long> ids = _jdbcTemplate.query(_selectIdByParameters, new EntityIdRowMapper(), parameterSource);
When typeId ( which is a Long ) is null, then the query looks in the following way:
SELECT id FROM XXX WHERE typeId = null
whereas I would expect it to generate
SELECT id FROM XXX WHERE typeId IS NULL
I've reported this issue and the response was that
You will have to provide the appropriate SQL statement based on your query parameters.
and as a consequence my code is littered with null checks.
Is there a more elegant way of handling null parameters sent to the SimpleJdbcTemplate?

They have a point - JdbcTemplate isn't a SQL interpreter, it just replaces your placeholders.
I suggest you construct your clause with a utility method, and concat it to the query string:
String createNullCheckedClause(String column, Object value) {
String operator = (value == null ? "is" : "=");
return String.format("(%s %s ?)", column, operator);
}
...
String query = "select * from table where " + createNullCheckedClause("col", x);
Not very pretty. Alternatively, perhaps you can configure MySQL to allow "= NULL", but I don't think that's an option.