We Keep Coding

How to pass query param to REST API to do a IN query in DB [duplicate]

This question already has answers here:
PreparedStatement IN clause alternatives?
(33 answers)
Closed 5 years ago.
Say that I have a query of the form
SELECT * FROM MYTABLE WHERE MYCOL in (?)
And I want to parameterize the arguments to in.
Is there a straightforward way to do this in Java with JDBC, in a way that could work on multiple databases without modifying the SQL itself?
The closest question I've found had to do with C#, I'm wondering if there is something different for Java/JDBC.

There's indeed no straightforward way to do this in JDBC. Some JDBC drivers seem to support PreparedStatement#setArray() on the IN clause. I am only not sure which ones that are.
You could just use a helper method with String#join() and Collections#nCopies() to generate the placeholders for IN clause and another helper method to set all the values in a loop with PreparedStatement#setObject().
public static String preparePlaceHolders(int length) {
return String.join(",", Collections.nCopies(length, "?"));
}
public static void setValues(PreparedStatement preparedStatement, Object... values) throws SQLException {
for (int i = 0; i < values.length; i++) {
preparedStatement.setObject(i + 1, values[i]);
}
}
Here's how you could use it:
private static final String SQL_FIND = "SELECT id, name, value FROM entity WHERE id IN (%s)";
public List<Entity> find(Set<Long> ids) throws SQLException {
List<Entity> entities = new ArrayList<Entity>();
String sql = String.format(SQL_FIND, preparePlaceHolders(ids.size()));
try (
Connection connection = dataSource.getConnection();
PreparedStatement statement = connection.prepareStatement(sql);
) {
setValues(statement, ids.toArray());
try (ResultSet resultSet = statement.executeQuery()) {
while (resultSet.next()) {
entities.add(map(resultSet));
}
}
}
return entities;
}
private static Entity map(ResultSet resultSet) throws SQLException {
Enitity entity = new Entity();
entity.setId(resultSet.getLong("id"));
entity.setName(resultSet.getString("name"));
entity.setValue(resultSet.getInt("value"));
return entity;
}
Note that some databases have a limit of allowable amount of values in the IN clause. Oracle for example has this limit on 1000 items.

Since nobody answer the case for a large IN clause (more than 100) I'll throw my solution to this problem which works nicely for JDBC. In short I replace the IN with a INNER JOIN on a tmp table.
What I do is make what I call a batch ids table and depending on the RDBMS I may make that a tmp table or in memory table.
The table has two columns. One column with the id from the IN Clause and another column with a batch id that I generate on the fly.
SELECT * FROM MYTABLE M INNER JOIN IDTABLE T ON T.MYCOL = M.MYCOL WHERE T.BATCH = ?
Before you select you shove your ids into the table with a given batch id.
Then you just replace your original queries IN clause with a INNER JOIN matching on your ids table WHERE batch_id equals your current batch. After your done your delete the entries for you batch.

The standard way to do this is (if you are using Spring JDBC) is to use the org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate class.
Using this class, it is possible to define a List as your SQL parameter and use the NamedParameterJdbcTemplate to replace a named parameter. For example:
public List<MyObject> getDatabaseObjects(List<String> params) {
NamedParameterJdbcTemplate jdbcTemplate = new NamedParameterJdbcTemplate(dataSource);
String sql = "select * from my_table where my_col in (:params)";
List<MyObject> result = jdbcTemplate.query(sql, Collections.singletonMap("params", params), myRowMapper);
return result;
}

I solved this by constructing the SQL string with as many ? as I have values to look for.
SELECT * FROM MYTABLE WHERE MYCOL in (?,?,?,?)
First I searched for an array type I can pass into the statement, but all JDBC array types are vendor specific. So I stayed with the multiple ?.

I got the answer from docs.spring(19.7.3)
The SQL standard allows for selecting rows based on an expression that includes a variable list of values. A typical example would be select * from T_ACTOR where id in (1, 2, 3). This variable list is not directly supported for prepared statements by the JDBC standard; you cannot declare a variable number of placeholders. You need a number of variations with the desired number of placeholders prepared, or you need to generate the SQL string dynamically once you know how many placeholders are required. The named parameter support provided in the NamedParameterJdbcTemplate and JdbcTemplate takes the latter approach. Pass in the values as a java.util.List of primitive objects. This list will be used to insert the required placeholders and pass in the values during the statement execution.
Hope this can help you.

AFAIK, there is no standard support in JDBC for handling Collections as parameters. It would be great if you could just pass in a List and that would be expanded.
Spring's JDBC access supports passing collections as parameters. You could look at how this is done for inspiration on coding this securely.
See Auto-expanding collections as JDBC parameters
(The article first discusses Hibernate, then goes on to discuss JDBC.)

See my trial and It success,It is said that the list size has potential limitation.
List l = Arrays.asList(new Integer[]{12496,12497,12498,12499});
Map param = Collections.singletonMap("goodsid",l);
NamedParameterJdbcTemplate namedParameterJdbcTemplate = new NamedParameterJdbcTemplate(getJdbcTemplate().getDataSource());
String sql = "SELECT bg.goodsid FROM beiker_goods bg WHERE bg.goodsid in(:goodsid)";
List<Long> list = namedParameterJdbcTemplate.queryForList(sql, param2, Long.class);

There are different alternative approaches that we can use.
Execute Single Queries - slow and not recommended
Using Stored Procedure - database specific
Creating PreparedStatement Query dynamically - good performance but loose benefits of caching and needs recompilation
Using NULL in PreparedStatement Query - I think this is a good approach with optimal performance.
Check more details about these here.

sormula makes this simple (see Example 4):
ArrayList<Integer> partNumbers = new ArrayList<Integer>();
partNumbers.add(999);
partNumbers.add(777);
partNumbers.add(1234);
// set up
Database database = new Database(getConnection());
Table<Inventory> inventoryTable = database.getTable(Inventory.class);
// select operation for list "...WHERE PARTNUMBER IN (?, ?, ?)..."
for (Inventory inventory: inventoryTable.
selectAllWhere("partNumberIn", partNumbers))
{
System.out.println(inventory.getPartNumber());
}

One way i can think of is to use the java.sql.PreparedStatement and a bit of jury rigging
PreparedStatement preparedStmt = conn.prepareStatement("SELECT * FROM MYTABLE WHERE MYCOL in (?)");
... and then ...
preparedStmt.setString(1, [your stringged params]);
http://java.sun.com/docs/books/tutorial/jdbc/basics/prepared.html

How to get parameterized query with iBatis-Spring

I'm working on an application with spring-ibatis integration in which I have to log some of the query performed. So what I'd like to do, is basically getting the SQL from the ibatis mapped statements in the XML config file and then add somehow the parameters. I've been able to get the query with this lines of code:
MappedStatement ms = (MappedStatement) ((SqlMapClientImpl) sqlMapClient)
.getDelegate().getMappedStatement(queryId);
ms.setParameterClass(HashMap.class);
RequestScope scope = new RequestScope();
scope.setStatement(ms);
String sql = ((DynamicSql) ms.getSql()).getSql(scope, params);
So with the first row I get the MappedStatement and with the last one I get the raw query. The problem is that even if I'm passing to it the object with the query parameters, the SQL still has the parameters placeholders '?' (in the XML query they are named parameters, not positionals).
I have tried to set the parameterClass field instead of the parameterMap as suggested here but with no success. I'm not sure on how to work with the inline parameters.
I'm using ibatis-sqlmap 2.3.0 and spring-ibatis 2.0.8.
As you have probably noticed I have little to no knowledge of iBatis. Also, please I know that this is dirty and that I'm using classes that I'm not supposed to, no need to point that out.
Thank you for the help.

I've solved this problem and I want to share the solution for future readers that may have the same issue. Before doing that, keep in mind that this is NOT the way you should work with iBatis but only a dirty workaround to get the underlying SQL.
First of all, we need to group iBatis queries in at least 2 groups:
Static queries, they are simple mapped statements without any conditional elements.
Dynamic queries, they are mapped statements with conditional elements (e.g. isEqual, isGreaterThan, isNull...).
Once you have done this difference, here's the code to get the SQL:
public static String getSQLFromDynamicQuery(SqlMapClient sqlMapClient,
String queryId, Object paramObject) {
// Gets the SQL and parameters.
MappedStatement ms = ((SqlMapClientImpl) sqlMapClient).getDelegate()
.getMappedStatement(queryId);
RequestScope scope = new RequestScope();
scope.setStatement(ms);
String sql = ((DynamicSql) ms.getSql()).getSql(scope, paramObject);
Object[] params = ms.getSql().getParameterMap(scope, paramObject)
.getParameterObjectValues(scope, paramObject);
// Adds params to the query.
return bindQueryParam(sql, params);
}
public String getSQLFromStaticQuery(SqlMapClient sqlMapClient,
String queryId, Object... params) {
// Gets the SQL.
String sql = ((StaticSql) ((SqlMapClientImpl) sqlMapClient)
.getDelegate().getMappedStatement(queryId).getSql()).getSql(
null, null);
// Adds params to the query.
if (params != null) {
sql = bindQueryParam(sql, params);
}
return sql;
}
public static String bindQueryParam(String sql, Object... params) {
String result = sql;
for (Object param : params) {
result = result.replaceFirst("\\?",
param == null ? "null" : param.toString());
}
return result;
}
The bindQueryParam method replaces the question marks in the query with an array of object. For a static query, you will have to pass that array meanwhile for the dynamic one you can pass an Object or a java.util.Map, according to what is your parameterClass of the Mapped Statement.
Both methods use explicit subcasting (I've spent a lot of time looking at the source code to figure out how to make this work as you can imagine), so you may want to pay attention to call the right method according to the Mapped Statement you are processing or you will get a ClassCastException.
Again, this is not the recommended way but it works if you need it.

Maximo: How to read a field of an application using JAVA?

This question probably is easy. I am trying to read a field of a IBM Maximo application and use this value in the method getList(). The value I want to use was not saved in the database yet.
Here is some pseudocode:
#Override
public MboSetRemote getList() throws MXException, RemoteException {
MboSetRemote result = super.getList();
//Here is where i dont know how to do it
Date field = getFieldValue(FieldName)
//Here is where i want to use the value
String string = "....field..."
result.setWhere(string);
return result;
}
Thanks everyone,
Regards

I think the easiest and safest means to achieve your end of using the field value in your where clause is to use a bind variable, like this:
#Override
public MboSetRemote getList() throws MXException, RemoteException {
MboSetRemote result = super.getList();
//Here is where i want to use the value
String string = "....:fieldName...";
result.setWhere(string);
return result;
}
Notice the colon on the front of :fieldName in string. When Maximo sees this, it will look (not case-sensitive) on the current record / Mbo for an attribute named fieldName and replace :fieldName with the value in the attribute -- wrapped in quotes or whatever, as applicable to the attribute's type (ALN, UPPER, DATE, etc).
This approach is better than the approach you presented because it will employ Maximo's framework to prevent SQL injection attacks and etc.
That said, the way to get the field value would be as follows:
Date fieldValue = getMboValue("FieldName").getDate();
Further, I strongly suggest you get yourself a copy of Maximo's JavaDocs. You can do that here.

Java create PreparedStatement with no SQL content

Is it possible to create a PreparedStatement in java without setting the initial SQL query?
Example code:
#Override
public List<AccountBean> search(AccountConstraint... c) {
if (c.length == 0) {
throw new IllegalArgumentException("dao.AccountDAO.search: c.length == 0");
}
try {
List<AccountBean> beans = new ArrayList<>();
for (AccountConstraint ac : c) {
PreparedStatement ps = connection.prepareStatement(null);
QueryBuilder queryBuilder = new QueryBuilder(ps, "SELECT * FROM accounts");
queryBuilder.add(ac.getAccountIdConstraint());
queryBuilder.add(ac.getUsernameConstraint());
queryBuilder.add(ac.getPasswordConstraint());
queryBuilder.add(ac.getEmailConstraint());
//INSERT QUERY INTO PS
ResultSet rs = ps.executeQuery();
while (rs.next()) {
beans.add(new AccountBean(rs));
}
}
return beans;
} catch (SQLException ex) {
throw new RuntimeException(ex);
}
}
The trick is in QueryBuilder, this class is responsible for building parts of a query based on the initial SELECT part and then adds respective WHERE and AND clauses.
However to ensure that all data is safe, the actual arguments must also be put in the PreparedStatement, hence why it is being passed to the QueryBuilder.
Every QueryBuilder.add() adds some arguments into the PreparedStatement and appends a specific string to the end of the query.
I think some workarounds are possible, such as instead of giving a PreparedStatement to the QueryBuilder you would give a List<Object> and then you would write a custom function that puts them in the PreparedStatement later on.
But what are your thoughts, suggestions on this?
Regards.
Solution added
Few key changes first:
QueryBuilder now implements the Builder pattern properly.
QueryBuilder.add() accepts multiple Constraints at once.
AccountConstraint can give an array that gives all Constraints now.
#Override
public List<AccountBean> search(AccountConstraint... c) {
if (c.length == 0) {
throw new IllegalArgumentException("dao.AccountDAO.search: c.length == 0");
}
try {
List<AccountBean> beans = new ArrayList<>();
for (AccountConstraint ac : c) {
try (PreparedStatement ps = new QueryBuilder("SELECT * FROM accounts").add(ac.getConstraints()).build();ResultSet rs = ps.executeQuery()) {
while (rs.next()) {
beans.add(new AccountBean(rs));
}
}
}
return beans;
} catch (SQLException ex) {
throw new RuntimeException(ex);
}
}
ps. I get two statements in one try{ } because of the try-with-resources.

Preparing a statement means compiling it so you can efficiently execute it many times with different arguments. So, no it does not make sense to compile a query before it is defined.
As I understand, you want to use the Java compiler to assist you in dynamically defining the query. Why don't you just create the prepared statement in a compile() method, thus, as the result of your builder. Also, your code becomes more readable and more resembles a declarative query if you use the builder pattern such that each call to add() returns this. Then you can write your query like this:
PreparedStatement ps = new QueryBuilder()
.select("*")
.from("accounts")
.where()
.add(yourConstraint())
...
.compile();
However, you must create the prepared statement before the loop. Otherwise, if you keep a reference to the builder and call compile() in your loop you will get a new prepared statement on every call. So you won't get the benefit of reusing a precompiled query. In the loop you only assign values to the variables in your prepared statement.

You can't modify the prepared statement via the API after you crate it. You can't create it without an SQL statement either.
Why not create the query separately and then bind the parameters? You can use a Map to hold the parameter placeholders and their values so they can be set to the prepared statement.
Although I'd just use the Spring's JDBC templates to get the same thing done more quickly.

How to improve your SQL query builder
If you look at how popular query builders like jOOQ and others do it, the idea is that you separate your concerns more thoroughly. You should have:
An expression tree representation of your SQL statement (and ideally that doesn't directly operate on strings)
A way to construct that expression tree conveniently, e.g. by using a DSL
Some sort of execution lifecycle management that generates the SQL string, prepares the statement, binds the variables, etc.
Or in code (jOOQ example, but this could also apply to your own query builder):
Result<?> result =
// This constructs the expression tree through the jOOQ DSL
ctx.selectFrom(ACCOUNTS)
.where(ac.getAccountIdConstraint())
.and(ac.getUsernameConstraint())
.and(ac.getPasswordConstraint())
.and(ac.getEmailConstraint())
// This internally creates a PreparedStatement, binds variables, executes it, and maps results
.fetch();
Of course, your AccountConstraint.getXYZConstraint() methods would not return SQL string snippets, but again expression tree elements. In the case of jOOQ, this would be a Condition
(Disclaimer: I work for the vendor of jOOQ)
How to improve your SQL performance
I've noticed that you run N queries for N AccountConstraint values, and you mix the results in a way that it doesn't matter which AccountConstraint value produced which AccountBean. I strongly suggest you move that loop into the generated SQL query, as you're going to get much faster results on pretty much every database. I've blogged about this here.

Pass varargs from Java code to SQL or PL/SQL

I need to bind at maximum 8 variables. Each one of them could be null.
Is there any recommended way to achieve this? I know that I could simply check for null, but this seems tedious.
Additional details:
I'm going to call this sql from java code. It may be written using JPA 2.0 Criteria API, but most likely it's going to be a native query. The database is Oracle 10g, so I think I could make use of PL/SQL as well.
Edit1:
Maybe the title is a bit misleading, so I'll try to elaborate.
The resulting SQL would be something like:
...
WHERE var1 = :var1
AND var2 = :var2
...
AND var = :var8
Now I need to bind parameters from java code in the way like:
nativeQuery.setParameter("var1", var1)
...
nativeQuery.setParameter("var8", var8)
Some parameters could be null, so there is no need to bind them. But I see no way I can omit them in SQL.
Edit2:
I'm expecting to see SQL or PL/SQL procedure in your answers (if it's ever possible without null checking).
In fact, all of these variables are of the same type. I think it's not possible to find a solution using ANSI SQL, but maybe there are some PL/SQL procedures which allow to work with varargs?

The use of a criteria query is appropriate in this case, because if I understood correctly, you need to construct the SQL query dynamically. If all the variables except var1 are null, the where clause would be
where var1 = :var1
and if all variables except var2 and var5 are non null you would have
where var2 = :var2 and var5 = :var5
Is that right?
If so, then do what you plan to do, and construct the query dynamically using a criteria query. Something like this must be done:
CriteriaBuilder builder = em.getCriteriaBuilder();
Predicate conjunction = builder.conjunction();
if (var1 != null) {
conjunction = builder.and(conjunction,
builder.equal(root.get(MyEntity_.var1),
var1));
}
if (var2 != null) {
conjunction = builder.and(conjunction,
builder.equal(root.get(MyEntity_.var2),
var2));
}
...
criteria.where(conjunction);

You don't specify the type of the objects you want to pass. So in this example I'm considering you will pass Object.
#Test(expected=IllegalArgumentException.class)
public void testMyMethod() {
List<Object> testList = new ArrayList<Object>();
testList.add("1");
testList.add("2");
testList.add(3);
myMethod(testList);
}
public void myMethod(List<Object> limitedList) {
final int MAX_SIZE = 2;
if (limitedList.size() > MAX_SIZE) {
throw new IllegalArgumentException("Size exceeded");
}
//my logic
}
In this example I'm passing the arguments as a List of Objects but you could use array (varargs) or another type of collection if you need to. If the client sends me more than the expected objects it will throw an IllegalArgumentException.
Also if you don't want to throw an exception you could just continue and iterate the list to bind the parameters but using the list size or MAX_SIZE as your limit. For example:
public void myMethod2(List<Object> limitedList) {
final int MAX_SIZE = 2;
int size = MAX_SIZE;
if (limitedList.size() < MAX_SIZE) {
size = limitedList.size();
}
//Iterate through the list
for (int i = 0; i < size; i++) {
Object obj = limitedList.get(i);
//Logic to bind the obj to the criteria.
}
}