I have a problem - I create my SQL queries dynamically and basing on user input options. So the user has 5 parameters (actually it's more) and he can choose to use some of them (all if he wants) or none and specify their value in the query. So I construct my query String (basic the WHERE conditions) by checking if a parameter was selected and if a value was provided. However now there is the problem of special characters like '. I could try to use replaceAll("'", "\\") but this is quite dull and I know that preparedStatement.setString() does the job better. However for me I would need than to check again if the parameter was provided and if the previous one were also (to specify the poison of ? and connect it to the right parameter). This causes a lot of combinations and does not look elegant.
So my question is - can I somehow receive the string preparedStatement.setString() produces? Or is there a similar function that would do the same job and give me the String so I can put it in the query manually.
Maybe the intro was too long but someone might have a better idea and I wanted to explain why I need it.
What you can do is construct the basic, unparameterized SQL query based on whether the parameters were specified, and then use the prepared statement to fill in the parameters.
It could look something like this (rough sketch):
Map<String, Object> parameterValues = /*from user*/;
List<String> parameterNames = Arrays.asList("field1", "field2", "field3");
List<Object> valueList = new ArrayList<Object>();
StringBuilder statementBuilder = new StringBuilder("select * from table where ");
for ( String parameterName : parameterNames ) {
if ( parameterValues.containsKey(parameterName) ) {
statementBuilder.append(parameterName + " = ? AND");
valueList.add(parameterValues.get(parameterName));
}
}
PreparedStatement st = conn.prepareStatement(statementBuilder.toString(),
valueList);
//set each parameter here.
It's only hard the first time; then you can make it generic. That said there are probably query builders that abstract all of this away for you. I use QueryDSL but that does not have bindings for pure JDBC but rather JPA and JDO, etc.
On another forum I was given a different, simpler and cleaner approach that work perfectly.
Here are some links for others with the same problem:
http://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:1669972300346534908
http://www.akadia.com/services/dyn_modify_where_clause.html
Related
Using Jooq, I am trying to fetch from a table by id first, if no matches found, then fetch by handle again.
And I want all fields of the returned rows, not just one.
Field<?> firstMatch = DSL.select(Tables.MY_TABLE.fields())
.from(Tables.MY_TABLE.fields())
.where(Tables.MY_TABLE.ID.eq(id))
.asfield(); // This is wrong, because it supports only one field, but above we selected Tables.MY_TABLE.fields(), which is plural.
Field<?> secondMatch = DSL.select(Tables.MY_TABLE.fields())
.from(Tables.MY_TABLE.fields())
.where(Tables.MY_TABLE.HANDLE.eq(handle))
.asfield(); // Same as above.
dslContext.select(DSL.coalesce(firstMatch, secondMatch))
.fetchInto(MyClass.class);
Due to the mistake mentioned above in the code, the following error occurs:
Can only use single-column ResultProviderQuery as a field
I am wondering how to make firstMatch and secondMatch two lists of fields, instead of two fields?
I tried
Field<?>[] secondMatch = DSL.select(Tables.MY_TABLE.fields())
.from(Tables.MY_TABLE.fields())
.where(Tables.MY_TABLE.HANDLE.eq(handle))
.fields();
but the following error occurred in the line containing DSL.coalesce
Type interface org.jooq.Field is not supported in dialect DEFAULT
Thanks in advance!
This sounds much more like something you'd do with a simple OR?
dslContext.selectFrom(MY_TABLE)
.where(MY_TABLE.ID.eq(id))
// The ne(id) part might not be required...
.or(MY_TABLE.ID.ne(id).and(MY_TABLE.HANDLE.eq(handle))
.fetchInto(MyClass.class);
If the two result sets should be completely exclusive, then you can do this:
dslContext.selectFrom(MY_TABLE)
.where(MY_TABLE.ID.eq(id))
.or(MY_TABLE.HANDLE.eq(handle).and(notExists(
selectFrom(MY_TABLE).where(MY_TABLE.ID.eq(id))
)))
.fetchInto(MyClass.class);
If on your database product, a query using OR doesn't perform well, you can write an equivalent query with UNION ALL, which might perform better.
I am using jdbc PreparedStatement for data insertion.
Statement stmt = conn.prepareStatement(
"INESRT INTO" + tablename+ "("+columnString+") VALUES (?,?,?)");
tablename and columnString are something that is dynamically generated.
I've tried to parameterise tablename and columnString but they will just resolve to something like 'tablename' which will violate the syntax.
I've found somewhere online that suggest me to lookup the database to check for valid tablename/columnString, and cache it somewhere(a Hashset perhaps) for another query, but I'm looking for better performance/ quick hack that will solve the issue, perhaps a string validator/ regex that will do the trick.
Have anyone came across this issue and how do you solve it?
I am not a java-guy, so, only a theory.
You can either format dynamically added identifiers or white-list them.
Second option is way better. Because
most developers aren't familiar enough with identifiers to format them correctly. Say, to quote an identifier, which is offered in the first comment, won't make it protected at all.
there could be another attack vector, not entirely an injection, but similar: imagine there is a column in your table, an ordinary user isn't allowed to - say, called "admin". With dynamically built columnString using data coming from the client side, it's piece of cake to forge a privilege escalation.
Thus, to list all the possible (and allowed) variants in your code beforehand, and then to verify entered value against it, would be the best.
As of columnString - is consists of separate column names. Thus, to protect it, one have to verify each separate column name against a white list, and then assemble a final columnString from them.
Create a method that generates the sql string for you:
private static final String template = "insert into %s (%s) values (%s)";
private String buildStmt(String tblName, String ... colNames) {
StringJoiner colNamesJoiner = new StringJoiner(",");
StringJoiner paramsJoiner = new StringJoiner(",");
Arrays.stream(colNames).forEach(colName -> {
colNamesJoiner.add(colName);
paramsJoiner.add("?");
});
return String.format(template, tblName, colNamesJoiner.toString(), paramsJoiner.toString());
}
Then use it...
Statement stmt = conn.prepareStatement(buildStmt(tablename, [your column names]));
As an elaboration on #Anders' answer, don't use the input parameter as the name directly, but keep a properties file (or database table) that maps a set of allowed inputs to actual table names.
That way any invalid name will not lead to valid SQL (and can be caught before any SQL is generated) AND the actual names are never known outside the application, thus making it far harder to guess what would be valid SQL statements.
I think, the best approach is to get table and columns names from database or other non user input, and use parameters in prepared statement for the rest.
There are multiple solutions we can apply.
1) White List Input Validation
String tableName;
switch(PARAM):
case "Value1": tableName = "fooTable";
break;
case "Value2": tableName = "barTable";
break;
...
default : throw new InputValidationException("unexpected value provided for table name");
By doing this input validation on tableName, will allows only specified tables in the query, so it will prevents sql injection attack.
2) Bind your dynamic columnName(s) or tableName(s) with special characters as shown below
eg:
For Mysql : use back codes (`)
Select `columnName ` from `tableName `;
For MSSQL : Use double codes(" or [ ] )
select "columnName" from "tableName"; or
select [columnName] from [tableName];
Note: Before doing this you should sanitize your data with this special characters ( `, " , [ , ] )
Note: This may be a simple question but I can't find a short way to make it clear. So, sorry for this long question.
In a project I am responsible, I use Spring 2.5, Hibernate 3.3.2 as middleware and Oracle database. As database is related to many other projects, some queries as really very complicated and I can't get a solution with Hibernate's solutions (HQL, Criteria, etc...). So I feel more comfortable with JdbcTemplate's queryForX() methods, as an example;
String sql = "select * from myTable";
jdbc.queryForList(sql);
Sure there are mostly "where" conditions and params indeed:
jdbc.querForList(sql, new Object[]{obj1,obj2,obj3 /* and many more arguments... */})
In this case, I must write question marks "?" for my parameters, so my SQL query string turns out some messy and hard to read; something like this:
select t1.col1, t2.col2, t1.col, --...some cols ,
sum(nvl(some_col1,?)-nvl(other_col2,?)) over (partition by col1,col2,col3,col4) sum_of_cols
from weird_table t1, another_table t2
where t1.col20=? and sum_of_cols>? and t1.col3=t2.col3 --and many ?'s...
and not exists (
select ? from boring_table t3 where -- many ?'s
)
--group by and order by order by etc
So now, which question mark is for which parameter? It is obvious but hard to read. But there are some other solutions for binded params like:
select * from a_table t where t.col1= :col1 and t.col2= :col2 -- and many more ":param"s
For this type query, we can write if it were Hibernate:
Query q = hibernateTemplate.createQuery();
q.setString("col1","a value");
q.setInteger("col2", 3);
I think it is more readable and easy to understand which value is what. I know I can do this with SQLQuery;
SQLQuery sq = hibernateTemplate.createSQLQuery();
/* same as above setInteger() etc. */
But this sq.list() gives me a list without a column name. so I have a basic array which is difficult to use:
[[1,2,"a"],[1,2,"b"], ...]
But with queryForList() I get better one:
[{COL1=1,COL2=2,COL3="a"},{COL1=1,COL2=2,COL3="b"},...]
So if I use queryForList(), I must write a very messy params Object;
or I use SQLQuery and then I have to get my list without a map as column names.
Is there a simple solution with mapped list using more readable param setting (like query.setX()) ?
Well you can use NamedParameterJdbcTemplate to do just that
Heres a sample
String query = "INSERT INTO FORUMS (FORUM_ID, FORUM_NAME, FORUM_DESC)
VALUES (:forumId,:forumName,:forumDesc)";
Map namedParameters = new HashMap();
namedParameters.put("forumId", Integer.valueOf(forum.getForumId()));
namedParameters.put("forumName", forum.getForumName());
namedParameters.put("forumDesc", forum.getForumDesc());
namedParameterJdbcTemplate.update(query, namedParameters);
You check the complete example with the source code in the below link
Spring NamedParameterJdbcTemplate Tutorial
I'm learning the Hibernate Search Query DSL, and I'm not sure how to construct queries using boolean arguments such as AND or OR.
For example, let's say that I want to return all person records that have a firstName value of "bill" or "bob".
Following the hibernate docs, one example uses the bool() method w/ two subqueries, such as:
QueryBuilder b = fts.getSearchFactory().buildQueryBuilder().forEntity(Person.class).get();
Query luceneQuery = b.bool()
.should(b.keyword().onField("firstName").matching("bill").createQuery())
.should(b.keyword().onField("firstName").matching("bob").createQuery())
.createQuery();
logger.debug("query 1:{}", luceneQuery.toString());
This ultimately produces the lucene query that I want, but is this the proper way to use boolean logic with hibernate search? Is "should()" the equivalent of "OR" (similarly, does "must()" correspond to "AND")?.
Also, writing a query this way feels cumbersome. For example, what if I had a collection of firstNames to match against? Is this type of query a good match for the DSL in the first place?
Yes your example is correct. The boolean operators are called should instead of OR because of the names they have in the Lucene API and documentation, and because it is more appropriate: it is not only influencing a boolean decision, but it also affects scoring of the result.
For example if you search for cars "of brand Fiat" OR "blue", the cars branded Fiat AND blue will also be returned and having an higher score than those which are blue but not Fiat.
It might feel cumbersome because it's programmatic and provides many detailed options. A simpler alternative is to use a simple string for your query and use the QueryParser to create the query. Generally the parser is useful to parse user input, the programmatic one is easier to deal with well defined fields; for example if you have the collection you mentioned it's easy to build it in a for loop.
You can also use BooleanQuery. I would prefer this beacuse You can use this in loop of a list.
org.hibernate.search.FullTextQuery hibque = null;
org.apache.lucene.search.BooleanQuery bquery = new BooleanQuery();
QueryBuilder qb = fulltextsession.getSearchFactory().buildQueryBuilder()
.forEntity(entity.getClass()).get();
for (String keyword : list) {
bquery.add(qb.keyword().wildcard().onField(entityColumn).matching(keyword)
.createQuery() , BooleanClause.Occur.SHOULD);
}
if (!filterColumn.equals("") && !filterValue.equals("")) {
bquery.add(qb.keyword().wildcard().onField(column).matching(value).createQuery()
, BooleanClause.Occur.MUST);
}
hibque = fulltextsession.createFullTextQuery(bquery, entity.getClass());
int num = hibque.getResultSize();
To answer you secondary question:
For example, what if I had a collection of firstNames to match against?
I'm not an expert, but according to (the third example from the end of) 5.1.2.1. Keyword queries in Hibernate Search Documentation, you should be able to build the query like so:
Collection<String> namesCollection = getNames(); // Contains "billy" and "bob", for example
StringBuilder names = new StringBuilder(100);
for(String name : namesCollection) {
names.append(name).append(" "); // Never mind the space at the end of the resulting string.
}
QueryBuilder b = fts.getSearchFactory().buildQueryBuilder().forEntity(Person.class).get();
Query luceneQuery = b.bool()
.should(
// Searches for multiple possible values in the same field
b.keyword().onField("firstName").matching( sb.toString() ).createQuery()
)
.must(b.keyword().onField("lastName").matching("thornton").createQuery())
.createQuery();
and, have as a result, Persons with (firstName preferably "billy" or "bob") AND (lastName = "thornton"), although I don't think it will give the good ol' Billy Bob Thornton a higher score ;-).
I was looking for the same issue and have a somewhat different issue than presented. I was looking for an actual OR junction. The should case didn't work for me, as results that didn't pass any of the two expressions, but with a lower score. I wanted to completely omit these results. You can however create an actual boolean OR expression, using a separate boolean expression for which you disable scoring:
val booleanQuery = cb.bool();
val packSizeSubQuery = cb.bool();
packSizes.stream().map(packSize -> cb.phrase()
.onField(LUCENE_FIELD_PACK_SIZES)
.sentence(packSize.name())
.createQuery())
.forEach(packSizeSubQuery::should);
booleanQuery.must(packSizeSubQuery.createQuery()).disableScoring();
fullTextEntityManager.createFullTextQuery(booleanQuery.createQuery(), Product.class)
return persistenceQuery.getResultList();
Title asks it all... I want to do a multi field - phrase search in Lucene.. How to do it ?
for example :
I have fields as String s[] = {"title","author","content"};
I want to search harry potter across all fields.. How do I do it ?
Can someone please provide an example snippet ?
Use MultiFieldQueryParser, its a QueryParser which constructs queries to search multiple fields..
Other way is to use Create a BooleanQuery consisting of TermQurey (in your case phrase query).
Third way is to include the content of other fields into your default content field.
Add
Generally speaking, querying on multiple fields isn’t the best practice for user-entered queries. More commonly, all words you want searched are indexed into a contents or keywords field by combining various fields.
Update
Usage:
Query query = MultiFieldQueryParser.parse(Version.LUCENE_30, new String[] {"harry potter","harry potter","harry potter"}, new String[] {"title","author","content"},new SimpleAnalyzer());
IndexSearcher searcher = new IndexSearcher(...);
Hits hits = searcher.search(query);
The MultiFieldQueryParser will resolve the query in this way: (See javadoc)
Parses a query which searches on the
fields specified. If x fields are
specified, this effectively
constructs:
(field1:query1) (field2:query2)
(field3:query3)...(fieldx:queryx)
Hope this helps.
intensified googling revealed this :
http://lucene.472066.n3.nabble.com/Phrase-query-on-multiple-fields-td2292312.html.
Since it is latest and best, I'll go with his approach I guess.. Nevertheless, it might help someone who is looking for something like I am...
You need to use MultiFieldQueryParser with escaped string. I have tested it with Lucene 8.8.1 and it's working like magic.
String queryStr = "harry potter";
queryStr = "\"" + queryStr.trim() + "\"";
Query query = new MultiFieldQueryParser(new String[]{"title","author","content"}, new StandardAnalyzer()).parse(queryStr);
System.out.println(query);
It will print.
(title:"harry potter") (author:"harry potter") (content:"harry potter")