I have a problem.
I have the following query:
SELECT
Agents.Owner,
Orders.*
FROM
Orders
INNER JOIN Agents ON Agents.id = Orders.agentid
WHERE
Agents.botstate = 'Active' AND Orders.state = 'Active' AND(
Orders.status = 'Failed' OR Orders.status = 'Processing' AND Orders.DateTimeInProgressMicro < DATE_SUB(NOW(), INTERVAL 10 SECOND))
ORDER BY
Orders.agentid
But now I need to convert this to JOOQ language. This is what I came up with:
create.select()
.from(DSL.table("Orders"))
.join(DSL.table("Agents"))
.on(DSL.table("Agents").field("Id").eq(DSL.table("Orders").field("AgentId")))
.where(DSL.table("Agents").field("botstate").eq("Active")
.and(DSL.table("Orders").field("state").eq("Active"))
.and((DSL.table("Orders").field("status").eq("Failed"))
.or(DSL.table("Orders").field("status").eq("Processing")))).fetch().sortAsc(DSL.table("Orders").field("AgentId"));
Now the first problem is that it doesn't like all the .eq() statements, because it gives me the error:
Cannot resolve method: eq(Java.lang.String). And my second problem is that I don't know how to write this statement in JOOQ: Orders.DateTimeInProgressMicro < DATE_SUB(NOW(), INTERVAL 10 SECOND).
The first problem is caused by the fact that I can't just use:
.on(Agents.Id).eq(Orders.AgentId)
But instead I need to enter for every table:
DSL.table("table_name")
And for every column:
DSL.field("column_name")
Without that it doesn't recognize my tables and columns
How can I write the SQL in the JOOQ version correctly or an alternative solution is that I can use normal SQL statements?
Why doesn't your code work?
Table.field(String) does not construct a path expression of the form table.field. It tries to dereference a known field from Table. If Table doesn't have any known fields (e.g. in the case of using DSL.table(String), then there are no fields to dereference.
Correct plain SQL API usage
There are two types of API that allow for working with dynamic SQL fragments:
The plain SQL API to construct plain SQL fragments and templates
The Name API to construct identifiers and jOOQ types from identifiers
Most people use these only when generated code isn't possible (see below), or jOOQ is missing some support for vendor-specific functionality (e.g. some built-in function).
Here's how to write your query with each:
Plain SQL API
The advantage of this API is that you can use arbitrary SQL fragments including vendor specific function calls that are unknown to jOOQ. There's a certain risk of running into syntax errors, SQL injection (!), and simple data type problems, because jOOQ won't know the data types unless you tell jOOQ explicitly
// as always, this static import is implied:
import static org.jooq.impl.DSL.*;
And then:
create.select()
.from("orders") // or table("orders")
.join("agents") // or table("agents")
.on(field("agents.id").eq(field("orders.id")))
.where(field("agents.botstate").eq("Active"))
.and(field("orders.state").eq("Active"))
.and(field("orders.status").in("Failed", "Processing"))
.orderBy(field("orders.agentid"))
.fetch();
Sometimes it is useful to tell jOOQ about data types explicitly, e.g. when using these expressions in SELECT, or when creating bind variables:
// Use the default SQLDataType for a Java class
field("agents.id", Integer.class);
// Use an explicit SQLDataType
field("agents.id", SQLDataType.INTEGER);
Name API
This API allows for constructing identifiers (by default quoted, but you can configure that, or use unquotedName()). If the identifiers are quoted, the SQL injection risk is avoided, but then in most dialects, you need to get case sensitivity right.
create.select()
.from(table(name("orders")))
.join(table(name("agents")))
.on(field(name("agents", "id")).eq(field(name("orders", "id"))))
.where(field(name("agents", "botstate")).eq("Active"))
.and(field(name("orders", "state")).eq("Active"))
.and(field(name("orders", "status")).in("Failed", "Processing"))
.orderBy(field(name("orders", "agentid")))
.fetch();
Using the code generator
Some use cases prevent using jOOQ's code generator, e.g. when working with dynamic schemas that are only known at runtime. In all other cases, it is very strongly recommended to use the code generator. Not only will building your SQL statements with jOOQ be much easier in general, you will also not run into problems like the one you're presenting here.
Your query would read:
create.select()
.from(ORDERS)
.join(AGENTS)
.on(AGENTS.ID.eq(ORDERS.ID))
.where(AGENTS.BOTSTATE.eq("Active"))
.and(ORDERS.STATE.eq("Active"))
.and(ORDERS.STATUS.in("Failed", "Processing"))
.orderBy(ORDERS.AGENTID)
.fetch();
Benefits:
All tables and columns are type checked by your Java compiler
You can use IDE auto completion on your schema objects
You never run into SQL injection problems or syntax errors
Your code stops compiling as soon as you rename a column, or change a data type, etc.
When fetching your data, you already know the data type as well
Your bind variables are bound using the correct type without you having to specify it explicitly
Remember that both the plain SQL API and the identifier API were built for cases where the schema is not known at compile time, or schema elements need to be accessed dynamically for any other reason. They are low level APIs, to be avoided when code generation is an option.
Related
How is this a duplicate as i am specifically asking about JOOQ here?
I am using JOOQ in my JAVA project to handle all my PostgreSQL queries. I read in this article that JOOQ uses prepared statements to execute all queries.
Is it than safe to assume that I don't need to worry about SQL injection or user input when executing my queries?
I don't need to worry about escaping the user input before giving it over to JOOQ?
On the side note, which other vulnerabilities are there to my DB in getting user input (apart from those that are solved by prepared statements), that I should be careful of?
1) Yes, as long as you use the provided API's correctly. It is still possible to inject plain sql queries though so be careful.
All methods in the jOOQ API that allow for plain (unescaped, untreated) SQL contain a warning message in their relevant Javadoc
// This query will use bind values, internally.
create.fetch("SELECT * FROM BOOK WHERE ID = ? AND TITLE = ?", 5, "Animal Farm");
// This query will not use bind values, internally.
create.fetch("SELECT * FROM BOOK WHERE ID = 5 AND TITLE = 'Animal Farm'");
See JOOQ docs here for a more in depth explanation: https://www.jooq.org/doc/3.9/manual/sql-building/bind-values/sql-injection/
2) No, see above.
3) Aside from that just beware of general DB security issues, such as user authentication/roles and storing sensitive data in an unecrypted format etc
Little risk when using jOOQ as intended
When you use jOOQ as intended, then you will run into little risk of SQL injection. The intended usage is:
Using source code generation to generate meta data for your tables / columns, etc.
Using the DSL for type safe embedded SQL
As others have mentioned, jOOQ will always use bind variables, properly escape all inlined values (constants, literals). But again, as others have mentioned, jOOQ still allows for using plain SQL templating for those cases where you need to work around a lack of functionality or vendor specific feature support. In those cases, you have to be as careful as with JDBC and make sure to explicitly use bind variables and avoid string concatenation, yourself.
Preventing accidents with the PlainSQLChecker annotation processor
One way to prevent accidentally using plain SQL templating, and to make sure no one on the team uses it without approval is to use jOOQ's checker framework / error prone integration and disallow all plain SQL usage by default. With Maven, you could configure this (leaving out the JDK version specific details):
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<annotationProcessors>
<annotationProcessor>org.jooq.checker.PlainSQLChecker</annotationProcessor>
</annotationProcessors>
</configuration>
</plugin>
And now your code using methods like DSL.query(String) won't compile anymore, until you explicitly allow it with the #Allow.PlainSQL annotation on the scope of your choice (method, class, package)
It's always possible to write unsafe queries, no matter what language and framework you use.
The naive way of concatenating variables into SQL creates an opportunity for SQL injection:
String unsafeString = "O'Reilly";
create.fetch("SELECT * FROM BOOK WHERE ID = 5 AND TITLE = '"+unsafeString+"'");
// results in SQL syntax error because of unmatched ' marks
Merely using prepared queries does NOT make an unsafe query into a safe query.
Use parameters to separate dynamic values from the SQL query. These are combined within the RDBMS at execution time. There is no way a parameter can cause an SQL injection vulnerability.
String unsafeString = "O'Reilly";
create.fetch("SELECT * FROM BOOK WHERE ID = 5 AND TITLE = ?", unsafeString);
// still OK
When you use parameters, you don't need to do any escaping of the variables. In fact, you must not, because you'll end up with escape symbols in your data.
Parameters are good for combining Java variables into an SQL query, but only in the place of an SQL scalar value. That is, where you would normally use a quoted string literal, quoted date literal, or numeric literal in your SQL, you can replace it with a parameter placeholder.
But you can't use parameters for anything else in SQL:
Table names
Column names
Lists of values, for example for an IN ( ... ) predicate—you must use one ? placeholder per individual value in the list.
SQL expressions
SQL keywords
You might like my presentation SQL Injection Myths and Fallacies (video), or my book, SQL Antipatterns Volume 1: Avoiding the Pitfalls of Database Programming
Re comment from #rehas:
Indeed, using prepared statements does not mean you are using parameters implicitly. I showed an example above (my first example), of concatenating an unsafe variable into an SQL string before it is sent to prepare().
Once the SQL string arrives in the RDBMS server, it has no way of knowing which parts of the string were legitimate and which parts were concatenated from unsafe variables. All it sees is one string containing an SQL statement.
The point of using parameters is to keep the (potentially unsafe) variables separated from the SQL string. Within the RDBMS server, the SQL string—still with parameter placeholders like ?—is parsed. Once it's parsed, it won't be parsed again, so it's safe for strings like "O'Reilly" to be bound to the parameter placeholders without risk of causing mismatched quotes or anything. A parameters is guaranteed to be treated as a single value in the SQL execution, even if the value of the parameter contains characters that would have changed the way the query was parsed, if it had been included before prepare().
It's not true that using prepare() means you're always using parameters.
It's accurate to say that using parameters requires use of prepare() and execute() as separate steps. But some frameworks do both steps for you. I'm sure if you were to read the jOOQ source code, you'd see it.
I'm using Apache Derby as an in-memory mock database for unit testing some code that works with MySQL using jOOQ.
The production database uses enums for certain fields (this is a given and out of scope of this question - I know enums are bad but I can't change this part now), so jOOQ generates code to handle the enums.
Unfortunately, Derby does not support enums and when I try to create the database in Derby (from jOOQ SQL generator), I get errors.
My solution was to user-defined types that mimic the enum by wrapping the relevant jOOQ generated enum Java class. So, for example, if I have an enum field kind in the table stuffs, jOOQ SQL generator creates Derby table creation SQL that talks about stuffs_kind.
To support this I created the class my.project.tests.StuffsKindDebyEnum that wraps the jOOQ generated enum type my.project.model.StuffsKind. I then run the following SQL through Derby, before running the jOOQ database creation SQL:
CREATE TYPE stuffs_kind EXTERNAL NAME 'my.project.tests.StuffsKindDerbyEnum' LANGUAGE JAVA
When I then use jOOQ to insert new records, jOOQ generates SQL that looks somewhat like this:
insert into "schema"."stuffs" ("text", "kind")
values (cast (? as varchar(32672)), cast(? as stuffs_kind)
But binds a string value to the kind argument (as expected), and it work for MySQL but with Derby I get an exception:
java.sql.SQLDataException: An attempt was made to get a data value of type
'"APP"."STUFFS_KIND"' from a data value of type 'VARCHAR'
After looking at all kinds of ways to solve this problem (including trying to treat enums as simple VARCHARs), and before I give up on being able to test my jOOQ-using code, is there a way to get Derby to "cast" varchar into user-defined types? If could put some Java code that can handle that, it will not be a problem as I can simply do StuffsKind.valueOf(value) to convert a string to the correct enum type, but after perusing the (very minimal) Derby documentation, I can't figure out if it is even should be possible.
Any ideas are welcome!
Implementing a dialect sensitive custom data type binding:
The proper way forward here would be to use a dialect sensitive, custom data type binding:
https://www.jooq.org/doc/latest/manual/sql-building/queryparts/custom-bindings
The binding could then implement, e.g. the bind variable SQL generation as follows:
#Override
public void sql(BindingSQLContext<StuffsKindDerbyEnum> ctx) throws SQLException {
if (ctx.family() == MYSQL)
ctx.render().visit(DSL.val(ctx.convert(converter()).value()));
else if (ctx.family() == DERBY)
ctx.render()
.sql("cast(
.visit(DSL.val(ctx.convert(converter()).value()))
.sql(" as varchar(255))");
else
throw new UnsupportedOperationException("Dialect not supported: " + ctx.family());
}
You'd obviously also have to implement the other methods that tell jOOQ how to bind your variable to a JDBC PreparedStatement, or how to fetch it from a ResultSet
Avoiding the MySQL enum
Another, simpler way forward might be to avoid the vendor-specific feature and just use VARCHAR in both databases. You can still map that VARCHAR to a Java enum type using a jOOQ Converter that will work the same way in both databases.
Simplify testing by avoiding Derby
A much simpler way forward is to test your application directly on MySQL, e.g. on an in-memory docker virtualisation. There are a lot of differences between database vendors and their features, and at some point, working around those differences just to get slightly faster tests doesn't seem reasonable.
The exception is, of course, if you have to support both Derby and MySQL in production, in case of which the data type binding is again the best solution.
I have a Java, GraphQL, Hibernate, PostgreSQL, QueryDSL application that queries a very large PostgreSQL table with over 275 columns.
I've created a GraphQL schema with the 25 most popular columns as query-able fields. I'd like to add a generic "field" input type that consists of a name (the db column name + "_" + operation (like gte, gt, contains, etc.) and a value (the value the user is searching for).
So when the user (in GraphiQL) enters something like (field:{name:"age_gt", value:"50"}) as a search input to the GraphQL query, I can come up with: "age > 50".
All that works fine, but when it's time to create the Predicate and add it to the whole query ( booleanBuilder.and(new Predicate) ), I cannot figure out how to create a Predicate that just contains a raw String of SQL ("age > 50").
I've created several Predicates the "right" way using my entity POJO tied to Hibernate and the jpa generated "Q" object. But I need the ability to add one or more Predicates that are just a String of SQL. I'm not even sure if the ability exists, the documentation for QueryDSL Predicates is non-existent.
I'm thinking PredicateOperation() might be the answer, but again, no documentation and I cannot find any examples online.
My apologies for not posting code, all my stuff is behind a firewall on a different network so there's no cut and paste to my internet machine.
In Hibernate its possible to inject arbitrary SQL using custom functions or the FUNCTION-function (introduced in JPA 2.1). In QueryDSL its possible to inject arbitrary JPQL/HQL through TemplateExpressions. Combined you get:
Expressions.numberTemplate("FUNCTION('SUM', {0}), x)
However, age > 50 as expression is probably valid JPQL as well, so one can just write:
Expressions.numberTemplate("SUM(age)")
Either way, its probably best to create a visitor that traverses the GraphQL query and creates the proper expression in QueryDSL, as TemplateExpressions are prone to SQL injection.
I'm currently evaluating JOOQ because I believe I started reinventing the wheel which looks very close to part of JOOQ :)
Now, while digging in great JOOQ documentation I've found that my use case lies somewhere between Using JOOQ as SQL Builder and Using JOOQ as SQL Builder with Code generation i.e. I would like to:
Create plain SQL strings like it is shown in Using JOOQ as SQL Builder part
Instead of using hard-coded DSL.fieldByName("BOOK","TITLE") constructs, I prefer storing name of a table along with it's column names and types like it's shown in Using JOOQ as SQL Builder with Code generation part
I prefer not to use code generation (at least not on regular basis), but rather creating TableImpl myself when new table is needed.
While digging in manual, I've found how table implementation should look like in chapter Generated tables. However, TableImpl class as well as Table interface should be parameterized with record type and the same goes for TableField class. I believe this is done for easier type inference when directly querying database and retrieving results, though I may be mistaken.
So my questions are:
Is there a guide in manual on how to create Table and TableField implementations? Or I can simply generate them once for my database schema and use generated code as a guideline?
How can I gracefully "discard" record type parameters in implemented classes? First, I thought about using java.lang.Void class as type parameter but then I noticed that only subclasses of Record are allowed... The reason is that I don't need record types at all because I plan to use generated by JOOQ SQL queries in something like Spring JdbcTemplate so mapping is done by myself.
Thanks in advance for any help!
Given your use-case, I'm not sure why you'd like to roll your own Table and TableField implementations rather than using the ones generated by jOOQ. As you stated yourself, you don't have to regenerate that code every time the DB schema changes. Many users will just generate the schema once in a while and then put the generated artefacts under version control. This will help you keep track of newly added changes.
To answer your questions:
Yes, there are some examples around the use of CustomTable. You may also find some people sharing similar experiences on the user group
Yes you can just use Record. Your minimal custom table type would then be:
class X extends TableImpl<Record> {
public X() {
super("x");
}
}
Note that you will be using jOOQ's internal API (TableImpl), which is not officially supported. While I'm positive that it'll work, it might break in the future, e.g. as the super constructor signature might change.
jOOQ runs on generated classes (from database schema). In my apps, there are many tables created while excuting. Then how can I query them using jOOQ? Or can I use something else to do this?
We just use jOOQ to generate SQL, not doing any CRUD operation.
While being the main use case for using jOOQ, you do not have to generate any source code with jOOQ. Several users have used jOOQ the way you do, by supplying dynamic table/column names at runtime to Factory methods such as:
DSL.tableByName(String...)
DSL.fieldByName(String...)
These methods take care of correctly escaping your schema, table, field, alias names, etc, in order to guarantee SQL syntax correctness and SQL injection protection.
Some related discussions on the jOOQ User Group are here:
Escaping SQL identifiers
Intantiating Field and Table by name
Specifying the alias name for a JOIN