I'm currently working on a project in which I want to parse an SQL-Query and create the tree of relational expressions for it.
The main goal is to identify the join partners in the Queries.
So I'd like to push the join-expressions down to the leafs of the tree.
To achieve this, I decided to use Apache Calcite.
My code looks roughly like this at the moment:
Planner planner = Frameworks.getPlanner(Frameworks.newConfigBuilder().defaultSchema(Frameworks.createRootSchema(false)).build());
SqlNode parsed = planner.parse("SELECT s.dnasamplename, e.Total_expression_level, e.Soluble_expression_level, s.id " +
"FROM table1 e" +
"JOIN table2 s on s.constructname = e.Clone_name" +
"WHERE e.Total_expression_level like '0:%'");
planner.validate(parsed);
RelRoot relRoot = planner.rel(parsed);
If I try to execute the rel-Method of Planner directly, I get the error that I have to validate the query first. And that`s the point where I face troubles. I don't have a valid Schema. I just want to convert the query to a relational query plan. Is there a way to kind of skip the validation process?
It's not possible to skip the validation process. The sql-to-relational converter relies on state that has previously been created by the validator.
People see that the validator throws an exception if the SQL is invalid and imagine that this is its only purpose; but building the state is equally important. The state includes of a map of the type of each SqlNode, expansions of "*" and "tableAlias.*" in SELECT clauses, and fully-qualified identifiers.
If your schema is missing or incomplete, you might try a similar approach to Apache Drill. If the validator asks for a particular column of a table, and the table does not have that column, the Calcite namespace for that table says "yes, I have that column!" and adds it. Thus, the validation process succeeds, and each namespace has a list of the extra columns needed by the query.
Problem: Report Templates are created by Admin Users who only decide what data to show where as the filter for the data is specified by the business user. In simpler SQL terms, query is specified by the Admin User, Business User specifies the WHERE clause.
Jasper allows user to specify parameters in SQL query like $P{city}. I have tried to retrieve the query dynamically using the method specfied in the link.
Possible Solution can be
Use WHERE clause parameters in JRXML and replace them while report creation - This will save me SQL parsing but I don't want to guide the admin user with this complexity. Parsing is not a huge problem.
Use my custom jdbc query executor and factory, only created to allow me extension point before jasper fire SQL query. I will be completely relying on vanilla Jasper JDBC data source but will only modify query before execution. JRAbstractQueryExecuter simplifies the query and replace the jasper related tokens before firing query - This will be very dirty and will force me to be implementation specific.
Do the same replacement as it is done in JRAbstractQueryExecuter in my application code base, parse the SQL query, modify it and set it again as specified in link
Can you please suggest a better way of doing this? I have a feeling that this can definitly be done in cleaner way.
Thanks for your help
You could create an input control to determine the desired WHERE clause and use a parameter to hold the contents of that WHERE clause. The default value expression would be something like:
$P{theParameter} == "value_1" ?
(" AND CONDITION_1 IN ('A', 'B', 'C') AND CONDITION_2 = 'Yes' "
) : " AND CONDITION_3 = 'Other' AND CONDITION_4 = 'No' "
Then in your WHERE clause you would reference it like:
WHERE
.... = .....
AND .... = ....
AND .... = ....
$P!{theParameter}
If your constraint columns are the same across your WHERE clauses, you could use $P! to bring in the parameter value literally, and reference it in your query:
WHERE
.... = .....
AND .... = ....
AND .... = ....
AND thisValue = $P!{theParameter}
I have a xml file and I want to store it contents in DB.How The format of my xml is not fixed 100%
I have xml file as shown below
<Jb>
<T1>false</T1>
<T2>
<email>dshetyo#adobe.com</email>
<userName>passwd</clusterName>
<passwd>adobe</clusterURL>
<url>dummy</url>
<param1>val </paramY>
<paramN> val </paramN>
</t2>
<fx>
<a >351108</a>
<b >4629 <b>
</fx >
</Jb>
I can have N params eg param1,param2 etc (N not fixed)How can i map it to a relatioanl table structure?
Any other approach?
This is a one to many relationship. You'll want a table like so:
account
id
email
usernmae
parameters
account_id
param_key
param_value
And then your SQL looks roughly like:
select * from account, parameters where account.id = parameters.account_id
Alternatively, you can use a single column to store all of your key value pairs as well. You can do this as a long string, or a clob (depending on what you think is more appropriate performance-wise and what your needs are in terms of size)
(so there would additional parsing necessary, but if you're not looking to select parameters via SQL, this is the way to go)
Usually, XML content is stored in a database in a single TEXT or VARCHAR column.
Because of its free-form structure, it is generally not suitable for O/R mapping.
One option is to insert the XML directly into the table, then use XPath commands to extract the fields on query. XPath is supported by the major databases: MySQL, Oracle, Postgres...
Oracle has XMLDB, their implementation of the XPath functions for searching based on XML. This means that you can define a column as being an XMLType data type, and attach/map schema to this (This aids in validating as well as evolving the XML over time).
XML in MySQL:
http://dev.mysql.com/tech-resources/articles/xml-in-mysql5.1-6.0.html
MySQL appear to support XPath functions as previously mentioned:
http://dev.mysql.com/doc/refman/5.6/en/xml-functions.html
But note, these features are still under development in MySQL.
I want to build an SQL string to do database manipulation (updates, deletes, inserts, selects, that sort of thing) - instead of the awful string concat method using millions of "+"'s and quotes which is unreadable at best - there must be a better way.
I did think of using MessageFormat - but its supposed to be used for user messages, although I think it would do a reasonable job - but I guess there should be something more aligned to SQL type operations in the java sql libraries.
Would Groovy be any good?
First of all consider using query parameters in prepared statements:
PreparedStatement stm = c.prepareStatement("UPDATE user_table SET name=? WHERE id=?");
stm.setString(1, "the name");
stm.setInt(2, 345);
stm.executeUpdate();
The other thing that can be done is to keep all queries in properties file. For example
in a queries.properties file can place the above query:
update_query=UPDATE user_table SET name=? WHERE id=?
Then with the help of a simple utility class:
public class Queries {
private static final String propFileName = "queries.properties";
private static Properties props;
public static Properties getQueries() throws SQLException {
InputStream is =
Queries.class.getResourceAsStream("/" + propFileName);
if (is == null){
throw new SQLException("Unable to load property file: " + propFileName);
}
//singleton
if(props == null){
props = new Properties();
try {
props.load(is);
} catch (IOException e) {
throw new SQLException("Unable to load property file: " + propFileName + "\n" + e.getMessage());
}
}
return props;
}
public static String getQuery(String query) throws SQLException{
return getQueries().getProperty(query);
}
}
you might use your queries as follows:
PreparedStatement stm = c.prepareStatement(Queries.getQuery("update_query"));
This is a rather simple solution, but works well.
For arbitrary SQL, use jOOQ. jOOQ currently supports SELECT, INSERT, UPDATE, DELETE, TRUNCATE, and MERGE. You can create SQL like this:
String sql1 = DSL.using(SQLDialect.MYSQL)
.select(A, B, C)
.from(MY_TABLE)
.where(A.equal(5))
.and(B.greaterThan(8))
.getSQL();
String sql2 = DSL.using(SQLDialect.MYSQL)
.insertInto(MY_TABLE)
.values(A, 1)
.values(B, 2)
.getSQL();
String sql3 = DSL.using(SQLDialect.MYSQL)
.update(MY_TABLE)
.set(A, 1)
.set(B, 2)
.where(C.greaterThan(5))
.getSQL();
Instead of obtaining the SQL string, you could also just execute it, using jOOQ. See
http://www.jooq.org
(Disclaimer: I work for the company behind jOOQ)
One technology you should consider is SQLJ - a way to embed SQL statements directly in Java. As a simple example, you might have the following in a file called TestQueries.sqlj:
public class TestQueries
{
public String getUsername(int id)
{
String username;
#sql
{
select username into :username
from users
where pkey = :id
};
return username;
}
}
There is an additional precompile step which takes your .sqlj files and translates them into pure Java - in short, it looks for the special blocks delimited with
#sql
{
...
}
and turns them into JDBC calls. There are several key benefits to using SQLJ:
completely abstracts away the JDBC layer - programmers only need to think about Java and SQL
the translator can be made to check your queries for syntax etc. against the database at compile time
ability to directly bind Java variables in queries using the ":" prefix
There are implementations of the translator around for most of the major database vendors, so you should be able to find everything you need easily.
I am wondering if you are after something like Squiggle (GitHub). Also something very useful is jDBI. It won't help you with the queries though.
I would have a look at Spring JDBC. I use it whenever I need to execute SQLs programatically. Example:
int countOfActorsNamedJoe
= jdbcTemplate.queryForInt("select count(0) from t_actors where first_name = ?", new Object[]{"Joe"});
It's really great for any kind of sql execution, especially querying; it will help you map resultsets to objects, without adding the complexity of a complete ORM.
I tend to use Spring's Named JDBC Parameters so I can write a standard string like "select * from blah where colX=':someValue'"; I think that's pretty readable.
An alternative would be to supply the string in a separate .sql file and read the contents in using a utility method.
Oh, also worth having a look at Squill: https://squill.dev.java.net/docs/tutorial.html
I second the recommendations for using an ORM like Hibernate. However, there are certainly situations where that doesn't work, so I'll take this opportunity to tout some stuff that i've helped to write: SqlBuilder is a java library for dynamically building sql statements using the "builder" style. it's fairly powerful and fairly flexible.
I have been working on a Java servlet application that needs to construct very dynamic SQL statements for adhoc reporting purposes. The basic function of the app is to feed a bunch of named HTTP request parameters into a pre-coded query, and generate a nicely formatted table of output. I used Spring MVC and the dependency injection framework to store all of my SQL queries in XML files and load them into the reporting application, along with the table formatting information. Eventually, the reporting requirements became more complicated than the capabilities of the existing parameter mapping frameworks and I had to write my own. It was an interesting exercise in development and produced a framework for parameter mapping much more robust than anything else I could find.
The new parameter mappings looked as such:
select app.name as "App",
${optional(" app.owner as "Owner", "):showOwner}
sv.name as "Server", sum(act.trans_ct) as "Trans"
from activity_records act, servers sv, applications app
where act.server_id = sv.id
and act.app_id = app.id
and sv.id = ${integer(0,50):serverId}
and app.id in ${integerList(50):appId}
group by app.name, ${optional(" app.owner, "):showOwner} sv.name
order by app.name, sv.name
The beauty of the resulting framework was that it could process HTTP request parameters directly into the query with proper type checking and limit checking. No extra mappings required for input validation. In the example query above, the parameter named serverId
would be checked to make sure it could cast to an integer and was in the range of 0-50. The parameter appId would be processed as an array of integers, with a length limit of 50. If the field showOwner is present and set to "true", the bits of SQL in the quotes will be added to the generated query for the optional field mappings. field Several more parameter type mappings are available including optional segments of SQL with further parameter mappings. It allows for as complex of a query mapping as the developer can come up with. It even has controls in the report configuration to determine whether a given query will have the final mappings via a PreparedStatement or simply ran as a pre-built query.
For the sample Http request values:
showOwner: true
serverId: 20
appId: 1,2,3,5,7,11,13
It would produce the following SQL:
select app.name as "App",
app.owner as "Owner",
sv.name as "Server", sum(act.trans_ct) as "Trans"
from activity_records act, servers sv, applications app
where act.server_id = sv.id
and act.app_id = app.id
and sv.id = 20
and app.id in (1,2,3,5,7,11,13)
group by app.name, app.owner, sv.name
order by app.name, sv.name
I really think that Spring or Hibernate or one of those frameworks should offer a more robust mapping mechanism that verifies types, allows for complex data types like arrays and other such features. I wrote my engine for only my purposes, it isn't quite read for general release. It only works with Oracle queries at the moment and all of the code belongs to a big corporation. Someday I may take my ideas and build a new open source framework, but I'm hoping one of the existing big players will take up the challenge.
Why do you want to generate all the sql by hand? Have you looked at an ORM like Hibernate Depending on your project it will probably do at least 95% of what you need, do it in a cleaner way then raw SQL, and if you need to get the last bit of performance you can create the SQL queries that need to be hand tuned.
You can also have a look at MyBatis (www.mybatis.org) . It helps you write SQL statements outside your java code and maps the sql results into your java objects among other things.
Google provides a library called the Room Persitence Library which provides a very clean way of writing SQL for Android Apps, basically an abstraction layer over underlying SQLite Database. Bellow is short code snippet from the official website:
#Dao
public interface UserDao {
#Query("SELECT * FROM user")
List<User> getAll();
#Query("SELECT * FROM user WHERE uid IN (:userIds)")
List<User> loadAllByIds(int[] userIds);
#Query("SELECT * FROM user WHERE first_name LIKE :first AND "
+ "last_name LIKE :last LIMIT 1")
User findByName(String first, String last);
#Insert
void insertAll(User... users);
#Delete
void delete(User user);
}
There are more examples and better documentation in the official docs for the library.
There is also one called MentaBean which is a Java ORM. It has nice features and seems to be pretty simple way of writing SQL.
Read an XML file.
You can read it from an XML file. Its easy to maintain and work with.
There are standard STaX, DOM, SAX parsers available out there to make it few lines of code in java.
Do more with attributes
You can have some semantic information with attributes on the tag to help do more with the SQL. This can be the method name or query type or anything that helps you code less.
Maintaince
You can put the xml outside the jar and easily maintain it. Same benefits as a properties file.
Conversion
XML is extensible and easily convertible to other formats.
Use Case
Metamug uses xml to configure REST resource files with sql.
If you put the SQL strings in a properties file and then read that in you can keep the SQL strings in a plain text file.
That doesn't solve the SQL type issues, but at least it makes copying&pasting from TOAD or sqlplus much easier.
How do you get string concatenation, aside from long SQL strings in PreparedStatements (that you could easily provide in a text file and load as a resource anyway) that you break over several lines?
You aren't creating SQL strings directly are you? That's the biggest no-no in programming. Please use PreparedStatements, and supply the data as parameters. It reduces the chance of SQL Injection vastly.
Can anyone suggest a good way of detecting if a database is empty from Java (needs to support at least Microsoft SQL Server, Derby and Oracle)?
By empty I mean in the state it would be if the database were freshly created with a new create database statement, though the check need not be 100% perfect if covers 99% of cases.
My first thought was to do something like this...
tables = metadata.getTables(null, null, null, null);
Boolean isEmpty = !tables.next();
return isEmpty;
...but unfortunately that gives me a bunch of underlying system tables (at least in Microsoft SQL Server).
There are some cross-database SQL-92 schema query standards - mileage for this of course varies according to vendor
SELECT COUNT(*) FROM [INFORMATION_SCHEMA].[TABLES] WHERE [TABLE_TYPE] = <tabletype>
Support for these varies by vendor, as does the content of the columns for the Tables view. SQL implementation of Information Schema docs found here:
http://msdn.microsoft.com/en-us/library/aa933204(SQL.80).aspx
More specifically in SQL Server, sysobjects metadata predates the SQL92 standards initiative.
SELECT COUNT(*) FROM [sysobjects] WHERE [type] = 'U'
Query above returns the count of User tables in the database. More information about the sysobjects table here:
http://msdn.microsoft.com/en-us/library/aa260447(SQL.80).aspx
I don't know if this is a complete solution ... but you can determine if a table is a system table by reading the table_type column of the ResultSet returned by getTables:
int nonSystemTableCount = 0;
tables = metadata.getTables(null, null, null, null);
while( tables.next () ) {
if( !"SYSTEM TABLE".equals( tables.getString( "table_type" ) ) ) {
nonSystemTableCount++;
}
}
boolean isEmpty = nonSystemTableCount == 0;
return isEmpty;
In practice ... I think you might have to work pretty hard to get a really reliable, truly generic solution.
Are you always checking databases created in the same way? If so you might be able to simply select from a subset of tables that you are familiar with to look for data.
You also might need to be concerned about static data perhaps added to a lookup table that looks like 'data' from a cursory glance, but might in fact not really be 'data' in an interesting sense of the term.
Can you provide any more information about the specific problem you are trying to tackle? I wonder if with more data a simpler and more reliable answer might be provided.
Are you creating these databases?
Are you creating them with roughly the same constructor each time?
What kind of process leaves these guys hanging around, and can that constructor destruct?
There is certainly a meta data process to loop through tables, just through something a little more custom might exist.
In Oracle, at least, you can select from USER_TABLES to exclude any system tables.
I could not find a standard generic solution, so each database needs its own tests set.
For Oracle for instance, I used to check tables, sequences and indexes:
select count(*) from user_tables
select count(*) from user_sequences
select count(*) from user_indexes
For SqlServer I used to check tables, views and stored procedures:
SELECT * FROM sys.all_objects where type_desc in ('USER_TABLE', 'SQL_STORED_PROCEDURE', 'VIEW')
The best generic (and intuitive) solution I got, is by using ANT SQL task - all I needed to do is passing different parameters for each type of database.
i.e. The ANT build file looks like this:
<project name="run_sql_query" basedir="." default="main">
<!-- run_sql_query: -->
<target name="run_sql_query">
<echo message="=== running sql query from file ${database.src.file}; check the result in ${database.out.file} ==="/>
<sql classpath="${jdbc.jar.file}"
driver="${database.driver.class}"
url="${database.url}"
userid="${database.user}"
password="${database.password}"
src="${database.src.file}"
output="${database.out.file}"
print="yes"/>
</target>
<!-- Main: -->
<target name="main" depends="run_sql_query"/>
</project>
For more details, please refer to ANT:
https://ant.apache.org/manual/Tasks/sql.html