I want to search in 16 different tables, but I don't wanna repeat the "select from DB" 16 times; I think that's not really help in performance!!!
I am using:
query="SELECT * FROM table1, table2,..., table16 WHERE id=?";
Is it correct ??
my problem is how to separate between data of tables ??
also maybe I can get from one table two or more results for one "id"; So I want to know which data is from which table !!
.
Best regards,
Your query will not work, because you are trying to join those multiple tables, whereas what you want to do is search (filter) those 16 tables.
You could use a union all to do this in a single query:
select xxx, 'table1' as source_table
from table1
where id = ?
union all
select xxx, 'table2' as source_table
from table2
where id = ?
and so on. The second derived field source_table can be used to determine which table returned which result.
You have to list all fields using aliases for fields with same name, and prefix with table names.
For example :
query = "SELECT table1.id as id_1, table2.id as id_2, ... WHERE id_1 = 23"
Probably a very long query to write, but you have solution to generate and paste it : You can do this for example with FlySpeed SqlQuery (free for personal use)
FlySpeed SqlQuery will generate all aliases for you, and automatically prefix with table names.
A little clarification would help. If all 16 tables have the same fields and you want them in a continuous list, you can use UNION as suggested above. On the other hand, if there are only a few fields that match and you want to compare the values for each table side-by-side, you'll want to use joins and provide aliases with the table names, as also suggested above.
However, looking at the snippet of code you've provided, I'm going to guess that you're either building some kind of stored procedure or else implementing SQL in some other language. If that's the case, how about loading your table names into an array and using a for loop to build the query, such as the following psuedo-code:
tableList = ["table1", "table2"...]
fieldnames = ["field1", "field2"...]
query = "SELECT "
for i = 0 to count(tableList):
for j = 0 to count(fieldnames):
query = query + tablelist[i] + "." + fieldnames[j] + ", "
j++
i++
query = query + "FROM "
for i = 0 to count(tableList):
query = query + tableList[i] + ", "
i++
query = query + "WHERE " ...
And so forth. Much of this depends on what exactly you're looking to do, how often you're looking to do it, and how often the variables (like which tables or fields you're using) are going to change.
Related
I want to get the number of results of a query in Spring Data Jpa, using a non-native #Query method. It consists of a basic group by plus a having clause.
My plain query looks like this (analogous example):
select count(*) from (
select 1 from table t
where t.field_a = 1
group by t.id
having count(*) = 2) a;
Since Hibernate 5 does not allow subqueries in the form clause, I have to find a workaround for that. The only one I found is very inefficient as per the query plan:
select count(*) from table t
where t.field_a = 1 and
2 = (select count(*) from table temp where temp.id = t.id);
Is there a way to write a Spring Data JPA query that's as efficient as the first one? I can think of no solution rather than selecting the inner query and taking its size() in java, but that can produce issues due to a ton of redundant data passing through the network.
There is no easy solution to count the results of a subquery in JPA but the a workaround is proposed here https://arjan-tijms.omnifaces.org/2012/06/counting-rows-returned-from-jpa-query.html.
The principle is to build a native query based on the initial Jpa subselect query.
This does the job if you accept to count the elements in java !
Query q = em.createQuery(
"select 1 from table t where field_a = 1 " +
"group by t.id having count(*) = 2");
int count = q.getResultList().size();
(performances depending on the number of lines returned, but the projection is very light : 1)
What alternatives do I have to implement a union query using hibernate? I know hibernate does not support union queries at the moment, right now the only way I see to make a union is to use a view table.
The other option is to use plain jdbc, but this way I would loose all my example/criteria queries goodies, as well as the hibernate mapping validation that hibernate performs against the tables/columns.
You could use id in (select id from ...) or id in (select id from ...)
e.g. instead of non-working
from Person p where p.name="Joe"
union
from Person p join p.children c where c.name="Joe"
you could do
from Person p
where p.id in (select p1.id from Person p1 where p1.name="Joe")
or p.id in (select p2.id from Person p2 join p2.children c where c.name="Joe");
At least using MySQL, you will run into performance problems with it later, though. It's sometimes easier to do a poor man's join on two queries instead:
// use set for uniqueness
Set<Person> people = new HashSet<Person>((List<Person>) query1.list());
people.addAll((List<Person>) query2.list());
return new ArrayList<Person>(people);
It's often better to do two simple queries than one complex one.
EDIT:
to give an example, here is the EXPLAIN output of the resulting MySQL query from the subselect solution:
mysql> explain
select p.* from PERSON p
where p.id in (select p1.id from PERSON p1 where p1.name = "Joe")
or p.id in (select p2.id from PERSON p2
join CHILDREN c on p2.id = c.parent where c.name="Joe") \G
*************************** 1. row ***************************
id: 1
select_type: PRIMARY
table: a
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 247554
Extra: Using where
*************************** 2. row ***************************
id: 3
select_type: DEPENDENT SUBQUERY
table: NULL
type: NULL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: NULL
Extra: Impossible WHERE noticed after reading const tables
*************************** 3. row ***************************
id: 2
select_type: DEPENDENT SUBQUERY
table: a1
type: unique_subquery
possible_keys: PRIMARY,name,sortname
key: PRIMARY
key_len: 4
ref: func
rows: 1
Extra: Using where
3 rows in set (0.00 sec)
Most importantly, 1. row doesn't use any index and considers 200k+ rows. Bad! Execution of this query took 0.7s wheres both subqueries are in the milliseconds.
Use VIEW. The same classes can be mapped to different tables/views using entity name, so you won't even have much of a duplication. Being there, done that, works OK.
Plain JDBC has another hidden problem: it's unaware of Hibernate session cache, so if something got cached till the end of the transaction and not flushed from Hibernate session, JDBC query won't find it. Could be very puzzling sometimes.
I have to agree with Vladimir. I too looked into using UNION in HQL and couldn't find a way around it. The odd thing was that I could find (in the Hibernate FAQ) that UNION is unsupported, bug reports pertaining to UNION marked 'fixed', newsgroups of people saying that the statements would be truncated at UNION, and other newsgroups of people reporting it works fine...
After a day of mucking with it, I ended up porting my HQL back to plain SQL, but doing it in a View in the database would be a good option. In my case, parts of the query were dynamically generated, so I had to build the SQL in the code instead.
I have a solution for one critical scenario (for which I struggled a lot )with union in HQL .
e.g. Instead of not working :-
select i , j from A a , (select i , j from B union select i , j from C) d where a.i = d.i
OR
select i , j from A a JOIN (select i , j from B union select i , j from C) d on a.i = d.i
YOU could do in Hibernate HQL ->
Query q1 =session.createQuery(select i , j from A a JOIN B b on a.i = b.i)
List l1 = q1.list();
Query q2 = session.createQuery(select i , j from A a JOIN C b on a.i = b.i)
List l2 = q2.list();
then u can add both list ->
l1.addAll(l2);
A view is a better approach but since hql typically returns a List or Set... you can do list_1.addAll(list_2). Totally sucks compared to a union but should work.
Perhaps I had a more straight-forward problem to solve. My 'for instance' was in JPA with Hibernate as the JPA provider.
I split the three selects (two in a second case) into multiple select and combined the collections returned myself, effectively replacing a 'union all'.
Hibernate 6 added support for UNION.
So, you can now use UNION in JPQL queries like this:
List<String> topics = entityManager.createQuery("""
select c.name as name
from Category c
union
select t.name as name
from Tag t
""", String.class)
.getResultList();
And you can also also use UNION ALL if there are no duplicates to be removed:
List<String> topics = entityManager.createQuery("""
select c.name as name
from Category c
union all
select t.name as name
from Tag t
""", String.class)
.getResultList();
Besides UNION, you can also use EXCEPT and INTERSECT.
I too have been through this pain - if the query is dynamically generated (e.g. Hibernate Criteria) then I couldn't find a practical way to do it.
The good news for me was that I was only investigating union to solve a performance problem when using an 'or' in an Oracle database.
The solution Patrick posted (combining the results programmatically using a set) while ugly (especially since I wanted to do results paging as well) was adequate for me.
Here is a special case, but might inspire you to create your own work around. The goal here is to count the total number of records from two different tables where records meet a particular criteria. I believe this technique will work for any case where you need to aggregate data from across multiple tables/sources.
I have some special intermediate classes setup, so the code which calls the named query is short and sweet, but you can use whatever method you normally use in conjunction with named queries to execute your query.
QueryParms parms=new QueryParms();
parms.put("PROCDATE",PROCDATE);
Long pixelAll = ((SourceCount)Fetch.row("PIXEL_ALL",parms,logger)).getCOUNT();
As you can see here, the named query begins to look an aweful lot like a union statement:
#Entity
#NamedQueries({
#NamedQuery(
name ="PIXEL_ALL",
query = "" +
" SELECT new SourceCount(" +
" (select count(a) from PIXEL_LOG_CURR1 a " +
" where to_char(a.TIMESTAMP, 'YYYYMMDD') = :PROCDATE " +
" )," +
" (select count(b) from PIXEL_LOG_CURR2 b" +
" where to_char(b.TIMESTAMP, 'YYYYMMDD') = :PROCDATE " +
" )" +
") from Dual1" +
""
)
})
public class SourceCount {
#Id
private Long COUNT;
public SourceCount(Long COUNT1, Long COUNT2) {
this.COUNT = COUNT1+COUNT2;
}
public Long getCOUNT() {
return COUNT;
}
public void setCOUNT(Long COUNT) {
this.COUNT = COUNT;
}
}
Part of the magic here is to create a dummy table and insert one record into it. In my case, I named it dual1 because my database is Oracle, but I don't think it matters what you call the dummy table.
#Entity
#Table(name="DUAL1")
public class Dual1 {
#Id
Long ID;
}
Don't forget to insert your dummy record:
SQL> insert into dual1 values (1);
As Patrick said, appending the LISTs from each SELECT would be a good idea but remember that it acts like UNION ALL. To avoid this side effect, just control if the object is already added in final collection or not. If no, then add it.
Something else that you should care about is that if you have any JOIN in each SELECT, the result would be a list of object array(List<Object[]>) so you have to iterate over it to only keep the object that you need.
Hope it works.
I'm trying to loop through multiple sql queries that are executed. I want to first get all the question information for a certain task and then get the keywords for that question. I have three records in my Questions table, but when the while loop at the end of list.add(keyword); is done, it jumps to the SELECT Questions.Question loop (as it should) and then just jumps out and gives me only one record and not the other 2.
What am I doing wrong? Can someone maybe help me fix my code? I've thought of doing batch sql executes (maybe that is the solution), but within each while loop, I need information from the previous sql statement, so I can't just do it all at the end of the batch.
SQL Code:
String TaskTopic = eElement.getElementsByTagName("TaskTopic").item(0).getTextContent();
// perform query on database and retrieve results
String sql = "SELECT Tasks.TaskNo FROM Tasks WHERE Tasks.TaskTopic = '" + TaskTopic + "';";
System.out.println(" Performing query, sql = " + sql);
result = stmt.executeQuery(sql);
Document doc2 = x.createDoc();
Element feedback = doc2.createElement("Results");
while (result.next())
{
String TaskNo = result.getString("TaskNo");
// perform query on database and retrieve results
String sqlquery = "SELECT Questions.Question, Questions.Answer, Questions.AverageRating, Questions.AverageRating\n" +
"FROM Questions\n" +
"INNER JOIN TaskQuestions ON TaskQuestions.QuestionID = Questions.QuestionID \n" +
"INNER JOIN Tasks ON Tasks.TaskNo = '" + TaskNo + "';";
result = stmt.executeQuery(sqlquery);
while (result.next())
{
String Question = result.getString("Question");
String Answer = result.getString("Answer");
String AverageRating = result.getString("AverageRating");
String sqlID = "SELECT QuestionID FROM Questions WHERE Question = '" + Question + "';";
result = stmt.executeQuery(sqlID);
while (result.next())
{
String ID = result.getString("QuestionID");
String sqlKeywords = "SELECT Keyword FROM LinkedTo WHERE QuestionID = '" + ID + "';";
result = stmt.executeQuery(sqlKeywords);
while (result.next())
{
String keyword = result.getString("Keyword");
list.add(keyword);
}
}
feedback.appendChild(x.CreateQuestionKeyword(doc2, Question, Answer, AverageRating, list));
}
}
Why this should be done in SQL
Creating loops is exponentially less efficient than writing a sql query. Sql is built to pull back this type of data and can plan out how it is going to get this data from the database (called an execution plan).
Allowing Sql to do its job and determine the best way to pull back the data instead of explicitly determining what tables you are going to use first and then calling them one at a time is better in terms of the amount of resources you will use, how much time it will take to get the results, code readability, and maintainability in the future.
What information you are looking for
In the psuedocode you provided, you are using the Keyword, Question, Answer, and AnswerRating values. Finding these values should be the focus of the sql query. Based on the code you have written, Question, Answer, and AnswerRating are coming from the Questions table and Keyword is coming from the LinkedTo table, so both of these tables should be available to have data pulled from them.
You can note at this point that we have essentially just mapped out what the Select and From portions of your query should look like.
It also looks like you have a parameter called TaskTopic so we need to include the table Tasks to make sure the correct data is returned. Lastly, the TaskQuestions table is the link between the tasks and the questions. Now that we know what the query should look like, let's see what the results are using sql syntax.
The Code
You did not include the declaration of stmt, but I assume that it is a PreparedStatement. You can add parameters to a prepared statement. Notice the ? in the sql code? The parameters you provide will be added in place of the ?. To do this, you should use stmt.setString(1, TaskTopic);. Note that if there were more than one parameter, you would need to add them in the order that they exists in the sql query (using 1, 2, ...)
SELECT l.Keyword,
q.Question,
q.Answer,
q.AverageRating
FROM LinkedTo l Inner Join
Questions q
on l.questionID = q.QuestionID
Where exists ( Select 1
From TaskQuestions tq INNER JOIN
Tasks t
on tq.TaskNo = t.TaskNo
Where t.TaskTopic = ?
and tq.QuestionID = q.QuestionID)
This is one way that you can write the query to return the same results. There are other ways to write this to get what you are looking for.
What's Going On?
There are a few things in this query you may not be familiar with. First are table aliases. Instead of writing the table name over and over again, you can alias your tables. I used the letter q to represent the Questions table. Any time you see q. you should recognize that I am referring to a column from Questions. The q after Questions is what gives the table its alias.
Exists Instead of doing a bunch of inner joins with tables that you are not selecting information from, you can use an exists to check if what you are looking for is in those tables. You can continue to do inner joins if you need data from the tables, but if you don't, Exists is more efficient.
I suspect you had issues with the query before (and probably the one you provided) because you did not provide any information to join TaskQuestions and Tasks together. That most likely resulted in the duplicates. I joined on TaskNo but this may not be the correct column depending on how the tables are set up.
My database table(geo ip lookup) is having 7 columns,of which 2 columns constitute < composite-id>.
Now when i lookup for a value using first 2 coloumns it takes me 12-14 seconds to fetch a record..
My DAO code looks like this:
String queryString = "from Igeo igeo where igeo.ip_from <= " + ip
+ "and igeo.ip_to >= " + ip;
Query q = session.createQuery(queryString);
List<Igeo> igeoList = q.list();
if(igeoList.size() > 0){
Igeo igeo = igeoList.get(0);
ISP = igeo.getIsp();
...
...
}
*Igeo = class in java represnting table
**Record is fetched when ip lies between values of composite-id columns eg.
ip_from = 1 ; ip_to = 3 ; ip = 2;
so above row will be returned
This table is only used to read records ,please suggest me a queryString which is more efficient than above
First remove hibernate and run your query in a query browser and see how long it takes to return. If it takes the same amount of time it's not Hibernate. It's the performance of the database. Make sure you add indexes onto the two columns ip_from and ip_to. You can also execute a query plan on your query to see what the database is running under the hood and try and optimize the query plan.
I would suggest NOT using concatenation onto your query as you are. That produces a security hole allowing potential SQL injection from outside parties. It's better to use the following:
Query q = session.createQuery("from Igeo igeo where igeo.from_ip >= ? and igeo.to_ip <= ?");
q.setString( 0, ip );
q.setString( 1, ip );
You could also used named parameters which might shorten it up a bit more.
If the table IGeo does not contain overlapping ranges of ip_from and ip_to, you might try this
String queryString = "FROM Igeo igeo"
+ " WHERE igeo.ip_to >= " + ip
+ " ORDER BY igeo.ip_to";
Then check the first item in the list (to see that ip_from <= ip).
Even if the table could contain overlapping ranges of ip_from, ip_to, I bet the above HQL will run faster.
<Aside> You really should not concatenate a raw string like "ip" into HQL or SQL. It leads to SQL Injection Attack vulnerabilities. Use a query parameter instead</Aside>
Also, verify that your database has an index on the column corresponding to Igeo.ip_to.
Sounds to me from your description, that the database has a primary key of Igeo.ip_from + IGeo.ip_to. If the values of ip_from and ip_to are not overlapping, that does not seem to be normalized. You should need only a single column for the primary key. If you have chosen to use both columns as a primary key, the above query will benefit by adding a single index.
Some databases will perform better if you add an index containing all the columns in the table, starting with ip_to and ip_from. (This enables the database to satisfy the query by accessing only the index). Not sure if MySQL can optimize to this extent, but I know DB2 and Oracle will provide this.
I am trying to build a dynamic sql query in java (shown below)
sqlStr = "Select * " +
"from " + tableName
if(tableName!=null){
if(tableName.equals("Table1"){
sqlStr = sqlStr.concat("order by city desc");
}else if(tableName.equals("Table2"){
sqlStr = sqlStr.concat("order by country desc");
}else if(tableName.equals("Table3"){
sqlStr = sqlStr.concat("order by price desc");
}
}
Now what i would like to do is to add a final 'else' statement which would order the query based on whether the table contains a column named 'custID'. There will be several tables with that column so i want to sort the ones that have that column by custID. (Rather than having hundreds of additional if statements for each table that does have that column name.) Is this possible? i have seen people using the 'decode' function but i cant work out how to use it here.
Use DatabaseMetaData to get the table information.
You can use the getTablexxx() and getColumnxx() methods to get the table information.
Connection conn = DriverManager.getConnection(.....);
DatabaseMetaData dbmd = conn.getMetaData();
dbmd.getxxxx();
Note: you forgot space in your code before ORDER BY clause.
If you are happy with hardcoding things, a way to avoid multiple conditionals would be to store a list of all the tables that include custID.
private final static String tablesWithCustID = "/TableX/TableY/TableZ/";
...
if (tablesWithCustID.contains( tableName )) {
sqlStr = sqlStr.concat("order by custID")
}
You could use a List instead of a simple delimited string if you like.
Perhaps better, you could store a map with table names as the key, and the sort string as the value. Load it up once, then you don't need any conditional at all.
The most straight-forward way to do it is to read the column definitions from USER_TAB_COLUMNS or ALL_TAB_COLUMNS and check for the existence of a custID column. Without crazy PL/SQL tricks, you won't be able to solve this in SQL alone.
BTW, there is a " " missing between tableName and the order by clauses.
I understand that you're looking for a solution that can do this in one query, i.e. without running a separate metadata query beforehand.
Unfortunately, this won't be possible. The decode function can do some dynamic things with column values, but not with column name. And you're looking for a solution dynamically derive the column name.
An alternative might be to just add ORDER BY 1, 2. This is an old syntax that means order by the first and than by the second column. It might be a good solution if the custID column is the first column. Otherwise it at least gives you some sorting.
How about ArrayList.contains()?
You can create a list of tables which have that column, and just check for tables.contains(tablename) in the final if condition.