Let’s say I have an album table where partition key is author and sort key is album. Each item also has a price, startDate and endDate attributes. Let say I want to find all the albums that “author=a”, “album=b”, “startDate < c”, “endDate > d” and “price is between e and f”, sorted by price. Is the most efficient way to do that is query on partition key and sort key, and then filter the results on conditions c, d, e and f, and then sort by price? Can secondary index help here? (It seems one secondary index can only be used for query on one or two non-key attributes, but my use case requires < and > operations on multiple non-key attributes and then sorting)
Thanks!
I am working through a similar schema design process.
The short answer is it will depend on exactly how much data you have that falls into the various categories, as well as on the exact QUERIES you hope to run against that data.
The main thing to remember is that you can only ever QUERY based on your Sort Key (where you know the Partition Key) but you ALSO have to maintain uniqueness in order to not overwrite needed data.
A good way to visualize this in your case would be as follows:
Each Artist is Unique (Artist seems to me like a good Partition Key)
Each Artist can have Mutliple Albums making this a good Sort Key (in cases where you will search for an Album for a known Artist)
In the above case your Sort Key is being combined with your Partition Key to create your Hash Key per the following answer (which is worth a read!) to allow you to write a query where you know the artist but only PART of the title.
Ie. here artist = "Pink Floyd" QUERY where string album contains "Moon"
That would match "Pink Floyd" Dark Side of the Moon.
That being said you would only ever have one "Price" for Pink Floyd - Dark Side of the Moon since the Partition Key and Sort Key combine to handle uniqueness. You would overwrite the existing object when you updated the entry with your second price.
So the real question is, what are the best Sort Keys for my use case?
To answer that you need to know what your most frequent QUERIES will be before you build the system.
Price Based Queries?
In your question you mentioned Price attributes in a case where you appear to know the artist and album.
“author=a”, “album=b”, “startDated” and “price is between e and f”, sorted by price"
To me in this case you probably DO NOT know the Artist, or if you do you probably do not know the Album, since you are probably looking to write a Query that returns albums from multiple artists or at least multiple Albums from the same artist.
HOWEVER
That may not be the case if you are creating a database that contains multiple entries (say from multiple vendors selling the same artist / album at different prices). In which case I would say the easiest way to either store only ONE entry for an Artist-Album (partition key) at a given price (sort key) but you would lose all other entries that match that same price for the Artist-Album.
Multiple Queries MAY require Multiple Tables
I had a similar use case and ended up needing to create multiple tables in order to handle my queries. Data is passed / processed from one table and spit out into another one using a Lambda that is triggered on insertion. I then send some queries to one table and some other queries to the initial table.
I have to work with a POJO "Order" that 8 fields and each of these fields is a column in the "order" table. The DB schema is denormalized (and worse, deemed final and unchangeable) so now I have to write a search module that can execute a search with any combination of the above 8 fields.
Are there any approaches on how to do this? Right now I get the input in a new POJO and go through eight IF statements looking for values that are not NULL. Each time I find such a value I add it to the WHERE condition in my SELECT statement.
Is this the best I can hope for? Is it arguably better to select on some minimum of criteria and then iterate over the received collection in memory, only keeping the entries that match the remaining criteria? I can provide pseudo code if that would be useful. Working on Java 1.7, JSF 2.2 and MySQL.
Each time I find such a value I add it to the WHERE condition in my SELECT statement.
This is a prime target for Sql Injection attacks!
Would something like the following work with MySql?
SELECT *
FROM SomeTable
WHERE (#param1 IS NULL OR SomeTable.SomeColumn1 = #param1) OR
(#param2 IS NULL OR SomeTable.SomeColumn2 = #param2) OR
(#param3 IS NULL OR SomeTable.SomeColumn3 = #param3) OR
/* .... */
I have a table which I need to query, then organize the returned objects into two different lists based on a column value. I can either query the table once, retrieving the column by which I would differentiate the objects and arrange them by looping through the result set, or I can query twice with two different conditions and avoid the sorting process. Which method is generally better practice?
MY_TABLE
NAME AGE TYPE
John 25 A
Sarah 30 B
Rick 22 A
Susan 43 B
Either SELECT * FROM MY_TABLE, then sort in code based on returned types, or
SELECT NAME, AGE FROM MY_TABLE WHERE TYPE = 'A' followed by
SELECT NAME, AGE FROM MY_TABLE WHERE TYPE = 'B'
Logically, a DB query from a Java code will be more expensive than a loop within the code because querying the DB involves several steps such as connecting to DB, creating the SQL query, firing the query and getting the results back.
Besides, something can go wrong between firing the first and second query.
With an optimized single query and looping with the code, you can save a lot of time than firing two queries.
In your case, you can sort in the query itself if it helps:
SELECT * FROM MY_TABLE ORDER BY TYPE
In future if there are more types added to your table, you need not fire an additional query to retrieve it.
It is heavily dependant on the context. If each list is really huge, I would let the database to the hard part of the job with 2 queries. At the opposite, in a web application using a farm of application servers and a central database I would use one single query.
For the general use case, IMHO, I will save database resource because it is a current point of congestion and use only only query.
The only objective argument I can find is that the splitting of the list occurs in memory with a hyper simple algorithm and in a single JVM, where each query requires a bit of initialization and may involve disk access or loading of index pages.
In general, one query performs better.
Also, with issuing two queries you can potentially get inconsistent results (which may be fixed with higher transaction isolation level though ).
In any case I believe you still need to iterate through resultset (either directly or by using framework's methods that return collections).
From the database point of view, you optimally have exactly one statement that fetches exactly everything you need and nothing else. Therefore, your first option is better. But don't generalize that answer in way that makes you query more data than needed. It's a common mistake for beginners to select all rows from a table (no where clause) and do the filtering in code instead of letting the database do its job.
It also depends on your dataset volume, for instance if you have a large data set, doing a select * without any condition might take some time, but if you have an index on your 'TYPE' column, then adding a where clause will reduce the time taken to execute the query. If you are dealing with a small data set, then doing a select * followed with your logic in the java code is a better approach
There are four main bottlenecks involved in querying a database.
The query itself - how long the query takes to execute on the server depends on indexes, table sizes etc.
The data volume of the results - there could be hundreds of columns or huge fields and all this data must be serialised and transported across the network to your client.
The processing of the data - java must walk the query results gathering the data it wants.
Maintaining the query - it takes manpower to maintain queries, simple ones cost little but complex ones can be a nightmare.
By careful consideration it should be possible to work out a balance between all four of these factors - it is unlikely that you will get the right answer without doing so.
You can query by two conditions:
SELECT * FROM MY_TABLE WHERE TYPE = 'A' OR TYPE = 'B'
This will do both for you at once, and if you want them sorted, you could do the same, but just add an order by keyword:
SELECT * FROM MY_TABLE WHERE TYPE = 'A' OR TYPE = 'B' ORDER BY TYPE ASC
This will sort the results by type, in ascending order.
EDIT:
I didn't notice that originally you wanted two different lists. In that case, you could just do this query, and then find the index where the type changes from 'A' to 'B' and copy the data into two arrays.
I have 2 entities Book and BookProperty in a one-to-many relationship. I need to retrieve books sorted by upload_date, english language books coming first using JPA Criteria with pagination and stuff (JPQL/native sql is not an option).
This is what the native query for this operation would look like (postgres):
select distinct
b.book_id,
b.name, b.author, ... ,
b.upload_date,
case when p.property_name='language' and lower(p.property_value)='english'
then 0 else 1 end as book_language,
from books b
left outer join book_properties p
on b.book_id=p.book_id
order by book_language asc, b.upload_date desc;
The problem is that I can't get the 'case' part to be selected via criteria API and sort by it. I know that it is possible via a multiselect and a tuple but I would like to avoid that, because really I do not need this column in the application. I would like to just retrieve sorted Book objects and not tuples of (Book, Integer).
I tried to move the case part into the order by and managed to compute the query via JPA Criteria, but in that case setDistinct(true) resulted in an error, because all columns in order by must be part of distinct. So moving the case part to order doesn't look like an option.
Please help me implement this query using JPA Criteria, preferably without using tuples or wrapper objects, but that will do as well if there are no other options.
I have a java object 'star' that consists of two columns, string name (the name of the star) and string List fans (the list of fans of this star). I'd like to persist this class using JPA1 or hibernate. I've done so using the annotation #collectionOfElements on the list. It works fine, and creates two tables.
Now I'd like to get all stars whose fans are 'alice' or 'bob' or 'charlie'. How can I do that in the easiest way (only one query rather than 3, and without using 'OR' statements if possible), using jpa queries (hibernate if it's a must), and without retrieving the whole list of fans ?
Thanks
The following query should help you:
select s.* from star s where s.fans.name in ('alice', 'bob', 'charlie')