I'm writing this on the fly on my phone, so forgive the crappy code samples.
I have entities with a manytomany relationship:
#JoinTable(name="foo", #JoinColum="...", #InverseJoinColumn="...")
#ManyToMany
List list = new ArrayList();
I want their data to be retrieved in a paginated way.
I know about setFirstResult and setMaxResults. Is there a way to use this with the mapping? As in, I retrieve the object and get the list filled with contents equal to the amount of records for a single page, with the appropriate offset.
I guess I'm just unclear of the best way to do this. I could just manually use hibernate criteria to have the effect, but I feel thats missing the API. I have this mapping, I want to see if there's a way to use it in a paginated way.
PS. If this is impractical, just say. Also, if it is, can I still use the mapping to add new entries to the join table. As in, if the entity is a persisted entity in the DB, but I haven't fetched the manytomany list, can I add something new to it and when its persisted with cascade all it'll be added to the join table without clearing the other entries?
The type of the relationship between entities that are part of your query isn't that important. There are a couple of ways to tackle this.
If your database supports the LIMIT keyword in it's queries, you would be able to use it to get data sets, assuming you sort your data. Note that if your data changes while your user is navigating between pages, you might see some duplication or miss some records. You'll be stuck having to rewrite if your database changes to one that doesn't have the LIMIT keyword.
If you need to freeze the data at the point of the original query you need to use a 3rd party framework or write your own to fetch a list of Ids for your query then split up that list and fetch by id in a subset for pagination. This is more reliable can be made to work for any database.
Displaytag is a data paging framework I've used and that I therefore can tell you works well for large datasets. It's also one of the older solutions for this problem and is not part of an extended framework.
http://displaytag.sourceforge.net/11/tut_externalSortAndPage.html
Table sorter is another one I came across. This one uses JQuery and fetches the entire data set in one query, so strictly speaking it doesn't meet your "fetches the data in a paginated way" criteria. (This might not be appropriate for large sets).
http://tablesorter.com/docs/
This tutorial might be helpful:
http://theopentutorials.com/examples/java-ee/jsp/pagination-in-servlet-and-jsp/
If you're already using a framework take a look at whether that framework has tackled pagination:
Spring MVC provides a data pager
http://blog.fawnanddoug.com/2012/05/pagination-with-spring-mvc-spring-data.html
GWT provides a data pager:
http://www.gwtproject.org/javadoc/latest/com/google/gwt/user/cellview/client/SimplePager.html
The following refrences might be helpful too:
JDBC Pagination
which also points to:
http://java.avdiel.com/Tutorials/JDBCPaging.html
Related
I have a Java application where we use spring data JPA to query our oracle database. For one use case, I need to fetch all the records present in the table. Now the table has record count of 400,000 thousand and it might grow in the near future. I don't feel comfortable pulling all records into the JVM since we don't know how large they can be. So, I want to configure the code to fetch specific number of records at a time say 50,000 and process before it goes to next 50,000. Is there a way I can achieve this with JPA? I came across this JDBC property that can be used with hibernate hibernate.jdbc.fetch_size. What I am trying to understand is if I use repository.findAll() returning List<Entity>How can a fetch Size work in this case? because List will have all the entities. I was also looking into repository methods returning Stream<>, not sure if I have to use that. Please do suggest. If there can be better solution for this use case?
Thanks
With JPA you can use the Pagination feature, means you tell the Repository how many result should be present at one page. (E.g. 50 000)
For more information follow up here https://www.baeldung.com/jpa-pagination
This is related to my other question here. I'm simply looking for a way (an annotation, system configuration etc) to make Hibernate skip over, or ignore columns in models, which do not align with the current ResultSet. To add clarification- I'm not referring to Transient properties, as these properties are all persisted, just each is not involved in every query.
Here is a quick summary of the situation:
I do not have control over the tables, or the queries, everything is done with stored procedures (business decision, not mine).
Because each stored procedure can vary vastly between one another, my idea is to consolidate "branching models" (models that only differ from each other by 1-2 columns) into several larger models, which I can then use to insert or retrieve data, while doing the mapping using DTOs.
The only hang-up in this at the moment, is that Hibernate will throw a SQLException during hydration of a model entity, if it cannot find a matching column in the current ResultSet (funny enough, it doesn't seem to care about the other way around- i.e. if there's little to no columns in the entity, as long as one matches to the current ResultSet).
Any help would be much appreciated, as I've been trying to figure this out for a while now, and I'm about to go back to making each stored procedure have its own entity.
That's not really possible and I would also not recommend this as that could hide mapping errors. If you can't convince the database team to add at least synthetic columns with constant values such that you can reuse the same mapping, you are unfortunately out of luck and will have to continue with your multiple mappings.
I am attempting to write a fairly simply program that can read from and write to a SQL-server database. I've got the writing-to part down pretty well, but am having difficulty using the JPA queries effectively. As nearly as I can tell, the only way to actually access and use data retrieved by a query is to store it in an instance or list of the appropriate entity class, like so:
Query searchQuery = em.createNamedQuery("Document.findByDocTitle");
searchQuery.setParameter("docTitle", title);
List<Document> docList = searchQuery.getResultList();
I'm sure that there must be other means of accessing the data from a query, but if so I haven't been able to find them.
This method has worked for me so far, until I needed to use data from a query across joined tables. In this case it's a m:n relationship between the "document" and "author" tables. I obviously can't store the results of a joined query in a list of document objects, so I'm not sure how to proceed.
I realize that the answer is probably something simple that I just don't know about, but I've been googling JPA stuff all day without any luck. Also, if anyone knows of a good comprehensive book or tutorial for JPA, I would appreciate being pointed in the right direction.
I'm trying to cut down the number of n+1 selects incurred by my application, the application uses EclipseLink as an ORM and in as many places as possible I've tried to add the batch read hint to queries. In a large number of places in the app I don't always know exactly what relationships I'll be traversing (My view displays fields based on user preferences). At that point I'd like to run one query to populate all of those relationships for my objects.
My dream is to call something like ReadAllRelationshipsQuery(Collection,RelationshipName) and populate all of these items so that later calls to:
Collection.get(0).getMyStuff will already be populated and not cause a db query. How can I accomplish this? I'm willing to write any code I need to but I can't find a way that work with the eclipselink framework?
Why don't I just batch read all of the possible fields and let them load lazily? What I've found is that the batch value holders that implement batch reads don't behave well with the eclipselink cache. If a batch read value holder isn't "evaluated" and ends up in the eclipse link cache it can become stale and return incorrect data (This behavior was logged as an eclipselink bug but rejected...)
edit: I found the link to the bug here: https://bugs.eclipse.org/bugs/show_bug.cgi?id=326197
How do I avoid N+1 selects for objects I already have a reference to?
You have three basic ways to load data into objects from a JPA-based solution. These are:
Load dynamically by object traversal (e.g. myObject.getMyCollection().get()).
Load graphs of objects by prefetching dynamically using JPA QL (e.g. FETCH JOINs as described at the Oracle JPA tutorial )
Load by setting the fetch mode ( Is there a way to change the JPA fetch type on a method? )
Each of these has pros and cons.
Loading dynamically by object transversal will generate more (highly targeted queries). These queries are usually small (not large SQL statements, but may load lots of data) and tend to play nicely with a second level cache, but you can get lots and lots of little queries.
Prefetching with JPA QL will give you exactly what you want, but that assumes that you know what you want.
Setting the fetch mode to EAGER will load lots and lots of data for you automatically, but depending on the configuration and usage this may not actually help much (or may make things a lot worse) as you may wind up dragging a LOT of data from the DB into your app that you didn't expect.
Regardless, I highly recommend using p6spy ( http://sourceforge.net/projects/p6spy/ ) in conjunction with any JPA-based application to understand the effects of your tuning.
Unfortunately, JPA makes some things easy and some things hard - mainly, side-effects of your usage. For example, you might fix one problem by setting the fetch mode to eager, and then create another problem where the eager fetch pulls in too much data. EclipseLink does provide tooling to help sort this out ( EclipseLink Performance Tools )
In theory, if you wanted to you could write a generic JavaBean property walker by using something like Apache BeanUtils. Usually just calling a method like size() on a collection is enough to force it to load (although using a collection batch fetch size might complicate things a bit).
One thing to pay particular attention to is the scope of your session and your use of caches (EclipseLink cache).
Something not clear from your post is the scope of a session. Is a session a one shot affair (e.g. like a web page request) or is it a long running thing (e.g. like a classic client/server GUI app)?
It is very difficult to optimize the retrieval of relationships if you do not know what relationships you require.
If you application is requesting what relationships it wants, then you must know at some level which relationships you require, and should be able to optimize these in your query for the objects.
For an overview of relationship optimization techniques see,
http://java-persistence-performance.blogspot.com/2010/08/batch-fetching-optimizing-object-graph.html
For Batch Fetching, there are three types, JOIN, EXISTS, and IN. The problem you outlined of changes to data affecting the original query for cache batched relationships only applies to JOIN and EXISTS, and only when you have a selection criteria based on updateale fields, (if the query you are optimizing is on id, or all instances you are ok). IN batch fetching does not have this issue, so you can use IN batch fetching for all the relationships and not have this issue.
ReadAllRelationshipsQuery(Collection,RelationshipName)
How about,
Query query = em.createQuery("Select o from MyObject o where o.id in :ids");
query.setParameter(ids, ids);
query.setHint("eclipselink.batch", relationship);
If you know all possible relations and the user preferences, why don't you just dynamically build the JPQL string (or Criteria) before executing it?
Like:
String sql = "SELECT u FROM User u"; //use a StringBuilder, this is just for simplity's sake
if(loadAdress)
{
sql += " LEFT OUTER JOIN u.address as a"; //fetch join and left outer join have the same result in many cases, except that with left outer join you could load associations of address as well
}
...
Edit: Since the result would be a cross product, you should then iterate over the entities and remove duplicates.
In the query, use FETCH JOIN to prefetch relationships.
Keep in mind that the resulting rows will be the cross product of all rows selected, which can easily be more work than the N+1 queries.
How do I enable JTable icons and behaviors for sorting table rows by a column, without letting it use a comparison predicate to do the sorting? That is to say, how do I tell the table headers to show the arrow for ascending/descending sort order in the column being used, and get it to call appropriate methods when sort order/column change?
I am trying to create an (editable, filterable, sortable) JTable backed by an SQL query or view. The rows may not fit in memory, and may not map cleanly to java objects, so I want to do all sorting/filtering within SQL. I have already written the code for changing a query to accommodate sorting by column, filtering by values, and visible columns.
To use this, I am planning to write a JTableModel based on a ResultSet with TYPE_SCROLL_SENSITIVE, and CONCUR_UPDATABLE, so changes to the DB get propagated to the ResultSet. I will periodically (several times a second) force a refresh of the visible JTable from the ResultSet, so changes to the database become visible to the user. User changes to the table will be passed to the updateable ResultSet after validation.
I've looked a little bit at how sorting is done normally, but most implementations seems to rely on the JTable creating a javax.swing.RowSorter with a Comparator predicate, or on maintaining a sorted list of rows that fires events when changed. So, my questions:
ORM frameworks are NOT an answer to this question, because the data do not map well to entity objects. Also, the DBMS I am using is H2.
EDIT: Sortable JTable libraries based on applying Comparators or sorting predicates to row objects are also unsuitable, unfortunately. I do not believe I will be able to hold all objects in memory in order to perform sorting. This problem prevents me from using the SwingX JXTables, GlazedLists, or similar libraries. I wish I could, but I can't. Period.
** I will be dealing with many thousand rows, potentially millions, with numerous columns. Yes, I really DO need to use SQL to do the sorting and filtering.**
Questions: (in descending importance)
How do I show indicators for which column is used to sort rows?
How do I get the JTable to fire appropriate events when the column headers are LEFT-clicked to change sort order?
Is there an easier way to force the JTable to update when the database changes?
Is there a library that would make all this considerably easier (connecting DB queries or views and JTables)?
Am I going to run into horrible, horrible problems when I design the system like this?
I have never used it myself but JIDE Data Grids provides a DatabaseTableModel that provides filtering and sorting support using SQL WHERE and ORDER BY.
In answer to 1 and 2, check out SwingX, which already includes a table class with built-in sorting (and filtering). You may be able to adapt this.
Am I going to run into horrible, horrible problems when I design the system like this?
From experience, yes. I worked on a project almost exactly the same as this, where someone had designed a JTable that supposedly 'magically' bound to a database table. This coupled display logic and database access together in one big horrible mess, which we replaced entirely with reflection-driven table models and separate record CRUD operations.
You say that ORM is not the answer...
If the format of the data doesn't change, then it's worth considering anyway. Your 'entity' classes need not represent real-world entities.
If (as I suspect) your entity format changes, it might be worth considering:
A flexible map-based Record class which stores records as key-value pairs;
Dynamically-built table models for your display logic, built by querying record keys, plugged into SwingX tables to get sort and filter for free;
A similarly-designed Repository class which encapsulates your database access separately from the table itself, responsible for loading and saving Records. This acts as an adapter between your updateable ResultSet and the view (although I'd check whether using a ResultSet this way is going to require an open database connection whilst data is visible...).
This separation into 'a table that displays and sorts records' and 'a repository that manages the data' means:
You can reuse the table for non-database-bound data;
You can display database-bound records in things other than tables;
You won't go mad trying to build and test the thing :)
You should be able to subclass javax.swing.RowSorter in order to create a row sorter that does the sorting in the database. From the API docs:
"RowSorter implementations typically don't have a one-to-one mapping with the underlying model, but they can. For example, if a database does the sorting, toggleSortOrder might call through to the database (on a background thread), and override the mapping methods to return the argument that is passed in."
http://docs.oracle.com/javase/6/docs/api/javax/swing/RowSorter.html
Leaving aside the database stuff there's a class called SortableTable that's a part of JIDE Grids. It displays the sorting with a little ^ or v in the table header, and supports sorting by more than 1 column (1v, 2v, etc.).