Our team is building a small application wherein a UI has about 10 drop-down list boxes. ( DDLB ).
These list boxes will be populated by selecting data from different tables.
Our JAVA person feels that making separate database call for each list will be very expensive and wants to make a single database call for all lists.
I feel it is impractical to populate all lists in one database call due to following reason
a. Imagine an end user chooses state = 'NY' from one DDLB.
b. The next drop down should be populated with values from ZIP_CODES table for STATE='NY'
Unless we know ahead of time what state a user will be choosing - our only choice is to populate a java structure with all values from ZIP_CODES table. And after the user has selected the state - parse this structure for NY zipcodes.
And imagine doing this for all the DDLB in the form. This will not only be practical but also resource intensive.
Any thoughts ?
If there are not many items in those lists and memory amount allows you could load all values for all drop boxes into memory at application startup and then filter data in memory. It will be better then execute SQL query for every action user makes with those drop boxes.
You could also use some cache engines (like EhCache) that could offload data to disk and store only some fraction in memory.
You can run some timings to see, but I suspect you're sweating something that might take 100th of a second to execute. UI design wise I never put zip codes in selection menus because the list is too long and people already know it well enough to just punch in. When they leave the zip code field I will query the city and state and pre-fill those fields if they're not already set.
Related
I have an SQLite database which I have to be constantly retrieving data from. Changes may be done to the data between each retrieval.
My goal is to maximize the app performance, so what is the fastest way to do this retrieving?
I can imagine 2:
constantly opening and closing new cursors
query all data at the beginning and store it in an ArrayList. When changing the data, change both SQLite DB and the ArrayList using indexOf.
---- EDITED ----
I need the data to create markers in a google's map.
I have considered using CursorLoader but as I don't need to interact whith other apps I don't want to use Content Providers.
Would creating a custom loader be a good idea?
In short, while it's not always that simple, the fastest way to do things is all at once.
Constantly making calls to and from a database can really make your apps performance bottleneck, especially if it's to a server and not just your devices SQLite database.
Depending on what you're doing with the data, you may be able to look into something like a CursorAdapter which handles the display of rows from the database, and each time you insert/update a row, the CursorAdapter will update the ListView accordingly. It also handles the opening/closing/moving to next of the Cursor, making it very readable and easy for developers to follow.
Again, however, try to do things in as few calls as possible. If you stick to using an ArrayList:
Make one call in the beginning for all items.
Loop through that cursor and add items to an array list.
Use the array list as a cache. Sure, you could update the DB each time you update the list (which might be safest, IMO), or you can just loop through the list and insert/update/delete when the app closes. If you take that approach, make sure you do so in a method like onPause(), as it is one of the earliest methods in which an Activity can be killed.
Perfect use case for a CursorLoader. Given a query, it'll keep your list adapter up to date with the latest data, assuming you notify when changes happen in the DB. It also conveniently handles activity lifecycle events for your (ie. it'll close the cursor when the activity finishes, stop updating when it pauses, etc.).
The fastest way is obviously to not use a database at all. However, that is clearly not a solution unless you find some way of exposing your array to access from elsewhere.
Using a database is a convenient way of centralising the data so many users can access the data and have the data up-to-date at all times. Unfortunately this is the slowest option.
Choosing your middle-ground between speed and availability is a difficult task. You have to find a balance between stale data and throughput.
If, for example, you would be comfortable with a picture of the data that was valid just 5 seconds ago then you could probably cache the data locally in your array and arrange for some mechanism to keep it up-to-date running behind the scenes.
If a 5 minute lag was acceptable you could probably arrange for a regular push to database.
Also, any mechanism you use must also handle parallel changes to the data - perhaps two users change the same datum at the same time.
You just need to decide on where to strike your balance.
I have a case of following:
I have an object called Song with one of its attribute is Name.
I have a sqlite database that can contain the Song up to million records.
I need to do a live search (upon user tapping a letter, the app will directly search it) for the Song's Name attribute.
The solution that I've been thinking (I haven't done any code) is:
Directly querying the sqlite (I believe this is the worst solution because I need to fetch from sqlite and put it to Cursor)
Put all the sqlite records to List or ArrayList, then I'm not sure where to proceed as I believe to do for loop is not the best solution as I need to store to other List to prepopulate my ListView
Can anyone give me the idea, maybe the concept only, to solve this?
You should do caching for your queries. Build an in-memory data structure for this, with a cascading structure (one letter at a time).
Limit the number of the returned objects (e.g. return max 10) for a specific query. Obviously you don't want want to return all Songs that begin with letter A. Also, this list may be based on importance (or selection frequency) - you can store that as well.
For the internal representation of the cache I recommend something like this:
public class SongCacheNode {
private String selector;
private Map<Character, SongCacheNode> subCaches =
new HashMap<Character, SongCacheNode>();
private List<Song> selection = new ArrayList<Song>(10);
private boolean leafNode = false;
private boolean containsAll = false;
}
You can build a tree-like structure of this.
The selector would store the string selector of this node (e.g. prefix of the song title).
The subCaches can store the next cache for the next letter.
The selection can store the picked song titles
The leafNode would say that there are no more data stored in cache, you can either use selection, or you have to do SQL - that's stored in containsAll
if containsAll is true, then all the possible songs are stored in selection, if not, you have to still do SQL
This structure allows you to build a variable-depth cache, depending the song title distribution. Also, for the selection you can select any match, not only the prefix match (e.g. many song title begin with 'The'), make a case-insensitive query, etc. You can do partial caching as well, to limit memory usage (e.g. to store maximum 5 characters, or do not store infrequent queries even with large number of songs) - just set containsAll to false.
Your Options
By loading all the Songs into a list, you will speed up searching time. However, this will impact massively on loading time.
If you do not load all of the songs into the program, then you will have a much shorter loading time, but presumably longer searching lag.
Which One I would go for
Simply, I would choose the latter. The advantage here is that you can make the database call in another thread, while the user is typing. Multi-threading your application means that one the one thread, the user can happily type away, while on the other thread, the arduous database calls will occur, and the data can be loaded in.
Some things to watch out for / Ideas you might want to use.
If your database has a million songs, it's safe to say there will be alot of songs that start with the same letter. Perhaps not making a database call on the first few key presses will serve to your advantage for a number of reasons:
You will not have reams and reams of data coming in after every key press.
It will lessen the amount of data your program has to handle, thus lowering the performance hit.
This will provide a less intimidating user interface for the client. Imagine typing in a song and seeing a list of a hundred thousand songs come out of nowhere. It doesn't make for fluid usability.
Another idea is to cache the data, on each query. With this in mind, the same data can be obtained very quickly.
SQL lite allows full text search functionality. Given that song names can also be searched with partial terms like 'thrill' | 'thriller' | 'thriller MJ', you should go with that.
While the full text search would be indexed, depending on the load it might be too much for the DB to take, in which case you can build a trie of song names. All matching songs can be obtained from the child nodes of the last matching node and this structure would be in memory. That should make it fast.
Loading song details into a list / set will not help since you cannot make partial matches. List / Set either contains 'thril' or it does not. There is no way to know if the partial string matches something in your DB unless you use full text search or an in-memory DS like a trie.
I need to keep certain objects in an arbitrary order (dictated by the user, at will) but I do not know which would be the best approach.
I have been thinking in just set an integer order field in my entities and make the user order them but this approach troubles me because we have 3 servers dispatching requests and I believe that if I update the order in the database it forces me to update/merge my entitites every time I want to make calculations based on that order.
The other approach would be using an IMDG or maybe a cache to set the order in a shared location and query such location for those calculations but I believe this would be overkill for such task.
Which would be the best approach? or is it other?
The method with the integer order column is a possibility. Then you also add a version column into the database (probably you already have one), and when one user changes the order, all records with the new order number have to be saved. Due to the version column other users get informed they have to reload their data because of the modified order. Instead of the integer you also can use a number with some digits after the decimal point or a String - both makes it easier to update only that rows which really have a new place in the order instead of updating all rows in the table.
The inconvenience of this method is, a user has to reload the row (which means he has to re-do his actual modification) only because a different user changed the order. If this condition rarely happens, you can live with it, but if changes in the order happen quite often, this might be unacceptable.
To avoid this inconvenience you can create a separate table only for the order (as columns the order number, the key of your data table and a version column). This table has a 1:1 relation to the data table. When the order changes, then only this table has to be updated, which means other users do not get hassled when they modify the data of any records. In this case you even can realize different orders (for example every user can define his own order) - which would change the relation with the data table from 1:1 to n:1.
A cache in a shared location I only would do if there are performance problems or if there is no need to persist an order into the data base (order is valid only for one session).
i am not sure if I understood the problem correctly. If different user could define his own order, then the order should somehow go to user's profile, it could be a separated table or a field.
why you want to add user-specific information to your entity table? Or I misunderstood the question?
I have a customer with a very small set of data and records that I'd normally just serialize to a data file and be done but they want to run extra reports and have expandability down the road to do things their own way. The MySQL database came up and so I'm adapting their Java POS (point of sale) system to work with it.
I've done this before and here was my approach in a nutshell for one of the tables, say Customers:
I setup a loop to store the primary key into an arraylist then setup a form to go from one record to the next running SQL queries based on the PK. The query would pull down the fname, lname, address, etc. and fill in the fields on the screen.
I thought it might be a little clunky running a SQL query each time they click Next. So I'm looking for another approach to this problem. Any help is appreciated! I don't need exact code or anything, just some concepts will do fine
Thanks!
I would say the solution you suggest yourself is not very good not only because you run SQL query every time a button is pressed, but also because you are iterating over primary keys, which probably are not sorted in any meaningful order...
What you want is to retrieve a certain number of records which are sorted sensibly (by first/last name or something) and keep them as a kind of cache in your ArrayList or something similar... This can be done quite easily with SQL. When the user starts iterating over the results by pressing "Next", you can in the background start loading more records.
The key to keep usability is to load some records before the user actually request them to keep latency small, but keeping in mind that you also don't want to load the whole database at once....
Take a look at indexing your database. http://www.informit.com/articles/article.aspx?p=377652
Use JPA with the built in Hibernate provider. If you are not familiar with one or both, then download NetBeans - it includes a very easy to follow tutorial you can use to get up to speed. Managing lists of objects is trivial with the new JPA and you won't find yourself reinventing the wheel.
the key concept here is pagination.
Let's say you set your page size to 10. This means you select 10 records from the database, in a certain order, so your query should have an order by clause and a limit clause at the end. You use this resultset to display the form while the users navigates with Previous/Next buttons.
When the user navigates out of the page then you fetch an other page.
https://www.google.com/search?q=java+sql+pagination
I currently have a Richfaces dataTable bound to a backing bean that contains all the data. This table is bound to a Richfaces data scroller.
Users can then enter scroll between the data.
There is also a text box that dynamically updates the data table results displayed to them, based on text they enter into it. It reRenders the dataTable and datascroller on the keyUp event.
The backing bean it’s bound to first pulls all the data from a number of database tables. This data is pulled when the user submits a normal post request to the server, and it’s the results of this request that are used for all subsequent Ajax related queries (Results saved as list of objects, no more database calls made).
I have a problem in that the dataset can be huge at times, 100,000s of records.
This causes the initial request to the server to take a very long time.
A possible solution I'm looking at is pulling back only a small amount of the data in one thread for the initial user port request. This data can then be displayed in the data table while the main thread works in the background pulling the bulk of data back.
Is this feasible? Would it be possible to update my datatable/datascoller as the main thread pulls back new data? Would it be difficult?
Database and query optimization has been looked at, so no more improvements of any significance can be made there.
Thanks in advance (I know this probably is not an easy question to answer)
Implement SerializableDataModel to hold your data for sorting and paginating.
Seems like you need exactly what Seam Application Framework's Query object is providing.
If you don't want to use Seam, you can either view the source, and copy how they are doing it and just steal the idea.
Basically what you need to do is fetch a given result set for each time the user press next, previous, first, last etc