We Keep Coding

Insert unique entities to the sqlite DB

There will be up to 100k entities in sqlite DB with following structure:
ID (Numeric, PK)
KEY (Varchar, Unique/PK)
Other fields, mostly varchars
I have a list of about 100-1k entities. I want to add to the DB only those entities, which KEY is not present in the DB and show the list of added ones.
Example
As an relevant example you may consider something like this: library with books as entities. Each Book has global unique ISBN number (KEY) and unique id (ID) in the library catalog.
Some person brings to the library set of books. Library checks the books by ISBN in the catalog, takes 'new' books and shows to the person list of taken books.
Some thoughts how it can be achieved:
1) select all KEYs from the DB, put them into [hash]set, in loop verify that KEY from new entities does not exist in the set.
2) like #1 but instead of selecting all KEYs, select only KEYs that present both in DB and the list
3) in loop check existence of entity with additional select query
4) enable constraints in the DB, check existence by catching exceptions
All of them have their own disadvantages, I believe. Can you suggest something better?
For now I'm asking mostly about 'best practices', I believe any of the approach will work for my case without huge performance issues (no actual tests for now, I'm just in analysis phase), but how should it be done better?
Code will be in Java, I plan to use simple DAOs with JDBC, but if someone suggests Hibernate as an alternative approach, I will reconsider my thoughts.

Your suggested solutions are all valid and possible situations, altough Number #1 and Number #4 are rarely good ideas. Number #2 and #3 are solutions that are often used. To choose from the two you need to answer the following questions:
What will be the typical use case for your application? (sometimes you just get 1 new book, you will always get hundreds of books at once, mixed)
Based on what metrics will this code be "judged", what are you trying to optimize for? (readability/maintainability, performance, etc)
Based on the answers to these questions above you can either pick #2 or #3.

Hibernate Search query for class

I'm using hibernate search 4.4.0. And I met a problem recently.
E.g, I have 2 classes INDEXING and DATA_PROPERTY. There is no association between 2 of them. And I can't change them or creat a new class to associate 2 of them.
Part of Lucene indexing:
mapping.entity(DatatypeProperty.class).indexed().providedId()
.property("rdfResource",ElementType.FIELD).field().analyze(Analyze.NO).store(Store.YES)
.property("partitionValue", ElementType.FIELD).field().analyze(Analyze.NO)
mapping.entity(Indexing.class).indexed().providedId()
.property("rdfResource",ElementType.FIELD).field().analyze(Analyze.NO).store(Store.YES)
Now in the SQL, I use
SELECT IND.RDF_RESOURCE
FROM INDEXING IND, DATA_PROPERTY DP
WHERE IND.RDF_RESOURCE = DP.RDF_RESOURCE
AND IND.OBJECT_TYPE_ID_INDEXED IN (........)
AND DP.PARTITION_VALUE IN (......)
AND .......
How can I translate IND.RDF_RESOURCE = DP.RDF_RESOURCE in Hibernate Search???
I thought maybe I can use the query to find all the RDF_RESOURCE of class DatatypeProperty and matching all of them in the query for class Indexing. But it seems very inefficiency.
Does anyone has a better way for this?

I have 2 classes INDEXING and DATA_PROPERTY. There is no association
between 2 of them. And I can't change them or create a new class to
associate 2 of them.
In this case you are between a rock and a hard place. You will need to associate the records somehow and the most obvious choice is via an association. Also, you cannot compare a SQL join with a free text based index provided by Lucene.
One potential solution could be to write a custom bridge which at indexing time executes the join and indexes the relevant data, so that you can target it directly via your query. Whether this works for you will depend on your use case. In your example setup, I don't see any field which would benefit from free text search. I can only assume that you are only showing parts of your code. If not, why don't you just stick with SQL?

How to Iterate across records in a MySql Database using Java

I have a customer with a very small set of data and records that I'd normally just serialize to a data file and be done but they want to run extra reports and have expandability down the road to do things their own way. The MySQL database came up and so I'm adapting their Java POS (point of sale) system to work with it.
I've done this before and here was my approach in a nutshell for one of the tables, say Customers:
I setup a loop to store the primary key into an arraylist then setup a form to go from one record to the next running SQL queries based on the PK. The query would pull down the fname, lname, address, etc. and fill in the fields on the screen.
I thought it might be a little clunky running a SQL query each time they click Next. So I'm looking for another approach to this problem. Any help is appreciated! I don't need exact code or anything, just some concepts will do fine
Thanks!

I would say the solution you suggest yourself is not very good not only because you run SQL query every time a button is pressed, but also because you are iterating over primary keys, which probably are not sorted in any meaningful order...
What you want is to retrieve a certain number of records which are sorted sensibly (by first/last name or something) and keep them as a kind of cache in your ArrayList or something similar... This can be done quite easily with SQL. When the user starts iterating over the results by pressing "Next", you can in the background start loading more records.
The key to keep usability is to load some records before the user actually request them to keep latency small, but keeping in mind that you also don't want to load the whole database at once....

Take a look at indexing your database. http://www.informit.com/articles/article.aspx?p=377652

Use JPA with the built in Hibernate provider. If you are not familiar with one or both, then download NetBeans - it includes a very easy to follow tutorial you can use to get up to speed. Managing lists of objects is trivial with the new JPA and you won't find yourself reinventing the wheel.

the key concept here is pagination.
Let's say you set your page size to 10. This means you select 10 records from the database, in a certain order, so your query should have an order by clause and a limit clause at the end. You use this resultset to display the form while the users navigates with Previous/Next buttons.
When the user navigates out of the page then you fetch an other page.
https://www.google.com/search?q=java+sql+pagination

Show all children given a parentNode with Hibernate

It seems this problem is quite popular on programming.
I have a table that among its fields have Id and ParentId.
The question is, how to, using Hibernate, handle properly a task like "retrieve all children of a given node"
Right now I got something like a recursive loop that finds children given a parentId, but at 10000 rows in the table is just insane.
All I can find is that people say this DB structure is not adequate for big amounts of data, but I'm already on that path. What can I do other than killing my DB with 10000 small queries?

When a hibernate object has a one-to-many property, each call to get the next child forces a query. This is called "lazy loading" and increases performance by grabbing them on demand. You can force hibernate to "eagerly load" the set all in one query by setting the lazy property on your mapping. More information is available in the documentation:
http://docs.jboss.org/hibernate/core/3.3/reference/en/html/performance.html#performance-fetching-lazyproperties

Persisting Ordered Domain Objects

A very common use case in many web applications is that domain objects can be ordered by a user in the web interface - and that order is persisted for later; but I've noticed that every time I need to implement this, I always end up coming up with a solution that is different and adds considerable weight and complexity to a simple domain object. As an example, suppose I have the persistent entity Person
Person {
long id
String name
}
A user goes to /myapp/persons and sees all people in the system in the order in which they receive compensation (or something) - all of the people can be clicked and dragged and dropped into another position - when the page is loaded again, the order is remembered. The problem is that relational databases just don't seem to have a good way of doing this; nor do ORMs (hibernate is what I use)
I've been thinking of a generic ordering methodology - but there will be some overhead as the data would be persisted separately which could slow down access in some use cases. My question is: has anyone come up with a really good way to model the order of persistent domain objects?
I work in JavaEE with Hibernate or JDBCTemplate - so any code examples would be most useful with those technologies as their basis. Or any conceptual ideas would be welcome too.
UPDATE: I'm not sure where I went wrong, but it seems I was unclear as most have responded with responses that don't really match my question (as it is in my head). The problem is not how do I order rows or columns when I fetch them - it is that the order of the domain objects change - someone clicks and drags a "person" from the bottom of the list to the top of the list - they refresh the page and the list is now in the order they specified.

When fetching results, just build HQL queries with a different ORDER BY clause, depending on what the user has used last time

The problem is that relational databases just don't seem to have a good way of doing this; nor do ORMs (hibernate is what I use)
I'm not sure where you would get this impression. Hibernate specifically has support for mapping indexed collections (which is a "list" by another name), which usually boils down to storing a "list-index" column in the table holding the collection of items.
An example taken directly from the manual:
<list name="carComponents"
table="CarComponents">
<key column="carId"/>
<list-index column="sortOrder"/>
<composite-element class="CarComponent">
<property name="price"/>
<property name="type"/>
<property name="serialNumber" column="serialNum"/>
</composite-element>
</list>
This would allow a List<CarComponents> to be associated with your root entity, stored in the CarComponents table with a sortOrder column.

One possible generic solution:
Create a table to persist sort information, simplest case would be be one sortable field per entity with a direction:
table 'sorts'
* id: PK
* entity: String
* field: String
* direction: ASC/DESC enumeration (or ascending boolean flag)
It could be made more complicated by adding a userId to do per-user sorting or by adding a sort_items table with a foreign key to support sorting by multiple fields at a time.
Once you're persisting the sort information, it's a simple matter of adding Order instances to criteria (if that's what you're using) or concatenating order by statements to your HQL.
This also keeps your entities themselves free of and ordinal information, which in this case sounds like the right approach since the ordering is purely for user interaction purposes.
Update - Persisting entity order
Given the fact that you want to be able to reorder entities, not just define a sort for them, then you really do need to make an ordinal or index value part of the entity's definition.
The problem, as I'm sure you realize is the number of entities that would need to be updated, with the worst case scenario being moving the last entity to the top of the list.
You could use an increment value other than 1 (say 10) so you would have:
ordinal | name
10 | Crosby
20 | Stills
30 | Nash
40 | Young
Most of the time, updating the row would involve selecting two items and updating one. If I want to move Young to position 2, I select current item 2 and the previous item from the database to get the ordinals 10 and 20. Use these to create the new ordinal ((20 - 10) / 2 + 10 = 15). Now do a single update of Young with an ordinal of 15.
If you get to the point where division by two yields the same index as one of the entities you just loaded, that means it's time to spawn a task to normalize the ordinal values according to your original increment.

As far as I know, JPA 2.0 provides support for ordered lists:
https://secure.wikimedia.org/wikibooks/en/wiki/Java_Persistence/Relationships#Order_Column_.28JPA_2.0.29

I think that relational databases cannot do better than a dedicated ordering column.
The idea of "order" is not really defined in SQL for anything but cursors, and they are not a core relational concept but rather an implementation detail.
For all I know the only thing to do is to abstract the ordering column away with #OrderColumn (JPA2, so Hibernate 3.5+ compatibile).