We Keep Coding

How to map custom data structures to bean entities with JPA/Hibernate?

We have a (possibly large) custom data structure implemented in Java (8+). It has a simple and optimal API for querying pieces of data. The logical structure is roughly similar to an RDMS (it has e. g. relations, columns, primary keys, and foreign keys), but there is no SQL driver.
The main goal is to access the data via ORM (mapping logical entities to JPA annotated beans). It would be nice if we could use JPQL. Hibernate is preferred but other alternatives are welcome too.
What is the simplest way to achieve this? Which are the key parts of such an implementation?
(P. S. Directly implementing SessionImplementor, EntityManagerImplementor etc. seems to be too complicated.)

You have two possibilities.
Implement a JDBC compliant driver for your system, so you can use a JPA implementation such as Hibernate "directly" (although you may need to create a custom dialect for your system).
Program directly against the JPA specification like ObjectDB does, which bypasses the need to go through SQL and JPA implementations completely.
The latter one is probably easier, but you'd still need to implement the full JPA API. If it's a custom in-house-only system, there's very little sense in doing either one.

One idea I thought up just now, that I feel may work is this:
Use an existing database implementation like H2 and use the JPA integration with that. H2 already has a JPA integration libraries, so it should be easy.
In this database, create a Java stored procedure or function and call it from your current application through JPA. See this H2 documentation on how to create a Java stored procedure or function. (You may want to explore the section "Using a Function as a Table" also.)
Define a protocol for the service methods and encapsulate it in a model class. An instance of this model class may be passed to the function/SP and responses retrieved.
Caveat: I have never done this myself but I think it will work.
Edit: Here is a diagram representing the thought. Though the diagram show H2 separately, it will most probably be in the same JVM as "Your Java/JEE application". However, since it is not necessary to use H2, I have shown it as as separate entity.

Why most hibernate applications are using sequence for id generation?

Why most hibernate application are using sequence for id generation?
Why not use the default GenerationType=AUTO in #GeneratedValue annotation?
P.S. In my professional career I see everybody is use sequences, but I don't understand why they bother with harder to deploy solution (there is always sequence create SQL command in deployment instructions).

I see several reasons:
The most used database in enterprise apps is probably Oracle, and Oracle doesn't have auto-generated IDs, but sequences.
Sequences allows having the ID before inserting a new row, rather than after inserting the new row. This is easier to use and more efficient because you can batch insert statements at the end of the transaction but still have IDs definned in the middle of the transaction.
Sequences allow using hilo algorithms (which is the default with the hibernate sequence generation), and thus make only one DB call for several inserts, thus increasing performance.
AUTO varies between databases, whereas sequence always uses the same strategy.

From the excellent book Pro JPA 2 Mastering Java Persistence API by Mike Keith and Merrick Schincario.
From Chapter 4: Object Relational Mapping, section Identifier Generation.
[...] If an application does not care what
kind of generation is used by the
provider but wants generation to
occur, it can specify a strategy of
AUTO.
There is a catch to using AUTO,
though. The provider gets to pick its
own strategy to store the identifiers,
but it needs to have some kind of
persistent resource in order to do so.
For example, if it chooses a
table-based strategy, it needs to
create a table; if it chooses a
sequence-based strategy, it needs to
create a sequence. The provider can’t
always rely on the database connection
that it obtains from the server to
have permissions to create a table in
the database. This is normally a
privileged operation that is often
restricted to the DBA. There will need
to be some kind of creation phase or
schema generation to cause the
resource to be created before the AUTO
strategy is able to function.
The AUTO mode is really a generation
strategy for development or
prototyping. It works well as a means
of getting you up and running more
quickly when the database schema is
being generated. In any other
situation, it would be better to use
one of the other generation strategies
discussed in the later sections [...]

At least for Oracle: one reason is to be able to track the number of objects in a table (for which the table-specific sequence is good, if no objects are deleted from the table). Using GenerationType=AUTO uses a global sequence number, which results in gaps in id numbers when having more than one table in the database.

There are different considerations for choosing identity generator, the most important ones are performance and portability but also clustering and data migration might be a consideration.
In practice in the latest Hibernate versions (if not all of them) the SEQUENCE strategy is actually a sequence based HiLo and not a pure sequence as must people assume.
You can read a pretty details post regarding identity generation strategies at my blog: here
Eyal

XML vs. object trees

In my current project (an order management system build from scratch), we are handling orders in the form of XML objects which are saved in a relational database.
I would outline the requirements like this:
Selecting various details from anywhere in the order
Updating / enriching data (e.g. from the CRM system)
Keeping a record of the changes (invalidating old data, inserting new values)
Details of orders should be easily selectable by SQL queries (for 2nd level support)
What we did:
The serialization is done with proprietary code, disassembling the order into tables like customer, address, phone_number, order_position etc.
Whenever an order is processed a bit further (e.g. due to an incoming event), it is read completely from the database and assembled back into a XML document.
Selection of data is done by XPath (scattered over code).
Most updates are done directly in the database (the order will then be reloaded for the next step).
The problems we face:
The order structure (XSD) evolves with every release. Therefore XPaths and the custom persistence often breaks and produces bugs.
We ended up having a mixture of working with the document and the database (because the persistence layer can not persist the changes in the document).
Performance is not really an issue (yet), since it is an offline system and orders are often intentionally delayed by days.
I do not expect free consultancy here, but I am a little confused on how the approach could be improved (next time, basically).
What would you think is a good solution for handling these requirements?
Would working with an object graph, something like JXPath and OGNL and an OR mapper be a better approach? Or using XML support of e.g. the Oracle database?

If your schema changes often, I would advise against using any kind of object-mapping. You'd keep changing boilerplate code just for the heck of it.
Instead, use the declarative schema definition to validate data changes and access.
Consider an order as a single datum, expressed as an XML document.
Use a document-oriented store like MongoDB, Cassandra or one of the many XML databases to manipulate the document directly. Don't bother with cutting it into pieces to store it in a relational db.
Making the data accessible via reporting tools in a relational database might be considered secondary. A simple map-reduce job on a MongoDB, for example, could populate the required order details into a relational database whenever required, separating the two use cases quite naturally.

The standard Java EE approach is to represent your data as POJOs and use JPA for the database access and JAXB to convert the objects to/from XML.
JPA
Object-to-Relational standard
Supported by all the application server vendors.
Multiple available implementations EclipseLink, Hibernate, etc.
Powerful query language JPQL (that is very similar to SQL)
Handles query optimization for you.
JAXB
Object-to-XML standard
Supported by all the application server vendors.
Multiple implementations available: EclipseLink MOXy, Metro, Apache JaxMe, etc.
Example
http://bdoughan.blogspot.com/2010/08/creating-restful-web-service-part-15.html

Creating tables dynamically in Hibernate environment

There's an enterprise application using Java + Hibernate + PostgreSQL. Hibernate is configured via annotations in the Java source code. So far the database schema is fixed, but I faced the problem that it needs to be dynamic:I can receive data from different locations and I have to store these in different tables. This means that I have to create tables run-time.
Fortunately, it seems that all of these data coming from the different institutes can have the same schema. But I still don't know how to do that using Hibernate. There are two main problems:
How to tell to Hibernate that many different tables have the same structure? For example the "Patient" class can be mapped to not just the "patient" table, but the "patient_mayo_clinic" table, "patient_northwestern" table, etc. I can feel that this causes ambiguity: how Hibernate knows which table to access when I do operations on the Patient class? It can be any (but only one) of the former listed tables.
How can I dynamically create tables with Hibernate and bind a class to them?
Response to suggestions:
Thanks for all of the suggestions. So far all of the answers discouraged the dynamic creation of tables. I'll mark Axel's answer, since it achieves certain goals, and it is a supported solution. More specifically it's called multi-tenancy. Sometimes it's important to know some important phrases which describes our problem (or part of our problem).
Here are some links about multi-tenancy:
Multi-tenancy in Hibernate
Hibernate Chapter 16. Multi-tenancy
Multi-tenancy Design
EclipseLink JPA multi-tenancy
In real world scenario multi-tenancy also involves the area of isolating the sets of data from each other (also in terms of access and authorization by different credentials) once they are shoved into one table.

You can't do this with Hibernate.
Why not extend your patient table with an institute column?
This way you'll be able to differentiate, without running into mapping issues.

I am afraid you can't do this easily in Hibernate. You would have to generate the Java source, compile it, add it to your classpath and load it dynamically with java.reflection package. If that works, which I doubt it, it will be an ugly solution (IMHO).
Have you consider using a schema less database i.e: NoSQL databases or RDF
databases. They are much more flexible in terms of what you can store in them , basically things are not tight up against a relational schema.

In most environments it is not a good idea to create tables dynamically simply because dbas will not give you the rights to create tables in production.
Axel's answer may be right for you. Also look into Inheritance Mapping for Hibernate.

I agree that its not advisable to create tables dynamically nevertheless it's doable.
Personally i would do as Axel Fontaine proposed but if dynamic tables is a must-have for you I would consider using Partitioning.
PostgreSQL allows you to create ona main table and few child tables (partitions), records are disjunctive between child tables, but every record from any child table is visible in parent table. This means that you can insert rows into any child table you want using just simple insert statement (its not cool but has the same level of complexity as composing and persisting an entity, so its acceptable in your case) and query database using HQL

Will hibernate java programs have no sql code?

I haven't worked with hibernate. I have little bit of experience in java. I was going through source of a beast of an java application created by Oracle(Retail Price Management). I was expecting a lot of sql code embedded in there as the application makes heavy use of database. But to my surprise, NO embedded SQL code! so far. I found that it was using what is called as "Hibernate" from the lot of .hbm.xml files. Is it a trademark for java programs using hibernate or maybe I haven't seen the complete codebase?. Could someone enlighten me how this is possible?. Thanks.

Hibernate, as all ORM tools, indeed lessens or eliminates the need to use raw SQL in Java code, due to the following:
many associations between various entities are recorded in the Hibernate mapping, so these are fetched automatically by Hibernate - i.e. if you have an aggregation relationshiop between two classes on the Java side, this may be mapped as a foreign key relationship in the DB, and Hibernate, whenever an instance of class A is loaded, can automatically load the associated instances of class B too,
many queries can be done in Hibernate's own HQL query language, or using its Criteria API.
Under the hood Hibernate does generate SQL to communicate with the DB, but this is not visible on the Java side. It can be seen in the logs though, if it is enabled.
Due to this, programs using Hibernate very rarely need to use JDBC or SQL directly. The exceptions are typically ralted to "tricky" legacy DB schemas which can't be fully handled by Hibernate.

Because that's the whole purpose of using Hibernate or any other object-relational mapping framework.
Hibernate solves object-relational impedance mismatch problems by replacing direct persistence-related database accesses with high-level object handling functions.

Hibernate generates SQL for all its standard database operations. It understands different SQL dialects, and the mapping files (.hbm.xml) tell it about the database structure so it knows how to construct its queries. There is a showSql setting you can turn on if you want to see it outputting its generated SQL as it runs.

Hibernate is an Object-Relational Mapper (ORM). ORMs are used to hide the ugly details of SQL incompatibility[sic] between databases from your program -- you define your tables and map them to an object hierarchy (the .hbm.xml files) and then Hibernate does the rest. Thus most programs that use Hibernate won't see a single phrase of SQL, unless there's a specific reason to execute a complicated query.

Hibernate is a tool, or technology that takes care of the interaction between the database and application for you. You have to tell the structure of the application and the database to it, this is what is in the .hbm.xml files.
The SQL is generated by Hibernate at runtime (kind of)
Say you have an Fruit class, and objects of this is persisted into a T_FRUIT table.
You say this to hibernate, via the .hbm.xml files. That there is a table T_FRUIT, this table is represented by the Fruit class, and which fields in the Fruit class correspond to which columns in th T_FRUIT table.
And then it knows whenever you are trying to save a fruit, it should insert/update to the T_FRUIT table.
When you want to create an Apple, you create an object of fruit corresponding to apple and save "save this fruit".
Hibernate takes care of persisting it.
You can have relationships defined between tables, and Hibernate is intelligent enough to persist in multiple tables.
When you fetch a fruit, hibernate fetches the details of the fruit and its children also(data from referencing tables). And you can say whether you want fetch all the children
at once, or as and when required.
And so on. Aim is to make your life easier, and code maintainable, easy to read, portable,...
With this info, let me redirect you.