We Keep Coding

Should I use a Java ORM or should I use my own sql queries? Or maybe a mix of both?

Here's currently what my database looks like and I have it hooked up properly with the Java Spring boot application. I would like to use some of the benefits of having a ORM but also would like to know if I can also write my own sql queries if need be.
Though I'm new to using ORM's and could use some advice on if it's possible to use a mix of an ORM and a raw sql queries.
ER-Diagram

Create an interface that extends JpaRepository or CrudRepository.
public interface <YourClassRepository> extends JpaRepository<YourClass, YourClassId> {
#Query("Your Query")
method();
}
This will allow you to perform CRUD operations.
If you want to use SQL for other purpose then you can use #Query
https://docs.spring.io/spring-data/jpa/docs/current/reference/html/#jpa.query-methods.at-query

This Question is opinion based.
However if you use an ORM Layer you can save time and code for the basic create read update and delete operations (see example from #iamsan).
The ORM layer makes sense in a lot of cases but there is no simple answer for your question. It depends on what you are doing and what you want to achieve.
If you want to implement ORM for learing, do it.
I am using it for a while now and it makes the persistense a lot easier. However you might have to deal with some configuration first.
These are just my two cents, you might get different answers from other users.

Best practices to separate concerns when accessing database

I've been trying to improve the separation of concerns when it comes to applications that access a database (via Hibernate).
On one of the applications I've been using the following approach:
Create services with business logic that have no connection/awareness of the database. They only communicate with GeneralDAO (and with other services);
A GeneralDAO responsible for CRUD/find operations, and with methods that involve more complex database queries.
The problems I see with this approach are:
GeneralDAO slowly becomes a God Object, when your application grows and require lots of specific database queries.
Sometimes the more specific Services become only proxies to the GeneralDAO, since the method is simple and only requires a database query. See example 1.
Example 1: Service is just a proxy
BookService manages things related to books in the Library application. Let's consider 2 methods:
archiveBook(Book)
findByIsbn(String isbn)
In archiveBook(Book) there might be considerable business logic involved - we might imagine this involves calls to:
distributionService.unbox(Book);
archivalBook.archive(Book);
librarianService.informNewBook(Book);
But findByIsbn(String isbn) is a lot more simple: it just needs to execute an SQL call to the database. So in this case I see two options:
Redirect the call to an object that can speak to the database to execute the query. For example generalDAO.findByIsbn(String isbn), that uses a db communication layer (in Hibernate it would use a sessionFactory or EntityManager) to execute the query.
Make that database layer available to the BookService, so that it executes the query itself
Questions/opinions (first number identifies the option above):
1.1. Isn't it strange to have 2 methods with the exact same signature, even if this is done to keep the BookService independent of the database layer (and ORM)?
1.2. How do you suggest avoiding The God anti-pattern? Would you suggest breaking the GeneralDAO into several DAOs depending on what the methods do? In this case, won't we risk needing to inject lots of DAOs into some Services, leading to a Service having too many objects injected into it?
2.1 What do you think of this alternative? Doesn't it break the "separation of concerns" by having the BookService be aware of objects at two different levels of abstraction (the DAO and the sessionFactory/EntityManager)?
3.1. Would you suggest any other approach/pattern/best practise?
Thanks!

1.2. How do you suggest avoiding The God anti-pattern? Would you suggest breaking the GeneralDAO into several DAOs depending on what
the methods do? In this case, won't we risk needing to inject lots of
DAOs into some Services, leading to a Service having too many objects
injected into it?
Generally, a DAO class should handle a specific entity.
If one of your entities require many kinds of queries, you could divide it again into two or more DAOs by grouping them by common concern (for example : reading, writing, selecting on agregates, etc...) as you said.
If you have too many queries and too many DAO, maybe, you should check if you don't write almost the same queries in several methods. It it the case, use specification or Criteria API to allow the client to custom queries by parameters. If the queries are really different, you have various processings. So, using multiple DAOs seems a suitable solution. It avoids increasing the complexity and the rise of god objects.
1.1. Isn't it strange to have 2 methods with the exact same signature, even if this is done to keep the BookService independent of the
database layer (and ORM)?
When you divide you app in logic layers, as you noticed, in some operations, some layers perform only delegation calls to the below layer. So in these cases, it is rather common to have method names which are the same. I would go further : it is a good practice to have the same name if it is just delegation call. Why do we create a variation in the conveyed behavior if they both address the same need?
2.1 What do you think of this alternative? Doesn't it break the "separation of concerns" by having the BookService be aware of objects
at two different levels of abstraction (the DAO and the
sessionFactory/EntityManager)?
BookService depends on DAOs but should not depend on sessionFactory/EntityManager which makes part of the DAO implementation.
BookService calls DAO which uses a sessionFactory/EntityManager.
If necessary, BookService may specify transactional details on itself or on its methods with #Transactional annotation.
3.1. Would you suggest any other approach/pattern/best practice?
As you use Spring, try to rely on the Sping JPA repository (less boiler plate to handle for common cases and extensible class)
Using specification or criteria patterns when you have several variants of some queries.

Correct way of using CRUD in JSF

I need advice on a few design principles regarding CRUD operations in my JSF project.
A very simple example:
I have a basic screen with a form that get submitted. In my bean I declare a database connection in my method and a string object which I populate with my script. I modify the string to get the data that have been submitted in the form. This is the way I was taught do it, but I'm suspecting it's not based on solid principles.
So I decided to start using prepared statements. Seems a bit better, but still not perfect in my mind.
My question is: instead of writing a new script for each CRUD method, is it better to perhaps create stored procedures instead, in my mind it looks like much neater code and perhaps has better readability.
Or is there an entirely different way of doing things? The only concerns I have is a very fragile OLTP database.

Your JSF,s should always redirect to a servlet which calls a service method, where you write all your business logic and call your Data Access Object to execute required sql query. U should never use your bean for database connection... You should use DataSource for your data base Connection. And yes a simple preparedStatement is enough to do. You should convert all your strings in your servlet only and then pass it to the next layer with the help of your bean which has your setters and getters for all your form fields.. And your DAo contains all the CRUD operations.

I don't like the idea of using stored procedures because they're hard to port and usually also hard to debug.
I've been working for years with something like this
JSF -> xhtml + #ViewScoped managed bean to accomodate the values
Stateless EJB for transactional methods called from managed beans
Entity DAOs, called from EJBs, reusing basic CRUD methods with generics. I think JPA here is great, specially when they use metamodel type-safe criteria (http://docs.oracle.com/javaee/6/tutorial/doc/gjivm.html)
Nowadays, it´s been easier to work with lightweight JavaEE stacks such as apache TomEE than using prepared statements.

Disadvantages of Object Relational Mapping

I am a fan of ORM - Object Relational Mapping and I have been using it with Rails for the past year and a half. Prior that, I use to write raw queries using JDBC and make Database do the heavy lifting via Stored Procedures. With ORM, I was initially happy to do stuff like coach.manager and manager.coaches which were very simple and easy to read.
But as time went by there were in-numerous associations creeping up and I ended up doing a.b.c.d which were firing queries in all directions, behind the scenes. With rails and ruby, the garbage collector went nuts and took insane time to load a very complex page which involves relatively lesser data. I had to replace this ORM style code by a simple Stored procedure and the result I saw was enormous. A page that took 50 seconds to load now takes only 2 seconds.
With this huge difference, should I continue using ORM? It is very clear it has severe overheads compared to a raw query.
In general, what are the general pitfalls of using an ORM framework like Hibernate, ActiveRecord?

An ORM is only a tool. If you don't use it correctly, you'll have bad results.
Nothing stops you from using dedicated HQL/criteria queries, with fetch joins or projections, to return the information that your page must display in as few queries as possible. This will take more or less the same time as dedicated SQL queries.
But of course, if you just get everything by ID and navigate through your objects without realizing how many queries it generates, it will lead to long loading times. The key is to know exactly what the ORM does behind the scene, and decide if it's appropriate or if another strategy must be adopted.

I think you've already identified the major tradeoff associated with ORM software. Every time you add a new layer of abstraction that tries to provide a generalized implementation of something that you used to do by hand there is going to be some loss of performance/efficiency.
As you noted, traversing multiple relationships such as a.b.c.d can be inefficient, because most ORM software will be doing an independent database query for each . along the way. But I'm not sure that means you should eliminate ORM altogether. Most ORM solutions (or at least, certainly Hibernate) allow you to specify custom queries where you can bring back exactly what you want in a single database operation. This should be about as fast as your dedicated SQL.
Really the issue is about understanding how the ORM layer is working behind the scenes, and realizing that while something like a.b.c.d is simple to write, what it causes the ORM layer to do as it is evaluated is not. As a general rule I always go with the simplest possible approach to begin, and then write optimized queries in areas where it makes sense/where it is obvious that the simple approach will not scale.

I'd say, one should use the appropriate tool for different tasks.
E.g., for CRUD operations, ORM frameworks like Hibernate can speed up development and it will perform well enough. Sometimes you need to do some necessary tweaks to achieve acceptable performance. I'm not sure, your task (what took 50 sec with Hibernate) could not be done properly with Hibernate, because you did not provide us with the details.
On the other hand, for example bulk operations involving hundreds of thousands of records is not the type of task you'd expect Hibernate will do without significant performance penalty.

As it was mentioned already, ORM is only a tool and you can use it eiter good or bad.
One of the most typical performance problems in ORMs is 1+N queries problem. It is caused by loading additional objects for each of objects from the list. This is caused by eager fetch of 1-to-n-relation entities for each element on list, the dealing is using HQL queries, specifying fields in projection or marking fetching 1-to-n relations to lazy.
Any time, you must exactly know what the ORM is doing in order to achieve good performance. Not understanding what operations are done in background is a way to disaster (slow, buggy and hard to analyze code because of unnecessary and wrongly written work-arounds).

I'm with Petar from your comments regarding the lazy fetching. Say you have an html table filled fields from object a.b.c.d. You could find your framework round-tripping the database thousands of times(possibly many more) . The disadvantage of ORM in this case is you have to read the documentation thoroughly. Most frameworks support disabling lazy fetching and many even support adding your own processing logic to bind the data set.
The net out is that almost any ORM is almost undoubtedly better than anything you are going to write yourself. You will find yourself saddled with maintaining huge libraries of boilerplate or worse writing the same code over and over again.

We are currently investigating to switch from our own data store layer with clean separation of transfer objects and data access objects to JPA. We used a generator to create the TOs, the DAOs and the SQL DDL as well from some documentation in docbook format. By this all of our stuff from documentation, the database structure and the generated Java classes where always in sync with a good documentation of the database itself.
What we discovered so far by using JPA:
Foreign key references cannot be used for imports, some special
queries and so on because they must not be placed in a managed
entity. JPA only allows the target class there.
Access to some user session scope is difficult upto impossible. We
still have no clue how to get the users id into the column
'userWhoLastMadeAnUpdate' in some PrePersist method.
Something expected to be quite easy with an ORM, namely "class
mapping" does not work at all. We are using HalDateTime
(http://sourceforge.net/projects/haldatetime/) internally.
Especially in the client. Mapping it with JPA directly is not
possible although HalDateTime supports it. Due to JPA restrictions
we have to use two fields in the entity.
JPA uses either one XML file to describe the mapping. So you have to
look at least into two files to even understand the relationship
between the Java class and the database. And the XML file becomes
huge for large applications.
Alternatively ORMs provide annotations in the Java class itself. So
its easier to learn and understand the relationship. But it forces
you to see all that database stuff in the client layer (which
completely breaks a proper layering).
You will have to restrict yourself to stay as close to a clean
database structure as anyhow possible. Otherwise you will for sure
end up with a mess of queries and statements by the ORM.
Use an ORM which provides a query language which is close to SQL
itself (JPA seems quite acceptable here). An ORM induced language
makes supporting a large application really expensive.

Migrating from hand-written persistence layer to ORM

We are currently evaluating options for migrating from hand-written persistence layer to ORM.
We have a bunch of legacy persistent objects (~200), that implement simple interface like this:
interface JDBC {
public long getId();
public void setId(long id);
public void retrieve();
public void setDataSource(DataSource ds);
}
When retrieve() is called, object populates itself by issuing handwritten SQL queries to the connection provided using the ID it received in the setter (this usually is the only parameter to the query). It manages its statements, result sets, etc itself. Some of the objects have special flavors of retrive() method, like retrieveByName(), in this case a different SQL is issued.
Queries could be quite complex, we often join several tables to populate the sets representing relations to other objects, sometimes join queries are issued on-demand in the specific getter (lazy loading). So basically, we have implemented most of the ORM's functionality manually.
The reason for that was performance. We have very strong requirements for speed, and back in 2005 (when this code was written) performance tests has shown that none of mainstream ORMs were that fast as hand-written SQL.
The problems we are facing now that make us think of ORM are:
Most of the paths in this code are well-tested and are stable. However, some rarely-used code is prone to result set and connection leaks that are very hard to detect
We are currently squeezing some additional performance by adding caching to our persistence layer and it's a huge pain to maintain the cached objects manually in this setup
Support of this code when DB schema changes is a big problem.
I am looking for an advice on what could be the best alternative for us. As far as I know, ORMs has advanced in last 5 years, so it might be that now there's one that offers an acceptable performance. As I see this issue, we need to address those points:
Find some way to reuse at least some of the written SQL to express mappings
Have the possibility to issue native SQL queries without the necessity to manually decompose their results (i.e. avoid manual rs.getInt(42) as they are very sensitive to schema changes)
Add a non-intrusive caching layer
Keep the performance figures.
Is there any ORM framework you could recommend with regards to that?
UPDATE To give a feeling of what kind of performance figures we are talking about:
The backend database is TimesTen, in-memory database that runs on the same machine as the JVM
We found out that changing rs.getInt("column1") to rs.getInt(42) brings the performance increase we consider significant.

If you want a standard persistence layer that lets you issue native SQL queries, consider using iBATIS. It's a fairly thin mapping between your objects and SQL. http://ibatis.apache.org/
For caching and lazy joins, Hibernate might be a better choice. I haven't used iBATIS for these purposes.
Hibernate provides a lot of flexibility in allowing you to specify certain defaults for lazy loading as you traverse your object graph, yet also pre-fetch data with SQL or HQL queries to your heart's content when you need better-known load times. However, the conversion effort will be complicated for you as it has a fairly high bar to entry in terms of learning and configuration. Annotations made this easier for me.
Two benefits you didn't mention about switching to a standard framework:
(1) running down bugs becomes easier when you have a wealth of sites and forums out there to support you.
(2) new hires are cheaper, easier and faster.
Good luck in addressing your performance and usability issues. The tradeoffs you point out are very common. Sorry if I evangelized.

For the bulk of your queries, I'd go with hibernate. It's widely used,well documented, and generally performant. You can drop down to hand-written SQL if hibernate isn't producing efficient enough queries. Hibernate gives you a lot of control in specifying the table names and columns that the domain objects map to, and in most cases you can retro fit it to an exisitng schema.
Find some way to reuse at least some of the written SQL to express mappings
The mappings are expressed in JPA using annotations. You can use the existing SQL as a guide when creating JPQL queries.
Add a non-intrusive caching layer
Caching in hibernate is automatic and transparent, unless you specifically choose to get involved. You can mark entities as read only, or evict from the cache, control when changes are flushed to the database (inside a transaction of course - automatic use of batching improves performance when network latency is a concern.)
Have the possibility to issue native
SQL queries without the necessity to
manually decompose their results (i.e.
avoid manual rs.getInt(42) as they
are very sensitive to schema changes)
Hibernate allows you to write SQL, and have this mapped to your entities. You don't deal with the ResultSet directly - hibernate takes care of the deconstruction into your entity. See Chpt 16, Native SQL in the hibernate manual.
Support of this code when DB schema changes is a big problem.
Managing schema changes can still be a pain, since you now effectively have two schemata - the database schema and the JPA mapping (an object schema). if you choose to let hibernate generate the db schema and move your data to that, you are no longer directly responsible for what goes into the database, and so you are then faced with manging automatic changes to a machine generated schema. There are tools that can assist, such as dbmigrate, and liquibase, but it's no walk in the park. Conversely, if you are managing the db schema by hand, then you will have to carefully recraft your entities, JPA annotations and queries to accomodate the schema changes. Adding columns and new entities is relatively trivial, but more complex changes such as changing a single property to a collection of properties, or restructing an object hierarchy will involve considerably more extensive changes. There is no easy way out of this - either the db or hibernate is the "master" that decides the schema, and when one changes, the other must follow. The code changes aren't so bad - in my experience, it's migrating the data that's difficult. But this is a basic issue with databases, and will be present in any solution you choose.
So, to sum up, I'd go with hibernate, and use the JPA interface.

I've recently drilled through a bunch of Java ORMs and didn't come up with anything much better than Hibernate. Hibernate's performance may get you there and satisfy your performance goals.
Lots of people think that moving to Hibernate will make everything so awesome, but it's really just moving a set of problems from JDBC queries into Hibernate tuning. Read a bunch of books or (better) hire a "Hibernate guy" to come in and help.
During your refactor, I'd recommend using JPA so you can un-plug and re-plug a new persistence provider when the Next Big Thing comes along (or you move to Oracle)

Do you really need to migrate? What's forcing you to move? Is there some REAL need here or someone just inventing work (an 'Astronaut architect')?
I agree with the above answers though - if you HAVE to move - Hibernate or iBatis are good choices. iBatis especially if you want to stay 'closer' to the SQL.

If you need more performance: drop the database (for on-line work) and handle the persistence direct. Adding caching is not going to help you with a TimesTen DB, it just adds an extra copy (slowing you down).
You might want to take a look at GemFire.

There is a lot of good advice already in here that I won't repeat. The only thing I didn't see suggested that might work for you is caching reference data in memory.
I have done quite a bit of this in the past and it does save a lot of time. If you have a large number of fairly static reference tables, load them all into memory at startup time and refresh them every couple minutes. That way you're not hitting the DB over and over again for data that never changes.