We Keep Coding

CSV file dynamic map to Database Tables using Hibernate

We have a requirement where around 25 CSV files would come each day & stored the Database in equivalent table structure.
Any of CSV file column structure could change in future by add new /remove columns & underlying DB table would align to the new format, without code change or redeployment.
Here are the choice of tech.
SpringBoot as Run time
Hibernate as JPA/DB Inetraction
Oracle DB as database
If using Hibernate, how to achieving this dynamic column management of the table as per the incoming CSV?
As far as I know, Hibernate would have Java Entity classes equivalent to the Table , which will be used to persists data. Any table change need Entity class change too.
Possible solution could be
just define basic JPA Entity & table structure (like ids & FKs linking to other tables etc) for CSV equivalent table,
then on arrival of CSV files, add the columns to the table by running the ALTER table command from application
In future 1st CSVs, if column added/removed , use similar alter commands
Is this achievable by Hibernate?
Or any other product better suited for this kind of tasks.

Task definition
We will have to implement a mechanism allowing for creating/deleting custom fields in real time avoiding the application restart, add a value into it and make sure the value is present in the application database. Besides we will have to make sure that the custom field can be used in queries.
Solution
Domain Model
We will first need a business entity class which we will experiment with. Let is be Contact class. There will be two persistent fields: id and name.
However besides these permanent and unchangeable fields the class should be some sort of construction to store values of custom fields. Map would be an ideal construction for this.
Let's create a base class for all business entities supporting custom fields - CustomizableEntity, that contains Map CustomProperties to work with custom fields:
package com.enterra.customfieldsdemo.domain;
import java.util.Map;
import java.util.HashMap;
public abstract class CustomizableEntity {
private Map customProperties;
public Map getCustomProperties() {
if (customProperties == null)
customProperties = new HashMap();
return customProperties;
}
public void setCustomProperties(Map customProperties) {
this.customProperties = customProperties;
}
public Object getValueOfCustomField(String name) {
return getCustomProperties().get(name);
}
public void setValueOfCustomField(String name, Object value) {
getCustomProperties().put(name, value);
}
}
Step 1 - base class CustomizableEntity
Inherit our class Contact from this base class:
package com.enterra.customfieldsdemo.domain;
import com.enterra.customfieldsdemo.domain.CustomizableEntity;
public class Contact extends CustomizableEntity {
private int id;
private String name;
public int getId() {
return id;
}
public void setId(int id) {
this.id = id;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
}
Step 2 - Class Contact inherited from CustomizableEntity.
We should not forget about the mapping file for this class:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE hibernate-mapping PUBLIC "-//Hibernate/Hibernate Configuration DTD//EN"
"http://hibernate.sourceforge.net/hibernate-mapping-3.0.dtd">
<hibernate-mapping auto-import="true" default-access="property" default-cascade="none" default-lazy="true">
<class abstract="false" name="com.enterra.customfieldsdemo.domain.Contact" table="tbl_contact">
<id column="fld_id" name="id">
<generator class="native"/>
</id>
<property name="name" column="fld_name" type="string"/>
<dynamic-component insert="true" name="customProperties" optimistic-lock="true" unique="false" update="true">
</dynamic-component>
</class>
</hibernate-mapping>
Step 3 - Mapping Class Contact.
Please note that properties id and name are done as all ordinary properties, however for customProperties we use a tag . Documentation on Hibernate 3.2.0GA says that the point of a dynamic-component is:
"The semantics of a mapping are identical to . The advantage of this kind of mapping is the ability to determine the actual properties of the bean at deployment time, just by editing the mapping document. Runtime manipulation of the mapping document is also possible, using a DOM parser. Even better, you can access (and change) Hibernate's configuration-time metamodel via the Configuration object."
Based on this regulation from Hibernate documentation we will be building this function mechanism.
HibernateUtil and hibernate.cfg.xml
After we are defined with the domain model of our application we have to create necessary conditions for Hibernate framework functioning. For this we have to create a configuration file hibernate.cfg.xml and class to work with the core Hibernate functions.
<?xml version='1.0' encoding='utf-8'?>
<!DOCTYPE hibernate-configuration
PUBLIC "-//Hibernate/Hibernate Configuration DTD//EN"
"http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<property name="show_sql">true</property>
<property name="dialect">
org.hibernate.dialect.MySQLDialect</property>
<property name="cglib.use_reflection_optimizer">true</property>
<property name="hibernate.connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="hibernate.connection.url">jdbc:mysql://localhost:3306/custom_fields_test</property>
<property name="hibernate.connection.username">root</property>
<property name="hibernate.connection.password"></property>
<property name="hibernate.c3p0.max_size">50</property>
<property name="hibernate.c3p0.min_size">0</property>
<property name="hibernate.c3p0.timeout">120</property>
<property name="hibernate.c3p0.max_statements">100</property>
<property name="hibernate.c3p0.idle_test_period">0</property>
<property name="hibernate.c3p0.acquire_increment">2</property>
<property name="hibernate.jdbc.batch_size">20</property>
<property name="hibernate.hbm2ddl.auto">update</property>
</session-factory>
</hibernate-configuration>
Step 4 - Hibernate configuration file.
The file hibernate.cfg.xml does not contain anything noticeable except for this string:
<property name="hibernate.hbm2ddl.auto">update</property>
Step 5 - using auto-update.
Later we will explain in details on its purpose and tell more how we can go without it. There are several ways to implement class HibernateUtil. Our implementation will differ a bit from well known due to changes into Hibernate configuration.
package com.enterra.customfieldsdemo;
import org.hibernate.*;
import org.hibernate.mapping.PersistentClass;
import org.hibernate.tool.hbm2ddl.SchemaUpdate;
import org.hibernate.cfg.Configuration;
import com.enterra.customfieldsdemo.domain.Contact;
public class HibernateUtil {
private static HibernateUtil instance;
private Configuration configuration;
private SessionFactory sessionFactory;
private Session session;
public synchronized static HibernateUtil getInstance() {
if (instance == null) {
instance = new HibernateUtil();
}
return instance;
}
private synchronized SessionFactory getSessionFactory() {
if (sessionFactory == null) {
sessionFactory = getConfiguration().buildSessionFactory();
}
return sessionFactory;
}
public synchronized Session getCurrentSession() {
if (session == null) {
session = getSessionFactory().openSession();
session.setFlushMode(FlushMode.COMMIT);
System.out.println("session opened.");
}
return session;
}
private synchronized Configuration getConfiguration() {
if (configuration == null) {
System.out.print("configuring Hibernate ... ");
try {
configuration = new Configuration().configure();
configuration.addClass(Contact.class);
System.out.println("ok");
} catch (HibernateException e) {
System.out.println("failure");
e.printStackTrace();
}
}
return configuration;
}
public void reset() {
Session session = getCurrentSession();
if (session != null) {
session.flush();
if (session.isOpen()) {
System.out.print("closing session ... ");
session.close();
System.out.println("ok");
}
}
SessionFactory sf = getSessionFactory();
if (sf != null) {
System.out.print("closing session factory ... ");
sf.close();
System.out.println("ok");
}
this.configuration = null;
this.sessionFactory = null;
this.session = null;
}
public PersistentClass getClassMapping(Class entityClass){
return getConfiguration().getClassMapping(entityClass.getName());
}
}
Step 6 - HibernateUtils class.
Alongside with usual methods like getCurrentSession(), getConfiguration(), which is necessary for regular work of the application based on Hibernate, we also have implemented such methods as: reset() and getClassMapping(Class entityClass). In the method getConfiguration(), we configure Hibernate and add class Contact into the configuration.
Method reset() has been used to close all used by Hibernate resources and clearing all of its settings:
public void reset() {
Session session = getCurrentSession();
if (session != null) {
session.flush();
if (session.isOpen()) {
System.out.print("closing session ... ");
session.close();
System.out.println("ok");
}
}
SessionFactory sf = getSessionFactory();
if (sf != null) {
System.out.print("closing session factory ... "); sf.close();
System.out.println("ok");
}
this.configuration = null;
this.sessionFactory = null;
this.session = null;
}
Step 7 - method reset()
Method getClassMapping(Class entityClass) returns object PersistentClass, that contains full information on mapping the related entity. In particular the manipulations with the object PersistentClass allow modifying the set of attributes of the entity class in the run-time.
public PersistentClass getClassMapping(Class entityClass){
return
getConfiguration().getClassMapping(entityClass.getName());
}
Step 8 - method getClassMapping(Class entityClass).
Manipulations with mapping
Once we have the business entity class (Contact) available and the main class to interact with Hibernate we can start working. We can create and save samples of the Contact class. We can even place some data into our Map customProperties, however we should be aware that this data (stored in Map customProperties) are not saved to the DB.
To have the data saved we should provide for the mechanism of creating custom fields in our classs and make it the way Hibernate knows how to work with them.
To provide for class mapping manipulation we should create some interface. Let's call it CustomizableEntityManager. Its name should reflect the purpose of the interface managing a business entity, its contents and attributes:
package com.enterra.customfieldsdemo;
import org.hibernate.mapping.Component;
public interface CustomizableEntityManager {
public static String CUSTOM_COMPONENT_NAME = "customProperties";
void addCustomField(String name);
void removeCustomField(String name);
Component getCustomProperties();
Class getEntityClass();
}
Step 9 - Interface CustomizableEntityManager
The main methods for the interface are: void addCustomField(String name) and void removeCustomField(String name). These should created and remove our custom field in the mapping of the corresponding class.
Below is the way to implement the interface:
package com.enterra.customfieldsdemo;
import org.hibernate.cfg.Configuration;
import org.hibernate.mapping.*;
import java.util.Iterator;
public class CustomizableEntityManagerImpl implements CustomizableEntityManager {
private Component customProperties;
private Class entityClass;
public CustomizableEntityManagerImpl(Class entityClass) {
this.entityClass = entityClass;
}
public Class getEntityClass() {
return entityClass;
}
public Component getCustomProperties() {
if (customProperties == null) {
Property property = getPersistentClass().getProperty(CUSTOM_COMPONENT_NAME);
customProperties = (Component) property.getValue();
}
return customProperties;
}
public void addCustomField(String name) {
SimpleValue simpleValue = new SimpleValue();
simpleValue.addColumn(new Column("fld_" + name));
simpleValue.setTypeName(String.class.getName());
PersistentClass persistentClass = getPersistentClass();
simpleValue.setTable(persistentClass.getTable());
Property property = new Property();
property.setName(name);
property.setValue(simpleValue);
getCustomProperties().addProperty(property);
updateMapping();
}
public void removeCustomField(String name) {
Iterator propertyIterator = customProperties.getPropertyIterator();
while (propertyIterator.hasNext()) {
Property property = (Property) propertyIterator.next();
if (property.getName().equals(name)) {
propertyIterator.remove();
updateMapping();
return;
}
}
}
private synchronized void updateMapping() {
MappingManager.updateClassMapping(this);
HibernateUtil.getInstance().reset();
// updateDBSchema();
}
private PersistentClass getPersistentClass() {
return HibernateUtil.getInstance().getClassMapping(this.entityClass);
}
}
Step 10 - implementing interface CustomizableEntityManager
First of all we should point out that when creating class CustomizableEntityManager we specify the business entity class the manager will operate. This class is passed as a parameter to designer CustomizableEntityManager:
private Class entityClass;
public CustomizableEntityManagerImpl(Class entityClass) {
this.entityClass = entityClass;
}
public Class getEntityClass() {
return entityClass;
}
Step 11 - class designer CustomizableEntityManagerImpl
Now we should get more interested in how to implement method void addCustomField(String name):
public void addCustomField(String name) {
SimpleValue simpleValue = new SimpleValue();
simpleValue.addColumn(new Column("fld_" + name));
simpleValue.setTypeName(String.class.getName());
PersistentClass persistentClass = getPersistentClass();
simpleValue.setTable(persistentClass.getTable());
Property property = new Property();
property.setName(name);
property.setValue(simpleValue);
getCustomProperties().addProperty(property);
updateMapping();
}
Step 12 - creating custom field.
As we can see from the implementation, Hibernate offers more options in working with properties of persistent objects and their representation in the DB. As per the essence of the method:
1) We create class SimpleValue that allow us to denote how the value of this custom field will be stored in the DB in which field and table of the DB:
SimpleValue simpleValue = new SimpleValue();
simpleValue.addColumn(new Column("fld_" + name));
simpleValue.setTypeName(String.class.getName());
PersistentClass persistentClass = getPersistentClass();
simpleValue.setTable(persistentClass.getTable());
Step 13 - creating new column of the table.
2) We create a property of the persistent object and add a dynamic component into it (!), that we have planned to be used for this purpose:
Property property = new Property()
property.setName(name)
property.setValue(simpleValue)
getCustomProperties().addProperty(property)
Step 14 - creating object property.
3) And finally we should make our application perform certain changes in the xml files and update the Hibernate configuration. This can be done via method updateMapping();
It is necessary to clarify the purpose of another two get-methods which have been used in the code above. The first method is getCustomProperties():
public Component getCustomProperties() {
if (customProperties == null) {
Property property = getPersistentClass().getProperty(CUSTOM_COMPONENT_NAME);
customProperties = (Component) property.getValue();
}
return customProperties;
}
Step 15 - getting CustomProperties as Component.
This method finds and returns object Component corresponding to the tag in the mapping of our business entity.
The second method is updateMapping():
private synchronized void updateMapping() {
MappingManager.updateClassMapping(this);
HibernateUtil.getInstance().reset();
// updateDBSchema();
}
Step 16 - method updateMapping().
The method is in charge for storing the updated mapping of the persistent class and updates the configuration status of Hibernate to make further changes that we make valid when the changes take effect.
By the way we should get back to the string:
<property name="hibernate.hbm2ddl.auto">update</property>
of the Hibernate configuration. If this string was missing we would have to launch executing updates of the DB schema using hibernate utilities. However using the setting allows us to avoid this.
Saving mapping
Modifications to mapping made in run-time do not save by themselves into the corresponding xml mapping file and to make the changes to get activated at next launch of the application we need to manually save changes to the corresponding mapping file.
To do this we will be using class MappingManager the main purpose of which is to save mapping of the designated business entity to its xml mapping file:
package com.enterra.customfieldsdemo;
import com.enterra.customfieldsdemo.domain.CustomizableEntity;
import org.hibernate.Session;
import org.hibernate.mapping.Column;
import org.hibernate.mapping.Property;
import org.hibernate.type.Type;
import org.w3c.dom.Document;
import org.w3c.dom.Element;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import java.util.Iterator;
public class MappingManager {
public static void updateClassMapping(CustomizableEntityManager entityManager) {
try {
Session session = HibernateUtil.getInstance().getCurrentSession();
Class<? extends CustomizableEntity> entityClass = entityManager.getEntityClass();
String file = entityClass.getResource(entityClass.getSimpleName() + ".hbm.xml").getPath();
Document document = XMLUtil.loadDocument(file);
NodeList componentTags = document.getElementsByTagName("dynamic-component");
Node node = componentTags.item(0);
XMLUtil.removeChildren(node);
Iterator propertyIterator = entityManager.getCustomProperties().getPropertyIterator();
while (propertyIterator.hasNext()) {
Property property = (Property) propertyIterator.next();
Element element = createPropertyElement(document, property);
node.appendChild(element);
}
XMLUtil.saveDocument(document, file);
} catch (Exception e) {
e.printStackTrace();
}
}
private static Element createPropertyElement(Document document, Property property) {
Element element = document.createElement("property");
Type type = property.getType();
element.setAttribute("name", property.getName());
element.setAttribute("column", ((Column)
property.getColumnIterator().next()).getName());
element.setAttribute("type",
type.getReturnedClass().getName());
element.setAttribute("not-null", String.valueOf(false));
return element;
}
}
Step 17 - the utility to update mapping of the persistent class.
The class literally performs the following:
Defines a location and loads xml mapping for the designated business entity into the DOM Document object for further manipulations with it;
Finds the element of this document . In particular here we store the custom fields and its contents we change;
Delete (!) all embedded elements from this element;
For any persistent property contained in our component that is in charge for the custom fields storage, we create a specific document element and define attributes for the element from the corresponding property;
Save this newly created mapping file.
When manipulating XML we use (as we can see from the code) class XMLUtil, that in general can be implemented in any way though it should correctly load and save the xml file.
Our implementation is given at the Step below:
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import org.w3c.dom.Document;
import org.xml.sax.SAXException;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.transform.TransformerException;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.Transformer;
import javax.xml.transform.OutputKeys;
import javax.xml.transform.stream.StreamResult;
import javax.xml.transform.dom.DOMSource;
import java.io.IOException;
import java.io.FileOutputStream;
public class XMLUtil {
public static void removeChildren(Node node) {
NodeList childNodes = node.getChildNodes();
int length = childNodes.getLength();
for (int i = length - 1; i > -1; i--)
node.removeChild(childNodes.item(i));
}
public static Document loadDocument(String file)
throws ParserConfigurationException, SAXException, IOException {
DocumentBuilderFactory factory =DocumentBuilderFactory.newInstance();
DocumentBuilder builder = factory.newDocumentBuilder();
return builder.parse(file);
}
public static void saveDocument(Document dom, String file)
throws TransformerException, IOException {
TransformerFactory tf = TransformerFactory.newInstance();
Transformer transformer = tf.newTransformer();
transformer.setOutputProperty(OutputKeys.DOCTYPE_PUBLIC, dom.getDoctype().getPublicId());
transformer.setOutputProperty(OutputKeys.DOCTYPE_SYSTEM, dom.getDoctype().getSystemId());
DOMSource source = new DOMSource(dom);
StreamResult result = new StreamResult();
FileOutputStream outputStream = new FileOutputStream(file);
result.setOutputStream(outputStream);
transformer.transform(source, result);
outputStream.flush();
outputStream.close();
}
}
Source: Please refer this article for more detail

Hibernate performance tuning tips

I'm developing a small application, which consists of a single table.
I am using technologies are:
NetBeans 8.1
Java 8
Hibernate 4.3.x
Informix
Primefaces 5
I had to investigate a time to connect with Informix Hibernate, but I got it, and the application displays the list with the requested data correctly.
The problem arises with the performance of Hibernate, which is very poor, especially considering that the table contains only 36000 records.
On each page change takes about 6 or 7 seconds.
I have been researching in the official documentation of Hibernate, but can not find concrete examples to improve performance.
Herewith the application code:
hibernate.cfg.xml:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE hibernate-configuration PUBLIC "-//Hibernate/Hibernate Configuration DTD 3.0//EN" "http://www.hibernate.org/dtd/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<property name="hibernate.dialect">org.hibernate.dialect.InformixDialect</property>
<property name="hibernate.connection.driver_class">com.informix.jdbc.IfxDriver</property>
<property name="hibernate.connection.url">jdbc:informix-sqli://127.0.0.1:1526/chicho:INFORMIXSERVER=ol_chicho</property>
<!--<property name="hibernate.connection.datasource">jdbc/votacion</property>-->
<property name="hibernate.connection.username">informix</property>
<property name="hibernate.connection.password">informix</property>
<property name="hibernate.connection.autocommit">true</property>
<property name="hibernate.current_session_context_class">thread</property>
<property name="hibernate.default_schema">informix</property>
<property name="hibernate.show_sql">true</property>
<property name="hibernate.hbm2ddl.auto">validate</property>
<property name="hibernate.cache.use_second_level_cache">true</property>
<mapping resource="pojos/Xxpuedenvotar1.hbm.xml"/>
</session-factory>
Pojo:
package pojos;
// Generated 23/08/2016 22:07:42 by Hibernate Tools 4.3.1
/**
* Xxpuedenvotar1 generated by hbm2java
*/
public class Xxpuedenvotar1 implements java.io.Serializable {
private Integer nroaccionista;
private Short estado;
private Integer idcliente;
private String razonSocial;
private Short idlocalidad;
private Short zona;
private String calle;
private String puerta;
private String localidad;
public Xxpuedenvotar1() {
}
public Xxpuedenvotar1(Integer nroaccionista) {
this.nroaccionista = nroaccionista;
}
public Xxpuedenvotar1(Integer nroaccionista, Short estado, Integer idcliente, String razonSocial, Short idlocalidad, Short zona, String calle, String puerta, String localidad) {
this.nroaccionista = nroaccionista;
this.estado = estado;
this.idcliente = idcliente;
this.razonSocial = razonSocial;
this.idlocalidad = idlocalidad;
this.zona = zona;
this.calle = calle;
this.puerta = puerta;
this.localidad = localidad;
}
public Integer getNroaccionista() {
return this.nroaccionista;
}
public void setNroaccionista(Integer nroaccionista) {
this.nroaccionista = nroaccionista;
}
public Short getEstado() {
return this.estado;
}
public void setEstado(Short estado) {
this.estado = estado;
}
public Integer getIdcliente() {
return this.idcliente;
}
public void setIdcliente(Integer idcliente) {
this.idcliente = idcliente;
}
public String getRazonSocial() {
return this.razonSocial;
}
public void setRazonSocial(String razonSocial) {
this.razonSocial = razonSocial;
}
public Short getIdlocalidad() {
return this.idlocalidad;
}
public void setIdlocalidad(Short idlocalidad) {
this.idlocalidad = idlocalidad;
}
public Short getZona() {
return this.zona;
}
public void setZona(Short zona) {
this.zona = zona;
}
public String getCalle() {
return this.calle;
}
public void setCalle(String calle) {
this.calle = calle;
}
public String getPuerta() {
return this.puerta;
}
public void setPuerta(String puerta) {
this.puerta = puerta;
}
public String getLocalidad() {
return this.localidad;
}
public void setLocalidad(String localidad) {
this.localidad = localidad;
}
}
DAO:
package Dao;
import Interfaces.InterfazSocios;
import java.util.List;
import org.hibernate.Query;
import org.hibernate.Session;
import pojos.Xxpuedenvotar1;
/**
*
* #author Gustavo
*/
public class DaoSocios implements InterfazSocios {
private List<Xxpuedenvotar1> listaSocios;
#Override
public List<Xxpuedenvotar1> verTodos(Session sesion) throws Exception {
String hql = "FROM Xxpuedenvotar1 ORDER BY NroAccionista";
//Query consulta = sesion.createQuery(hql).setCacheable(true);
this.listaSocios = sesion.createCriteria(Xxpuedenvotar1.class).list();
//this.listaSocios = consulta.list();
return this.listaSocios;
}
}
I think with these files is sufficient for analysis, since the application works well, with the exception of its slowness.
Thank in advance for your kind attention.

First of all, there are many things you can do to speed up Hibernate. Check out these High-Performance Hibernate Tips, or this High-Performance Hibernate video presentation.
Now, back to your question. You are using the DriverManagerConnectionProvider which only provides a rudimentary connection pooling solution. Better use HikariCP since it's the fastest one, and it's also available via the hibernate-hikaricp Maven dependency.
Related to your statement:
The problem arises with the performance of Hibernate, which is very
poor, especially considering that the table contains only 36000
records.
On each page change takes about 6 or 7 seconds.
Why would you want to fetch 36k records in a single shot?
You can't display them into a UI? That's why we have data pagination after all.
Even for a batch processor, you are better off splitting the whole workload into multiple smaller data sets that allow you to avoid long-running transactions, and even split the load among multiple worker threads.

Can try any of these or all of these depending on the configuration, availability of modifying the configuration properties.
hibernate.show_sql mark this to false, once you are done with your analysis of query formation. This takes time to log.
You can change connection pool size configuration, if you are using one
<property name="hibernate.c3p0.min_size">5</property>
<property name="hibernate.c3p0.max_size">20</property>
This helps in attaining the connection quickly from the pool. Avoids time to get the connection.
If you have huge data, Please mark the respective column in JPA class with #OrderBy.
Hibernate search provides #SortableField to annotate the field, This indexes internally and helps in retrieving the sorted results.
[EDIT]
If Integer nroaccionista is primary key, it is indexed by default. Mark the column in JPA class with #Id and the DDL creation statement generated with create the index by default. Indexed column helps in better search and sort results.
Hope this helps.

Why am I getting a PersistenceException of "type not a registered entity"?

I'm trying to create and save some questions / answers into a DB using Ebean in the play framework 2.3.8.
I have a question.java class in my model-package:
package model;
import java.util.List;
import javax.persistence.Entity;
import javax.persistence.Id;
import play.db.ebean.Model;
#Entity
public class Question extends Model {
// Auto-generated serial
private static final long serialVersionUID = 1L;
#Id
public String questionID;
public String questionText;
public Integer voteScore;
public String userID;
// Constructor
public Question(String inputID, String inputQuestionText, Integer inputVoteScore, String inputUserID){
this.questionID = inputID;
this.questionText = inputQuestionText;
this.voteScore = inputVoteScore;
this.userID = inputUserID;
}
// Default Constructor is needed for the form, else the play framework breaks!
public Question(){
}
public static Finder<String,Question> find = new Finder<String, Question>(
String.class, Question.class
);
public static List<Question> all() {
return find.all();
}
public static void create(Question question) {
question.save();
}
public void delete(String id) {
find.ref(id).delete();
}
// Mostly for debugging
public String toString(){
return "questionID = " + questionID + " questionText = " + questionText + " voteScore = " + voteScore + " userID = " + userID;
}
}
In my application.java controller-class I define a new question and try to put it into the DB via using the create method:
Question question1 = new Question("xyz", "Do Androids dream?", 127, "Marcus");
Question.create(question1);
I have used the #Entity tag and I have uncommented those three lines in the application.conf:
db.default.driver=org.h2.Driver
db.default.url="jdbc:h2:mem:play"
ebean.default="models.*"
Now, when I start my project, I get this error in my browser:
[PersistenceException: The type [class model.Question] is not a registered entity? If you don't explicitly list the entity classes to use Ebean will search for them in the classpath. If the entity is in a Jar check the ebean.search.jars property in ebean.properties file or check ServerConfig.addJar().]
It points to this line:
Question.java:48
47 public static void create(Question question) {
48 question.save();
49 }
Console output:
play.api.Application$$anon$1: Execution exception[[PersistenceException: The type [class model.Question] is not a registered entity? If you don't explicitly list the entity classes to use Ebean will s
earch for them in the classpath. If the entity is in a Jar check the ebean.search.jars property in ebean.properties file or check ServerConfig.addJar().]]
at play.api.Application$class.handleError(Application.scala:296) ~[play_2.11-2.3.8.jar:2.3.8]
at play.api.DefaultApplication.handleError(Application.scala:402) [play_2.11-2.3.8.jar:2.3.8]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:320) [play_2.11-2.3.8.jar:2.3.8]
at play.core.server.netty.PlayDefaultUpstreamHandler$$anonfun$3$$anonfun$applyOrElse$4.apply(PlayDefaultUpstreamHandler.scala:320) [play_2.11-2.3.8.jar:2.3.8]
at scala.Option.map(Option.scala:145) [scala-library-2.11.1.jar:na]
Caused by: javax.persistence.PersistenceException: The type [class model.Question] is not a registered entity? If you don't explicitly list the entity classes to use Ebean will search for them in the
classpath. If the entity is in a Jar check the ebean.search.jars property in ebean.properties file or check ServerConfig.addJar().
at com.avaje.ebeaninternal.server.persist.DefaultPersister.createRequest(DefaultPersister.java:1299) ~[avaje-ebeanorm-3.3.4.jar:na]
at com.avaje.ebeaninternal.server.persist.DefaultPersister.saveRecurse(DefaultPersister.java:273) ~[avaje-ebeanorm-3.3.4.jar:na]
at com.avaje.ebeaninternal.server.persist.DefaultPersister.save(DefaultPersister.java:244) ~[avaje-ebeanorm-3.3.4.jar:na]
at com.avaje.ebeaninternal.server.core.DefaultServer.save(DefaultServer.java:1610) ~[avaje-ebeanorm-3.3.4.jar:na]
at com.avaje.ebeaninternal.server.core.DefaultServer.save(DefaultServer.java:1600) ~[avaje-ebeanorm-3.3.4.jar:na]

Your configuration has
ebean.default="models.*"
which, if I'm interpreting the documentation correctly, means that it searches for entities in the modelspackage.
But the package of your entity is not models, it's model:
package model;

ObjectDB: Persistence Exception type not found error 301

while following this tutorial:
http://www.objectdb.com/tutorial/jpa/eclipse
I receive the following error:
Exception in thread "main" [ObjectDB 2.5.4_05] javax.persistence.PersistenceException
Type tutorial.model.Point is not found (error 301)
at com.objectdb.jpa.EMImpl.persist(EMImpl.java:437)
at objectdb.tutorial.main.Main.main(Main.java:25)
Caused by: com.objectdb.o.TEX: Type tutorial.model.Point is not found
at com.objectdb.o.MSG.e(MSG.java:107)
at com.objectdb.o.TYM.ay(TYM.java:1017)
at com.objectdb.o.TYM.ap(TYM.java:807)
at com.objectdb.o.TYM.ao(TYM.java:757)
at com.objectdb.o.TYM.at(TYM.java:873)
at com.objectdb.o.TYM.aw(TYM.java:945)
at com.objectdb.o.OBM.bB(OBM.java:371)
at com.objectdb.o.OBM.bB(OBM.java:257)
at com.objectdb.jpa.EMImpl.persist(EMImpl.java:434)
... 1 more
Caused by: java.lang.ClassNotFoundException: tutorial.model.Point
at com.objectdb.o.TYM.findClass(TYM.java:1033)
at com.objectdb.o.ACL.loadClass(ACL.java:131)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at com.objectdb.o.TYM.ay(TYM.java:1013)
... 8 more`
I am looked at this post (Problems searching empty user database with ObjectDB), but for some reason, my instance of EntityManager does not have the member function getMetamodel(). So, I tried to create my own persistence unit which I have below with my code. I still receive the runtime error which seems to occur on em.persist(p) in the first for loop.
package objectdb.tutorial.main;
import javax.persistence.*;
import objectdb.tutorial.model.Point;
import java.util.*;
public class Main {
public static void main(String[] args) {
// Open a database connection
// (create a new database if it doesn't exist yet):
EntityManagerFactory emf =
Persistence.createEntityManagerFactory("my-pu");
EntityManager em = emf.createEntityManager();
// Store 1000 Point objects in the database:
em.getTransaction().begin();
for (int i = 0; i < 1000; i++) {
Point p = new Point(i, i);
em.persist(p);
}
em.getTransaction().commit();
// Find the number of Point objects in the database:
Query q1 = em.createQuery("SELECT COUNT(p) FROM Point p");
System.out.println("Total Points: " + q1.getSingleResult());
// Find the average X value:
Query q2 = em.createQuery("SELECT AVG(p.x) FROM Point p");
System.out.println("Average X: " + q2.getSingleResult());
// Retrieve all the Point objects from the database:
Query query =
em.createQuery("SELECT p FROM Point p");
List<Point> results = query.getResultList();
for (Point p : results) {
System.out.println(p);
}
// Close the database connection:
em.close();
emf.close();
}
}
Entity:
package objectdb.tutorial.model;
import java.io.Serializable;
import javax.persistence.*;
#Entity
public class Point implements Serializable {
private static final long serialVersionUID = 1L;
#Id #GeneratedValue
private long id;
private int x;
private int y;
public Point() {
}
public Point(int x, int y) {
this.x = x;
this.y = y;
}
public Long getId() {
return id;
}
public int getX() {
return x;
}
public int getY() {
return y;
}
#Override
public String toString() {
return String.format("(%d, %d)", this.x, this.y);
}
}
persistence.xml :
<?xml version="1.0" encoding="UTF-8" ?>
<persistence xmlns="http://java.sun.com/xml/ns/persistence"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/persistence
http://java.sun.com/xml/ns/persistence/persistence_1_0.xsd" version="1.0">
<persistence-unit name="my-pu">
<description>My Persistence Unit</description>
<provider>com.objectdb.jpa.Provider</provider>
<!-- <mapping-file>META-INF/mappingFile.xml</mapping-file> Not sure what this is-->
<class>objectdb.tutorial.model.Point</class>
<properties>
<property name="javax.persistence.jdbc.url"
value="objectdb://localhost/tutorial.odb"/>
<property name="javax.persistence.jdbc.user" value="admin"/>
<property name="javax.persistence.jdbc.password" value="admin"/>
</properties>
</persistence-unit>
</persistence>
I appreciate any advice. Thank you.

The error message: "Type tutorial.model.Point is not found" specifies a Point class in a package tutorial.model. Your package name is objectdb.tutorial.model. Probably you had that package name previously, but you moved your package to a new location and name.
Try to delete the database file, since it still includes the class in its previous name (or if you need the data, follow the instructions on ObjectDB website regarding schema change).
If you do not have a getMetamodel method, you may have an old JPA 1.0 jar in your classpath before ObjectDB (which includes the JPA 2.1 jar). You have to fix your classpath to avoid additional issues.

Configuring HSQL Server for Hibernate tests

I'm just starting with Hibernate and getting my head around things.
Currently I'm trying to setup a testing environment where I can use a HSQL in-memory instance to test my project.
The error I'm running into is:
javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: invalid schema name: TSG
Here are the relevant parts of my project:
persistence.xml
<?xml version="1.0" encoding="UTF-8"?>
org.hibernate.ejb.HibernatePersistence
com.foo.api.models.tsg.AlgPpcAlgorithmOutputEntity
<persistence-unit name="TestingPersistenceUnit" transaction-type="RESOURCE_LOCAL">
<provider>org.hibernate.ejb.HibernatePersistence</provider>
<class>com.foo.api.models.tsg.AlgPpcAlgorithmOutputEntity
</class>
<properties>
<property name="dialect" value="org.hibernate.dialect.HSQLDialect"/>
<property name="hibernate.connection.driver_class" value="org.hsqldb.jdbcDriver"/>
<property name="hibernate.connection.url" value="jdbc:hsqldb:mem:tsg"/>
<property name="hbm2ddl.auto" value="create-drop"/>
<property name="hibernate.connection.autocommit" value="true"/>
<property name="hibernate.connection.username" value="sa"/>
<property name="hibernate.connection.password" value=""/>
<property name="hibernate.show_sql" value="true"/>
</properties>
</persistence-unit>
As you can see, I have one peristence-unit for production (which works fine) and one in-memory HSQL one for testing (which I can't get to work).
An example Hibernate entity:
package com.foo.api.models.tsg;
import javax.persistence.*;
import java.math.BigDecimal;
#IdClass(AlgPpcAlgorithmOutputEntityPK.class)
#Table(name = "alg_ppc_algorithm_output", schema = "", catalog = "tsg")
#Entity
public class AlgPpcAlgorithmOutputEntity {
private int parameterId;
#Column(name = "parameter_id")
#Id
public int getParameterId() {
return parameterId;
}
public void setParameterId(int parameterId) {
this.parameterId = parameterId;
}
private String matchType;
#Column(name = "matchType")
#Basic
public String getMatchType() {
return matchType;
}
// for brevity I have removed the rest of the implementation
// It was auto-generated by Hibernate and works fine in production.
}
Finally, a simple TestCase class:
package tests.integration;
import com.foo.api.models.tsg.AlgPpcAlgorithmOutputEntity;
import com.foo.api.util.HibernateUtil;
import org.hsqldb.Server;
import org.hsqldb.persist.HsqlProperties;
import org.hsqldb.server.ServerConfiguration;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import tests.util.HSQLServerUtil;
import javax.persistence.EntityManagerFactory;
import javax.persistence.EntityManager;
import javax.persistence.Persistence;
import com.foo.api.KeywordManager;
import java.sql.Date;
import java.util.HashSet;
public class KeywordManagerTestCase {
private static final Logger LOG = LoggerFactory.getLogger(KeywordManagerTestCase.class);
private EntityManagerFactory eMF;
protected EntityManager eM;
#Before
public void setUp() throws Exception {
HsqlProperties props = new HsqlProperties();
props.setProperty("server.database.0", "mem:tsg");
props.setProperty("server.dbname.0", "tsg");
ServerConfiguration.translateDefaultDatabaseProperty(props);
Server hsqlServer = new Server();
hsqlServer.setRestartOnShutdown(false);
hsqlServer.setNoSystemExit(true);
hsqlServer.setProperties(props);
hsqlServer.setTrace(true);
LOG.info("Configured the HSQLDB server...");
hsqlServer.start();
LOG.info("HSQLDB server started on port " + hsqlServer.getPort() + "...");
LOG.info("Loading hibernate...");
if (eMF == null) {
eMF = Persistence.createEntityManagerFactory("TestingPersistenceUnit");
}
eM = eMF.createEntityManager();
}
/**
* shutdown the server.
* #throws Exception in case of errors.
*/
#After
public void tearDown() throws Exception {
eM.close();
HSQLServerUtil.getInstance().stop();
}
/**
* Demo test to see that the number of user records in the database corresponds the flat file inserts.
*/
#Test
public void testDemo1() {
AlgPpcAlgorithmOutputEntity entity = new AlgPpcAlgorithmOutputEntity();
entity.setParameterId(200);
entity.setMatchType("aa");
KeywordManager km;
eM.persist(entity);
HashSet<Integer> params = new HashSet<Integer>();
params.add(200);
km = new KeywordManager(eM, params, new Date[2]);
HashSet<AlgPpcAlgorithmOutputEntity> res = km.pullKeywords(params);
for (AlgPpcAlgorithmOutputEntity s : res) {
System.out.println(s.getMatchType());
}
}
}
I'm sure I have set something up in a strange way, but as I say - this is my very first stab.
Here is what I'm trying to do:
Have the testing configuration (along with all the hibernate class mapping)for the HSQL db present in persistence.xml
Have the HSQL db start for unit tests and create and in-memory database with my project's schema (as described in persistence.xml under the class elements).
Create entity objects and add them to the test db at test-time so I can use the db for my integration tests.
I just can't get past this PersistenceException!
UPDATE
OK, so I realised that I didn't need to setup an explicit HSQL server in the setup of my test cases, because that's exactly what the persistence-unit entry in my persistence.xml is for. Also I have attempted to ALTER the CATALOG so that it matches the catalog used by my mapping classes. The setup of my test case now looks like:
private EntityManagerFactory eMF;
protected EntityManager eM;
#Before
public void setUp() throws Exception {
LOG.info("Loading hibernate...");
if (eMF == null) {
eMF = Persistence.createEntityManagerFactory("TestingPersistenceUnit");
}
eM = eMF.createEntityManager();
EntityTransaction eT = null;
eT = eM.getTransaction();
eT.begin();
Query q = eM.createNativeQuery("ALTER CATALOG PUBLIC RENAME TO TSG");
q.executeUpdate();
eT.commit();
// And also it seems I need to create the schema
eT = eM.getTransaction();
eT.begin();
q = eM.createNativeQuery("CREATE SCHEMA TSG AUTHORIZATION DBA");
q.executeUpdate();
eT.commit();
}
However, I just end up with a new error now, specifically:
javax.persistence.PersistenceException: org.hibernate.exception.SQLGrammarException: user lacks privilege or object not found: ALG_PPC_ALGORITHM_OUTPUT
So I'm getting somewhere but it seems the tables are not being created. I wonder if there is an issue with my persistence.xml?

As far as I know (and I am not a HSQL expert), the TSG you specify in the connection url is the database name which is not the same as the catalog. See http://hsqldb.org/doc/2.0/guide/databaseobjects-chapt.html#dbc_schemas_schema_objects
In HyperSQL, there is only one catalog per database. The name of the catalog is PUBLIC. You can rename the catalog with the ALTER CATALOG RENAME TO statement. All schemas belong the this catalog. The catalog name has no relation to the file name of the database.
As I read that, when HSQL creates a database it creates a catalog named PUBLIC within that database and a schema named PUBLIC within that catalog. There can only ever be one catalog per HSQL database. There can be multiple schemas within that single catalog.
The error you are getting comes actually from the attempt to specify catalog = "tsg" in your mapping. That catalog does not exist. Since the HSQL database can contain only one catalog, you will have to rename that PUBLIC catalog to TSG (or change up your mapping).

You are specifying server name in the URL but trying to use memory database, which is causing the issue.
Try using the DB URL as:
jdbc:hsqldb:mem:tsg
i.e.
<property name="hibernate.connection.url" value="jdbc:hsqldb:mem:tsg"/>
Also use ALTER CATALOG RENAME TO tsg to change the default catalog name(PUBLIC).
EDIT: To auto create the schema, update below in persistence.xml (hibernate.hbm2ddl.auto in place of hbm2ddl.auto)
<property name="hibernate.hbm2ddl.auto" value="create-drop"/>

You can override the entity mapping for your unit test by dropping in an orm.xml file to your src/test/resources/META-INF (in case of using the maven layout). In this you can override the JPA annotation mappings. For your needs you just need to override the table location, like this:
<?xml version="1.0" encoding="UTF-8"?>
<entity-mappings xmlns="http://java.sun.com/xml/ns/persistence/orm" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/persistence/orm http://java.sun.com/xml/ns/persistence/orm_1_0.xsd" version="1.0">
<entity class="com.foo.api.models.tsg.AlgPpcAlgorithmOutputEntity">
<table name="alg_ppc_algorithm_output"/>
</entity>
</entity-mappings>
This will place your table into the default public catalog of the hsqldb database, so you don't have to alter the hsqldb scheme. It may even work when your unit test uses multiple catalogs containing tables with the same name, as you just have to give different names in the table name attribute.

There is an obvious error in the setup. The connection URL must point to the Server:
<property name="hibernate.connection.url" value="jdbc:hsqldb:hsql://localhost/tsg"/>