We Keep Coding

Use Spring Data repositories to fill in test data

I'd like to ask whether it is alright to use apps repositories(Spring Data based) to fill in testing data. I know I can use sql file with data, but sometimes I need something more dynamical. I find writing sql or datasets definitions cumbersome(and hard to maintain in case of schema change). Is there anything wrong with using app repositories? There are all basic CRUD operations already there. Note we are talking especially about integration testing.
I feel it is kind of weird to use part of app to test itself. Maybe I can create another set of repositories to be used in test contexts.

No, there is absolutely nothing wrong with using Spring Data repositories to create test data.
I even prefer that since it often allows for simpler refactoring.
As with any use of JPA in tests you need to keep in mind that JPA implementations are a write-behind cache. You probably want to flush and clear the EntityManager after setting up the test data, so that you don't get anything from the 1st level cache that really should come from the database. Also, this ensures data is actually written to the database and problems with that will surface.
You might be interested in a couple of articles about testing with Hibernate. They don't use Spring Data, but it would work with Spring Data JPA just the same.

I would recommand to use Flyway to setup your databases and use Flyway test extension for integration testing.
So that you can do something like that:
#ContextConfiguration(locations = {"/context/simple_applicationContext.xml"})
#TestExecutionListeners({DependencyInjectionTestExecutionListener.class,
FlywayTestExecutionListener.class})
#Test
#FlywayTest(locationsForMigrate = {"loadmsql"}) // execution once per class
public class MethodTest extends AbstractTestNGSpringContextTests {
#BeforeClass
#FlywayTest(locationsForMigrate = {"loadmsql"}) // execution once per class
public static void beforeClass() {
// maybe some additional things
}
#BeforeMethod
#FlywayTest(locationsForMigrate = {"loadmsql"}) // execution before each test method
public void beforeMethod() {
// maybe before every test method
}
#Test
#FlywayTest(locationsForMigrate = {"loadmsql"}) // as method annotation
public void simpleCountWithoutAny() {
// or just with an annotation above the test method where you need it
}

Gemfire EntryNotFoundException for #CacheEvict

In short, when #CacheEvict is called on a method and if the key for the entry is not found, Gemfire is throwing EntryNotFoundException.
Now in detail,
I have a class
class Person {
String mobile;
int dept;
String name;
}
I have two Cache regions defined as personRegion and personByDeptRegion and the Service is as below
#Service
class PersonServiceImpl {
#Cacheable(value = "personRegion")
public Person findByMobile(String mobile) {
return personRepository.findByMobile(mobile);
}
#Cacheable(value = "personByDeptRegion")
public List<Person> findByDept(int deptCode) {
return personRepository.findByDept(deptCode);
}
#Caching(
evict = { #CacheEvict(value = "personByDeptRegion", key="#p0.dept"},
put = { #CachePut(value = "personRegion",key = "#p0.mobile")}
)
public Person updatePerson(Person p1) {
return personRepository.save(p1);
}
}
When there is a call to updatePerson and if there are no entries in the personByDeptRegion, this would throw an exception that EntryNotFoundException for the key 1 ( or whatever is the dept code ). There is a very good chance that this method will be called before the #Cacheable methods are called and want to avoid this exception.
Is there any way we could tweak the Gemfire behavior to gracefully return when the key is not existing for a given region ?.
Alternatively, I am also eager to know if there is a better implementation of the above scenario using Gemfire as cache.
Spring Data Gemfire : 1.7.4
Gemfire Version : v8.2.1
Note: The above code is for representation purpose only and I have multiple services with same issue in actual project.

First, I commend you for using Spring's Caching annotations on your application #Service components. All too often developers enable caching in their Repositories, which I think is bad form, especially if complex business rules (or even additional IO; e.g. calling a web service from a service component) are involved prior to or after the Repository interaction(s), particularly in cases where caching behavior should not be affected (or determined).
I also think your caching UC (updating one cache (personRegion) while invalidating another (personByDeptRegion) on a data store update) by following a CachePut with a CacheEvict seems reasonable to me. Though, I would point out that the seemingly intended use of the #Caching annotation is to combine multiple Caching annotations of the same type (e.g. multiple #CacheEvict or multiple #CachePut) as explained in the core Spring Framework Reference Guide. Still, there is nothing preventing your intended use.
I created a similar test class here, modeled after your example above, to verify the problem. Indeed the jonDoeUpdateSuccessful test case fails (with the GemFire EntryNotFoundException, shown below) since no people in Department "R&D" were previously cached in the "DepartmentPeople" GemFire Region prior to the update, unlike the janeDoeUpdateSuccessful test case, which causes the cache to be populated before the update (even if the entry has no values, which is of no consequence).
com.gemstone.gemfire.cache.EntryNotFoundException: RESEARCH_DEVELOPMENT
at com.gemstone.gemfire.internal.cache.AbstractRegionMap.destroy(AbstractRegionMap.java:1435)
NOTE: My test uses GemFire as both a "cache provider" and a System of Record (SOR).
The problem really lies in SDG's use of Region.destroy(key) in the GemfireCache.evict(key) implementation rather than, and perhaps more appropriately, Region.remove(key).
GemfireCache.evict(key) has been implemented with Region.destroy(key) since inception. However, Region.remove(key) was not introduced until GemFire v5.0. Still, I can see no discernible difference between Region.destroy(key) and Region.remove(key) other than the EntryNotFoundException thrown by Region.destroy(key). Essentially, they both destroy the local entry (both key and value) as well as distribute the operation to other caches in the cluster (providing a non-LOCAL Scope is used).
So, I have filed SGF-539 to change SDG to call Region.remove(key) in GemfireCache.evict(key) rather than Region.destroy(key).
As for a workaround, well, there is basically only 2 things you can do:
Restructure your code and your use of the #CacheEvict annotation, and/or...
Make use of the condition on #CacheEvict.
It is unfortunate that a condition cannot be specified using a class type, something akin to a Spring Condition (in addition to SpEL), but this interface is intended for another purpose and the #CacheEvict, condition attribute does not accept a class type.
At the moment, I don't have a good example of how this might work so I am moving forward on SGF-539.
You can following this ticket for more details and progress.
Sorry for the inconvenience.
-John

Grails save() tries to create new object when it should update

In my service code, I am trying to create or update a Person domain object:
#Transactional
def someServiceMethod(some params....) {
try{
def person = Person.findByEmail(nperson.email.toLowerCase())
if (!person) {
person = new Person()
person.properties = nperson.properties
} else {
// update the person parameters (first/last name)
person.firstName = nperson.firstName
person.lastName = nperson.lastName
person.phone = nperson.phone
}
if (person.validate()) {
person.save(flush: true)
//... rest of code
}
// rest of other code....
} catch(e) {
log.error("Unknown error: ${e.getMessage()}", e)
e.printStackTrace()
return(null)
}
Now above code OCCASIONALLY when trying to save a Person object with already existing email throws following exception:
Hibernate operation: could not execute statement; SQL [n/a]; Duplicate entry 'someemail#gmail.com' for key 'email_UNIQUE'; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry 'someemail#gmail.com' for key 'email_UNIQUE'
This is very strange because I am already finding the person by email and hence the save() should try to update the record instead of creating the new one.
I was wondering why is this happening!
EDIT:
I am on grails 2.4.5 and Hibernate plugin in BuildConfig is:
runtime ':hibernate4:4.3.8.1'
EDIT2:
My application is on multiple servers hence synchronized block won't work

If this is concurrency issue, here is what we do in such case. We have a lot of concurrent background processes which work on the same tables. If there is such operation it indeed is in synchronized block, so code may look like:
class SomeService {
static transactional = false //service cannot be transactional
private Object someLock = new Object() //synchronized block on some object must be used
def someConcurrentSafeMethod(){
synchronized(someLock){
def person = Person.findByEmail(nperson.email.toLowerCase())
...
person.save(flush: true) // flush is very important, must be done in synchronized block
}
}
}
There are few important points to make this working (from our experience, not official):
Service cannot be transactional - if service is transactional, transaction is commited after the method returns value and synchronization inside method will not be enough. Programmatic transactions may be another way
synchronized method is not enough synchronized def someConcurrentSafeMethod() will not work - probably because service is wrapped in proxy
Session MUST be flushed inside synchronized block
every object which will be saved, should be read in synchronized block, if you pass it from external method, you may run into optimistic locking failed exception
UPDATED
Because application is deployed on distributed system, above will not solve the issue here (still may help others). After discussion we had on Slack, I just summarize potential ways to do that:
pessimistic locking of updated objects and lock of whole table for inserts (if possible)
moving 'dangerous' database related methods to single server with some API like REST and calling it from other deployments (and using synchronized approach from above)
using multiple save approach - if operation fails, catch exception and try again. This is supported by integration libraries like Spring Integration or Apache Camel and is one of enterprise patterns. See request-handler-advice-chain for Spring Integration as an example
use something to queue operations, for example JMS server
If anyone has more ideas please share them.

Hibernate faster EntityManagerFactory creation

In my desktop application new databases get opened quite often. I use Hibernate/JPA as an ORM.
The problem is, creating the EntityManagerFactory is quite slow, taking about 5-6 Seconds on a fast machine. I know that the EntityManagerFactory is supposed to be heavyweight but this is just too slow for a desktop application where the user expects the new database to be opened quickly.
Can I turn off some EntityManagerFactory features to get an instance
faster? Or is it possible to create some of the EntityManagerFactory lazily to speed up cration?
Can I somehow create the EntityManagerFactory object before
knowing the database url? I would be happy to turn off all
validation for this to be possible.
By doing so, can I pool EntityManagerFactorys for later use?
Any other idea how to create the EntityManagerFactory faster?
Update with more Information and JProfiler profiling
The desktop application can open saved files. Our application document file format constists of 1 SQLite database + and some binary data in a ZIP file. When opening a document, the ZIP gets extracted and the db is opened with Hibernate. The databases all have the same schema, but different data obviously.
It seems that the first time I open a file it takes significantly longer than the following times.
I profiled the first and second run with JProfiler and compared the results.
1st Run:
create EMF: 4385ms
build EMF: 3090ms
EJB3Configuration configure: 900ms
EJB3Configuration <clinit>: 380ms
.
2nd Run:
create EMF: 1275ms
build EMF: 970ms
EJB3Configuration configure: 305ms
EJB3Configuration <clinit>: not visible, probably 0ms
.
In the Call tree comparison you can see that some methods are significantly faster (DatabaseManager. as starting point):
create EMF: -3120ms
Hibernate create EMF: -3110ms
EJB3Configuration configure: -595ms
EJB3Configuration <clinit>: -380ms
build EMF: -2120ms
buildSessionFactory: -1945ms
secondPassCompile: -425ms
buildSettings: -346ms
SessionFactoryImpl.<init>: -1040ms
The Hot spot comparison now has the interesting results:
.
ClassLoader.loadClass: -1686ms
XMLSchemaFactory.newSchema: -184ms
ClassFile.<init>: -109ms
I am not sure if it is the loading of Hibernate classes or my Entity classes.
A first improvement would be to create an EMF as soon as the application starts just to initialize all necessary classes (I have an empty db file as a prototype already shipped with my Application). #sharakan thank you for your answer, maybe a DeferredConnectionProvider would already be a solution for this problem.
I will try the DeferredConnectionProvider next! But we might be able to speed it up even further. Do you have any more suggestions?

You should be able to do this by implementing your own ConnectionProvider as a decorator around a real ConnectionProvider.
The key observation here is that the ConnectionProvider isn't used until an EntityManager is created (see comment in supportsAggressiveRelease() for a caveat to that). So you can create a DeferredConnectionProvider class, and use it to construct the EntityManagerFactory, but then wait for user input, and do the deferred initialization before actually creating any EntityManager instances. I'm written this as a wrapper around ConnectionPoolImpl, but you should be able to use any other implementation of ConnectionProvider as the base.
public class DeferredConnectionProvider implements ConnectionProvider {
private Properties configuredProps;
private ConnectionProviderImpl realConnectionProvider;
#Override
public void configure(Properties props) throws HibernateException {
configuredProps = props;
}
public void finalConfiguration(String jdbcUrl, String userName, String password) {
configuredProps.setProperty(Environment.URL, jdbcUrl);
configuredProps.setProperty(Environment.USER, userName);
configuredProps.setProperty(Environment.PASS, password);
realConnectionProvider = new ConnectionProviderImpl();
realConnectionProvider.configure(configuredProps);
}
private void assertConfigured() {
if (realConnectionProvider == null) {
throw new IllegalStateException("Not configured yet!");
}
}
#Override
public Connection getConnection() throws SQLException {
assertConfigured();
return realConnectionProvider.getConnection();
}
#Override
public void closeConnection(Connection conn) throws SQLException {
assertConfigured();
realConnectionProvider.closeConnection(conn);
}
#Override
public void close() throws HibernateException {
assertConfigured();
realConnectionProvider.close();
}
#Override
public boolean supportsAggressiveRelease() {
// This gets called during EntityManagerFactory construction, but it's
// just a flag so you should be able to either do this, or return
// true/false depending on the actual provider.
return new ConnectionProviderImpl().supportsAggressiveRelease();
}
}
a rough example of how to use it:
// Get an EntityManagerFactory with the following property set:
// properties.put(Environment.CONNECTION_PROVIDER, DeferredConnectionProvider.class.getName());
HibernateEntityManagerFactory factory = (HibernateEntityManagerFactory) entityManagerFactory;
// ...do user input of connection info...
SessionFactoryImpl sessionFactory = (SessionFactoryImpl) factory.getSessionFactory();
DeferredConnectionProvider connectionProvider = (DeferredConnectionProvider) sessionFactory.getSettings()
.getConnectionProvider();
connectionProvider.finalConfiguration(jdbcUrl, userName, password);
You could put the initial set up of the EntityManagerFactory on a separate thread or something, so that the user never has to wait for it. Then the only thing they'll wait for, after specifying the connection info, is the setting up of the connection pool, which should be fairly quick compared to parsing the object model.

Can I turn off some EntityManagerFactory features to get an instance faster?
Don't believe so. EMFs don't really have too many features, other than initializing a JDBC connection/pool.
Or is it possible to create some of the EntityManagerFactory lazily to
speed up cration?
Rather than creating the EMF lazily, when the user will notice the performance hit, I suggest you should head in the opposite direction - create the EMF proactively before the user actually needs it. Create it once, up-front, possibly in a separate thread during application initialisation (or at least as soon as you know about your database). Reuse it throughout the existence of your application/database.
Can I somehow create the EntityManagerFactory object before knowing the database url?
No - it creates a JDBC connection.
I think a better question is: why does your application dynamically discover database connection URLs? Are you saying your databases are created/made available on-the-fly and there's no way to anticipate in advance the connection parameters. That really is to be avoided.
By doing so, can I pool EntityManagerFactorys for later use?
No, you can't pool EMFs. It's the connections that you can pool.
Any other idea how to create the EntityManagerFactory faster?
I agree - 6 seconds is too slow for initialisation of EMFs.
I suspect it's more to do with your selected database technology than JPA/JDBC/JVM. My guess is that maybe your database is initialising itself as you connect. Are you using Access? What DB are you using?
Are you connecting to a database remotely located? Over a WAN? Is network speed/latency good?
Are the client PCs limited in performance?
EDIT: Added after comments
Implementing your own ConnectionProvider as a decorator around a real ConnectionProvider will not speed up the user's experience at all. The database instance still needs to be initialised, the EMF & EM created and the JDBC connection still needs to be subsequently established.
Options:
Share a common preloaded DB instance: seems not possible for your business scenario (although JSE technology supports this and also supports client-server design).
Change to a DB with a faster startup: Derby (a.k.a. Java DB) is included in modern JVMs and has a startup time of about 1.5 seconds (cold) and 0.7 seconds (warm - data pre-loaded).
In many (most?) scenarios, the fastest solution would be to load data directly into in-memory java objects using JAXB with STAX. Subsequently, use in-memory cached data (particularly using smart structures like maps, hashing and arraylists). Just as JPA can map POJO classes to database tables & columns, so JAXB can map POJO classes to XML schema & work with XML doc instances. If you have very complex queries using SQL set-based logic with multiple joins and strong use of DB indexes, this would be less desirable.
(2) would probably give the best improvement for limited effort.
Additionally:
- try to unzip the data files during deployment rather than during app usage.
- initialize the EMF in a startup thread that runs in parallel to the UI startup - try to start the DB initializing as one of the very first steps of the app (that means connecting to the actual instance using JDBC).

How should unit tests set up data sources when not running in an application server?

Thank you all for your help. A number of you posted (as I should have expected) answers indicating my whole approach was wrong, or that low-level code should never have to know whether or not it is running in a container. I would tend to agree. However, I'm dealing with a complex legacy application and do not have the option of doing a major refactoring for the current problem.
Let me step back and ask the question the motivated my original question.
I have a legacy application running under JBoss, and have made some modifications to lower-level code. I have created a unit test for my modification. In order to run the test, I need to connect to a database.
The legacy code gets the data source this way:
(jndiName is a defined string)
Context ctx = new InitialContext();
DataSource dataSource = (DataSource) ctx.lookup(jndiName);
My problem is that when I run this code under unit test, the Context has no data sources defined. My solution to this was to try to see if I'm running under the application server and, if not, create the test DataSource and return it. If I am running under the app server, then I use the code above.
So, my real question is: What is the correct way to do this? Is there some approved way the unit test can set up the context to return the appropriate data source so that the code under test doesn't need to be aware of where it's running?
For Context: MY ORIGINAL QUESTION:
I have some Java code that needs to know whether or not it is running under JBoss. Is there a canonical way for code to tell whether it is running in a container?
My first approach was developed through experimention and consists of getting the initial context and testing that it can look up certain values.
private boolean isRunningUnderJBoss(Context ctx) {
boolean runningUnderJBoss = false;
try {
// The following invokes a naming exception when not running under
// JBoss.
ctx.getNameInNamespace();
// The URL packages must contain the string "jboss".
String urlPackages = (String) ctx.lookup("java.naming.factory.url.pkgs");
if ((urlPackages != null) && (urlPackages.toUpperCase().contains("JBOSS"))) {
runningUnderJBoss = true;
}
} catch (Exception e) {
// If we get there, we are not under JBoss
runningUnderJBoss = false;
}
return runningUnderJBoss;
}
Context ctx = new InitialContext();
if (isRunningUnderJboss(ctx)
{
.........
Now, this seems to work, but it feels like a hack. What is the "correct" way to do this? Ideally, I'd like a way that would work with a variety of application servers, not just JBoss.

The whole concept is back to front. Lower level code should not be doing this sort of testing. If you need a different implementation pass it down at a relevant point.

Some combination of Dependency Injection (whether through Spring, config files, or program arguments) and the Factory Pattern would usually work best.
As an example I pass an argument to my Ant scripts that setup config files depending if the ear or war is going into a development, testing, or production environment.

The whole approach feels wrong headed to me. If your app needs to know which container it's running in you're doing something wrong.
When I use Spring I can move from Tomcat to WebLogic and back without changing anything. I'm sure that with proper configuration I could do the same trick with JBOSS as well. That's the goal I'd shoot for.

Perhaps something like this ( ugly but it may work )
private void isRunningOn( String thatServerName ) {
String uniqueClassName = getSpecialClassNameFor( thatServerName );
try {
Class.forName( uniqueClassName );
} catch ( ClassNotFoudException cnfe ) {
return false;
}
return true;
}
The getSpecialClassNameFor method would return a class that is unique for each Application Server ( and may return new class names when more apps servers are added )
Then you use it like:
if( isRunningOn("JBoss")) {
createJBossStrategy....etcetc
}

Context ctx = new InitialContext();
DataSource dataSource = (DataSource) ctx.lookup(jndiName);
Who constructs the InitialContext? Its construction must be outside the code that you are trying to test, or otherwise you won't be able to mock the context.
Since you said that you are working on a legacy application, first refactor the code so that you can easily dependency inject the context or data source to the class. Then you can more easily write tests for that class.
You can transition the legacy code by having two constructors, as in the below code, until you have refactored the code that constructs the class. This way you can more easily test Foo and you can keep the code that uses Foo unchanged. Then you can slowly refactor the code, so that the old constructor is completely removed and all dependencies are dependency injected.
public class Foo {
private final DataSource dataSource;
public Foo() { // production code calls this - no changes needed to callers
Context ctx = new InitialContext();
this.dataSource = (DataSource) ctx.lookup(jndiName);
}
public Foo(DataSource dataSource) { // test code calls this
this.dataSource = dataSource;
}
// methods that use dataSource
}
But before you start doing that refactoring, you should have some integration tests to cover your back. Otherwise you can't know whether even the simple refactorings, such as moving the DataSource lookup to the constructor, break something. Then when the code gets better, more testable, you can write unit tests. (By definition, if a test touches the file system, network or database, it is not a unit test - it is an integration test.)
The benefit of unit tests is that they run fast - hundreds or thousands per second - and are very focused to testing just one behaviour at a time. That makes it possible run then often (if you hesitate running all unit tests after changing one line, they run too slowly) so that you get quick feedback. And because they are very focused, you will know just by looking at the name of the failing test that exactly where in the production code the bug is.
The benefit of integration tests is that they make sure that all parts are plugged together correctly. That is also important, but you can not run them very often because things like touching the database make them very slow. But you should still run them at least once a day on your continuous integration server.

There are a couple of ways to tackle this problem. One is to pass a Context object to the class when it is under unit test. If you can't change the method signature, refactor the creation of the inital context to a protected method and test a subclass that returns the mocked context object by overriding the method. That can at least put the class under test so you can refactor to better alternatives from there.
The next option is to make database connections a factory that can tell if it is in a container or not, and do the appropriate thing in each case.
One thing to think about is - once you have this database connection out of the container, what are you going to do with it? It is easier, but it isn't quite a unit test if you have to carry the whole data access layer.
For further help in this direction of moving legacy code under unit test, I suggest you look at Michael Feather's Working Effectively with Legacy Code.

A clean way to do this would be to have lifecycle listeners configured in web.xml. These can set global flags if you want. For example, you could define a ServletContextListener in your web.xml and in the contextInitialized method, set a global flag that you're running inside a container. If the global flag is not set, then you are not running inside a container.