I`m using RedisTemplate provided by spring framework for In-Memory caching. And MySql as primary database. I need to update the cache whenever new row is added or updated in primary database.
How can i accomplish this using java?
Is there any inbuilt feature provided by Redis server to achieve this?
Update:
#Override public void getEmployeeDetailsForRedisTemplate(List<Employee> employee) {
logger.info("Saving " + employee.size() + " record to redis template");
for (Employee emp : employee) {
listOperations.leftPush(EnumConstants.EMPLOYEE_KEY.getValue(), emp);
}
}
I have been polling database in regular interval and based on status column in DB i was pushing updated data to redis server. Which is not a efficient way of doing it
Sorry I wanted to ask whether there is a code snippet of the Java code that is invoked when a new row is added/updated but I am new so I can't comment
If you are using Redis as a cache then you might want to use #Cacheable annotation provided by Spring-Data-Redis by setting up RedisCacheManager in your configuration
#Cacheable - Annotation indicating that the result of invoking a
method (or all methods in a class) can be cached. Each time an advised
method is invoked, caching behavior will be applied, checking whether
the method has been already invoked for the given arguments. A
sensible default simply uses the method parameters to compute the key,
but a SpEL expression can be provided via the key() attribute, or a
custom KeyGenerator implementation can replace the default one (see
keyGenerator()).
If no value is found in the cache for the computed key, the target
method will be invoked and the returned value stored in the associated
cache. Note that Java8's Optional return types are automatically
handled and its content is stored in the cache if present.
So you could actually use it like
#Override
#Cacheable(value = "cacheName", key = "#employee.id")
public void getEmployeeDetailsForRedisTemplate(List<Employee> employee) {
logger.info("Saving " + employee.size() + " record to redis template");
for (Employee emp : employee) {
listOperations.leftPush(EnumConstants.EMPLOYEE_KEY.getValue(), emp);
}
}
Hope this helps
Related
I need to create a spring boot batch job in which, I need to fetch data from mongoDB where I don't have info about collection name & fields to fetch at coding time. I get this info only when the batch starts.
E.g. When batch starts, I can read properties file where I get collection name by 1 property & another property gives list of fields to fetch, third filed provides criteria/condition for query
So, due to this, I cant have a Java POJO defined which have mapping for collection or I cant create any MongoRepository\Template (collection name is known at runtime).
What I want to know is, just like plain old native SQL, if i get to know fields name & table name, on the fly, SQL can be build & can be fired to get the data:
String dynamicQuery = "SELECT " + commaSeperatedFieldsList + " FROM " + tableName + " WHERE " + criteria;
Is there any way by which same thing can be achieved in spring boot + mongo DB?
You can use MongoTemplate for this which can be autowired as spring provides and configures it for you automatically.
It has a
find(Query query, Class<T> entityClass, String collectionName)
method which lets you define a custom collection name and a custom entityClass.
For the dynamic query use BasicQuery as Query impl to pass a raw mongo json query and fields/projection as json if you want to limit the fields returned.
Use org.bson.Document as entityClass which is basically a Map implementation which lets you iterate over the fields in a dynamic manner.
mongoTemplate.find(new BasicQuery("{ name: \"mongodb\"}", "{ name: 1}"), Document.class, "your-collection-name").forEach(x -> {
x.get("name"); // access a specific field
x.forEach((key, value) -> {
// iterate over all fields
});
});
When you deal with a large result consider using MongoTemplate's stream() method in the same way as this doesn't load all documents into memory at once and you can process it during execution one by one.
I'm using a CrudRepository for connecting to Redis in my Spring Boot application and a #TimeToLive annotated field in the entity for expiration:
#RedisHash("keyspace")
public class MyRedisEntity {
#Id String key;
MyPojo pojo;
#TimeToLive Long ttl;
}
public interface MyRedisRepository extends CrudRepository<MyRedisEntity, String>{}
Now when the expiration has taken place, myRedisRepo.findAll() returns null for the expired entities. I discovered redis (or spring-data redis) stores all inserted entities' id in a set with the keyspace as key:
redis-cli> smembers keyspace
0) key0
1) key1
2) key2
...
redis-cli> hgetall key0
(empty list or set)
I suspect this set is used for the findAll call, returning null for ids no longer present as hashmaps due to expiration. Also, I tried using a listener for RedisKeyExpiredEvent, using the repository's delete method in onApplicationEvent, but that doesn't help.
#Component
public class RedisExpirationListener implements ApplicationListener<RedisKeyExpiredEvent> {
private MyRedisRepository myRedisRepository;
#Autowired
public RedisExpirationListener(MyRedisRepository myRedisRepository) {
this.myRedisRepository = myRedisRepository;
}
#Override
public void onApplicationEvent(RedisKeyExpiredEvent redisKeyExpiredEvent) {
if (redisKeyExpiredEvent.getKeyspace().equals("keyspace")) {
myRedisRepository.deleteById(new String(redisKeyExpiredEvent.getId()));
}
}
}
What should I do to get only non null entries? Ideally I'd want the expired entries to be deleted entirely from redis and thus not appear in findAll, but it'd be sufficient if a repository method could return list of non null values.
(And yes, I know about the phantom behaviour, but I don't think it's relevant to what I want)
As mentioned by #Enoobong there is open issue https://jira.spring.io/browse/DATAREDIS-570 (and there is a workaround added this September)
It is open for 5 years.
That means you should use RedishHash with TTL very carefully:
Make sure your Redis server allows your app to 'CONFIGURE' keyspace events
Make sure you have a workaround from github issue
But that is still not recommended as production-ready solution (I mean configuring kayspace events from app annotation) so please consider putting this to the application logic to keep the scheduler to remove outdated records.
Also this meens you need to handle null values returned from the repo until that issue is fixed.
How is the #Cacheable annotation notified about the changes in the database. For eg : If I Cache the result of CRUDRepository.findAll() function call by annotating the function call by #Cacheable the result is stored in a named cache. The Cache however continues to be updated with DB changes. Consider the below example :
#Cacheable(value = "employee", key = "#id")
public Iterable<Employee> getAllEmployees(){
long startTime = System.currentTimeMillis();
Iterable<Employee> itr = employeeEntityRepository.findAll();
long stopTime = System.currentTimeMillis();
System.out.println("Time-->" + (stopTime - startTime));
return itr;
}
When this method is called the first time it takes 300 ms and the 2nd time it takes hardly 5 ms which means it Caches the result set. Fair enough. But now if I update a record in the employee table directly in the DB and call this method again two things happen :
The new record is returned by this method call.
The method returns very fast.
This means that the cache is updated from the DB every time. How does this Sync work if the resultset is returned from Cache ?
thanks.
The main question is: cache is updated from the DB every time. How does this Sync work if the resultset is returned from Cache?
As per spring doc (5.3.7) #Cacheable annotation is used when we need to cache the result of a call.
in the above scenario caching behavior will be applied, it's checked if the method was already invoked for the given arguments.
below attributes are used in your example. Their description is as below:
value : this name of cache, in above case its "employee"
Key : this is SpEl, Spring Expression Language, to compute the key dynamically.
Ideally, you should have "id" as method-argument in the method getAllEmployees().
But this method seems to return all employees, so it seems incorrect usage in this scenario.
SpEl expression is "#id" = a custom key generator
By default it is "" = all method parameter are considered as key.
If no value is found in the cache for the computed key, the target method will be invoked and returned value will be stored in the associated cache.
The scenario which I encountered is that the method which is annotated by #Cacheable
#Cacheable(value= myConstant,key="#accountId")
public Integer getAccountNumber( Integer accountId){.
// an API call is made here..}
So, cache is not updated from DB, it's updated from a call (API call) next, the result is stored in the cache when the method is invoked for the first time. When the method is called a second time the cached result is returned and API call in the method do not happen.
I have referred: https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/cache/annotation/Cacheable.html
I'm working on a legacy code base that uses JPA (not JPA-2), and have come across the following method in a DAO implementation class to retrieve a single entity by ID (which is also it's primary key):
public EmailTemplate findEmailTemplateById(long id) {
LOG.debug("Entering findEmailTemplateById(id='" + id + "')");
// Construct JPQL query
String queryString = "SELECT a FROM EmailTemplate a " +
"WHERE templateId = :templateId";
Query query = entityManager.createQuery(queryString);
query.setParameter("templateId", id);
LOG.debug("Using query " + queryString);
List<EmailTemplate> resultList = query.getResultList();
LOG.debug("Exiting findEmailTemplateByName(id='" + id + "') results size " + resultList.size() + " ( returns null if 0 )");
if (resultList.isEmpty() || resultList.size() == 0) {
return null;
} else {
return resultList.get(0);
}
}
I now need to write a similar DAO class for a different entity, and my method to find the entity by it's primary key looks a lot simpler! :
#Override
public EmailTemplateEdit findEmailTemplateEditById(long id) {
LOG.debug("Entering findEmailTemplateEditById(id={})", id);
return entityManager.find(EmailTemplateEdit.class, id);
}
The original author is not around to ask, so I'm wondering if anyone can suggest reasons as to why he constructed a JPQL query rather than simply using EntityManager#find(Class<T> entityClass, Object primaryKey)?
The javadoc for the find method says:
If the entity instance is contained in the persistence context, it is
returned from there.
which suggests some form of caching and/or delayed writes. The javadoc for the createQuery and getResultList methods don't say anything like this.
I am unaware of any business or technical requirement in this application that would preclude caching, or of any issues resulting from stale entities or similar. I will check these with the rest of the project team when available, but I just thought I'd canvas the opinion of the SO community to see if there might be other reasons why a query was constructed and executed instead of simply using find
(I've seen this: When use createQuery() and find() methods of EntityManager?. Whilst it answers the question re: difference between createQuery and find, it doesn't answer it in context of finding entities by primary key)
Updated with additional info
From looking at the other methods in the original DAO class, it looks like there has been a deliberate/conscious decision to not take advantage of JPA managed objects. As above, the method to find by primary key uses a JPQL query. The method to delete an entity also uses a JPQL query. And the method to update an entity makes a copy of the passed in entity object and calls EntityManager#merge with the copy (thus the copy is a managed object, but is never used or returned from the method)
Weird ....
Short answer, there is no difference between find and a select query.
Your question suggests that you are not entirely familiar with what an EntityManager and a Persistence context is.
EntityManager implementation are not required to be thread safe. If the EntityManager is injected by Spring or and EJB-container it is thread safe (because it is a thread-local proxy), if it is application managed (you created it by calling EntityManagerFactory.createEntityManager(), it is not thread safe, and you can't stor it in a variable, but have to create a new one every time.
The Persistence Context, is where entities live, whenever you create a new EntityManager you get a new Persistence context (there are exceptions to this rule). When you persist an Entity, or load an existing entity from the db (using find or query) it will be managed by the persistence context. When you commit a transaction JPA runs through ALL Entities managed by the Persistence context, and checks the state of the entity to find out which queries should be sent to the database.
The PersistenceContext can be seen as a first-level cache on top of the database. It is meant to have a short lifespan, typically no longer than the transaction. If you re-use the same entityManager for multiple transactions, the size could grow as more data is loaded, this is bad because every transaction has to run through all entities in the persistence context.
We are using Spring and IBatis and I have discovered something interesting in the way a service method with #Transactional handles multiple DAO calls that return the same record. Here is an example of a method that does not work.
#Transactional
public void processIndividualTrans(IndvTrans trans) {
Individual individual = individualDAO.selectByPrimaryKey(trans.getPartyId());
individual.setFirstName(trans.getFirstName());
individual.setMiddleName(trans.getMiddleName());
individual.setLastName(trans.getLastName());
Individual oldIndvRecord = individualDAO.selectByPrimaryKey(trans.getPartyId());
individualHistoryDAO.insert(oldIndvRecord);
individualDAO.updateByPrimaryKey(individual);
}
The problem with the above method is that the 2nd execution of the line
individualDAO.selectByPrimaryKey(trans.getPartyId())
returns the exact object returned from the first call.
This means that oldIndvRecord and individual are the same object, and the line
individualHistoryDAO.insert(oldIndvRecord);
adds a row to the history table that contains the changes (which we do not want).
In order for it to work it must look like this.
#Transactional
public void processIndividualTrans(IndvTrans trans) {
Individual individual = individualDAO.selectByPrimaryKey(trans.getPartyId());
individualHistoryDAO.insert(individual);
individual.setFirstName(trans.getFirstName());
individual.setMiddleName(trans.getMiddleName());
individual.setLastName(trans.getLastName());
individualDAO.updateByPrimaryKey(individual);
}
We wanted to write a service called updateIndividual that we could use for all updates of this table that would store a row in the IndividualHistory table before performing the update.
#Transactional
public void updateIndividual(Individual individual) {
Individual oldIndvRecord = individualDAO.selectByPrimaryKey(trans.getPartyId());
individualHistoryDAO.insert(oldIndvRecord);
individualDAO.updateByPrimaryKey(individual);
}
But it does not store the row as it was before the object changed. We can even explicitly instantiate different objects before the DAO calls and the second one becomes the same object as the first.
I have looked through the Spring documentation and cannot determine why this is happening.
Can anyone explain this?
Is there a setting that can allow the 2nd DAO call to return the database contents and not the previously returned object?
You are using Hibernate as ORM and this behavior is perfectly described in the Hibernate documentation. In the Transaction chapter:
Through Session, which is also a transaction-scoped cache, Hibernate provides repeatable reads for lookup by identifier and entity queries and not reporting queries that return scalar values.
Same goes for IBatis
MyBatis uses two caches: a local cache and a second level cache. Each
time a new session is created MyBatis creates a local cache and
attaches it to the session. Any query executed within the session will
be stored in the local cache so further executions of the same query
with the same input parameters will not hit the database. The local
cache is cleared upon update, commit, rollback and close.