I would like to load data in cache when my Spring Boot application starts.
I know there is an integrated way using BootstrapCacheLoader in Ehcache2.
How to load data from database to Ehcache when the application starts
But I don't see this in Ehcache3.
I still can do it manually within a #postConstruct method.
But I was wondering if there is an integrating solution (Spring 5, Ehcache 3)
Thank you.
I've ended up doing it after the Spring context has been initialized.
For each element in the DB collection, I call getResourceById() which has the #Cacheable annotation, thus populating the cache for the whole DB collection.
I don't recommend to run this code in a #PostConstruct as proxies may not have been created yet and annotations like #Cacheable may not be working yet.
Running this code when a ContextRefreshedEvent event is triggered (after initialization or after refresh) is a more appropriate place to load the cache.
public class CacheLoader {
private final Logger logger = LoggerFactory.getLogger(CacheLoader.class);
#Autowired
private ResourcePermissionRepository resourcePermissionRepository;
#Autowired
private ResourcePermissionService resourcePermissionService;
#EventListener
public void onApplicationEvent(ContextRefreshedEvent event) {
logger.info("Loading cache following start/refresh event");
for (PermissionGroup permissionGroup : permissionGroupRepository.findAll()) {
permissionGroupService.getGroupById(permissionGroup.getGroupName());
}
for(ResourcePermission resourcePermission: resourcePermissionRepository.findAll()) {
resourcePermissionService.getResourceById(resourcePermission.getResourceId());
}
logger.info("Finished loading cache");
}
public class ResourcePermissionService {
private final Logger logger = LoggerFactory.getLogger(ResourcePermissionService.class);
#Autowired
private ResourcePermissionRepository resourcePermissionRepository;
#Cacheable(value = "resources", sync = true)
public ResourcePermission getResourceById(String resourceId) {
logger.info("Cache miss for resource " + resourceId);
return resourcePermissionRepository.findById(resourceId).orElse(new NullResourcePermission());
}
#CachePut(value = "resources", key = "#result.resourceId")
public ResourcePermission addResourcePermission(ResourcePermission resourcePermission) {
return resourcePermissionRepository.save(resourcePermission);
}
#CacheEvict(value = "resources")
public void deleteById(String resourceId) {
resourcePermissionRepository.deleteById(resourceId);
}
}
Related
I want to load both the LookupCode and Location data from database
into cache memory using Spring ehCache when the application starts i.e
when the server starts before any other method is called. In future
few more dropdowns will be added. So there should be a common method
to cache whatever datas comes in based on the criteria of the dropdown
data.
There is a Entity, Repository and Service already written for
Lookupcode and Location
I have written the below for implementing caching framework:
ehcache.xml
<cache name= "LookupCodeRepository.getDropdownValues"/> <cache name= "LocationRepository.getDropdownValues"/>
application.properties
spring.jpa.properties.hibernate.cache.use_second_level_cache = false
spring.jpa.properties.hibernate.cache.use_query_cache = false
spring.jpa.properties.hibernate.cache.region.factory_class =
org.hibernate.cache.ehcache.EhCacheRegionFactory
spring.jpa.properties.hibernate.cache.provider_class =
org.hibernate.cache.EhCacheProvider
spring.jpa.properties.hibernate.cache.use_structured_entries = true
spring.jpa.properties.hibernate.cache.region_prefix =
spring.jpa.properties.hibernate.cache.provider_configuration_file_resource_path
= ehcache.xml spring.jpa.properties.hibernate.cache.use_second_level_cache
and using hibernate-ehcache jar in pom.xml
WebConfig.java
#Configuration public class WebConfig implements
ServletContextInitializer{
#Autowired CustomCache cache;
#Override public void onStartup ( ServletContext servletContext)
throws ServletException{
cache.loadCache();
}
CustomCache.java
public class CustomCache {
#Autowired private LookupCodeService lkupSer;
#Autowired private LocationService locSer;
public void loadCache(){
List<LookupCode> lkup = lkupServ.getDropdownValues();
List<Location> locat = locSer.getDropdownValues();
}
So here in loadCache() method instead of calling each individual
service it should be like, automatic. Whatever service is created
it should automatically be cached. So there should be a common method
to cache whatever datas comes in based on the criteria of the
dropdown data.
How to implement that?
The services you want to work with have a common method. Define an interface for that method:
interface ProvidesDropdownValues<T> {
List<T> getDropdownValues();
}
Now you can do:
class DropdownValuesService {
#Autowired ApplicationContext context;
#Cacheable List getDropdownValues(String beanName) {
ProvidesDropdownValues<?> bean = ((ProvidesDropdownValues) context.getBean(beanName));
return bean.getDropdownValues();
}
}
If your services don't have bean names you could work with class names instead.
For load on startup you could do:
class StartupWarmupService {
#Autowired ApplicationContext context;
#Autowired DropdownValuesService dropDowns;
#PostConstruct void startup() {
for (String n : context.getBeanNamesForType(ProvidesDropdownValues.class)) {
dropDowns.getDropdownValues(n);
}
}
}
I suggest that the load code only runs in the production application. That is why it makes sense to keep it separate from the general caching logic. For testing a single service you don't want to load everything. Startup times for developers should be fast.
Disclaimer: I am not a heavy Spring user, so details may be wrong but the basic approach should work out.
I have a Spring boot Application connecting to SQL Server Database. I need some help in using caching in my application. I have a table for CodeCategory which has a list of codes for Many codes. This table will be loaded every month and data changes only once in a month.
I want to cache this entire table when the Application starts. In any subsequent calls to the table should get value from this cache instead of calling the Database.
For Example,
List<CodeCategory> findAll();
I want to cache the above DB query value during application startup. If there is a DB call like List<CodeCategory> findByCodeValue(String code) should fetch the code result from the already Cached data instead of calling the Database.
Please let me know how this can be achieved using spring boot and ehcache.
As pointed out, It takes some time for ehcache to setup and it is not working completely with #PostConstruct. In that case make use of ApplicationStartedEvent to load the cache.
GitHub Repo: spring-ehcache-demo
#Service
class CodeCategoryService{
#EventListener(classes = ApplicationStartedEvent.class )
public void listenToStart(ApplicationStartedEvent event) {
this.repo.findByCodeValue("100");
}
}
interface CodeCategoryRepository extends JpaRepository<CodeCategory, Long>{
#Cacheable(value = "codeValues")
List<CodeCategory> findByCodeValue(String code);
}
Note: There are multiple ways as pointed by others. You can choose as per your needs.
My way is to define a generic cache handler
#FunctionalInterface
public interface GenericCacheHandler {
List<CodeCategory> findAll();
}
And its implementation as below
#Component
#EnableScheduling // Important
public class GenericCacheHandlerImpl implements GenericCacheHandler {
#Autowired
private CodeRepository codeRepo;
private List<CodeCategory> codes = new ArrayList<>();
#PostConstruct
private void intializeBudgetState() {
List<CodeCategory> codeList = codeRepo.findAll();
// Any customization goes here
codes = codeList;
}
#Override
public List<CodeCategory> getCodes() {
return codes;
}
}
Call it in Service layer as below
#Service
public class CodeServiceImpl implements CodeService {
#Autowired
private GenericCacheHandler genericCacheHandler;
#Override
public CodeDTO anyMethod() {
return genericCacheHandler.getCodes();
}
}
Use the second level hibernate caching to cache all the required db queries.
For caching at the application start-up, we can use #PostContruct in any of the Service class.
Syntax will be :-
#Service
public class anyService{
#PostConstruct
public void init(){
//call any method
}
}
Use CommandLineRunner interface.
Basically , you can create a Spring #Component and implement CommandLineRunner interface. You will have to override it's run method. The run method will be called at the start of the app.
#Component
public class DatabaseLoader implements
CommandLineRunner {
#override
Public void run(.... string){
// Any code here gets called at the start of the app.
}}
This approach is mostly used to bootstrap the application with some initial data.
In a spring boot application, I have multiple implementations of an interface, every one of which accesses to the database using a JPA repository.
#Autowired
List<? extends CrawlerService> crawlers;
This is the repository:
#Repository
public interface HotelRepository extends JpaRepository<Hotel, Long> {
public Optional<Hotel> findByTextAndSource(String text, String source);
}
And the implementations look like this:
#Service
public class MyService implements CrawlerService {
#Autowired
DocumentService documentService;
#Autowired
HotelRepository hotelRepository;
private Logger logger = LoggerFactory.getLogger("MyService");
private Set<String> visitedLinks = new HashSet<>();
#Override
public void start() {
execute("https://www.example.com/");
}
public void execute(String url) {
if(visitedLinks.contains(url))
return;
try {
logger.info("Connecting '{}' ...", url);
visitedLinks.add(url);
Document document = Jsoup.connect(url).get();
String text = documentService.getText(document);
// this is where I access to the database
if(hotelRepository.findByTextAndSource(text, "EXAMPLE").isPresent()) {
return;
}
Hotel hotel = new Hotel();
hotel.setUrl(url);
hotel.setText(text);
hotel.setSource("EXAMPLE");
hotelRepository.save(hotel);
// find other links
findLinks(document, url).forEach(this::execute);
} catch (IOException e) {
logger.error("Connection error for {}", url);
}
}
}
Now I'm using parallel stream to run each one of them in parallel:
crawlers.stream().parallel().forEach(CrawlerService::start);
They continue to work until the point that they want to access the database through the JPA repository. Then all of them stop, except the one assigned to the main thread. This is observed using the output of my logger:
2018-11-01 21:42:38.408 INFO 7641 --- [ main] MyService ...
Only this one continues to run.
I want all of them to continue to work in parallel. How can I solve this problem?
It is worth mentioning that I'm using MySQL and Spring Boot 2 with default settings for connection pool.
I am using Spring Boot 1.5.9 on Tomcat 9.0.2 and I am trying to cache lookups using spring #Cacheable scheduling a cache refresh job that runs on application startup and repeats every 24 hours as follows:
#Component
public class RefreshCacheJob {
private static final Logger logger = LoggerFactory.getLogger(RefreshCacheJob.class);
#Autowired
private CacheService cacheService;
#Scheduled(fixedRate = 3600000 * 24, initialDelay = 0)
public void refreshCache() {
try {
cacheService.refreshAllCaches();
} catch (Exception e) {
logger.error("Exception in RefreshCacheJob", e);
}
}
}
and the cache service is as follows:
#Service
public class CacheService {
private static final Logger logger = LoggerFactory.getLogger(CacheService.class);
#Autowired
private CouponTypeRepository couponTypeRepository;
#CacheEvict(cacheNames = Constants.CACHE_NAME_COUPONS_TYPES, allEntries = true)
public void clearCouponsTypesCache() {}
public void refreshAllCaches() {
clearCouponsTypesCache();
List<CouponType> couponTypeList = couponTypeRepository.getCoupons();
logger.info("######### couponTypeList: " + couponTypeList.size());
}
}
the repository code:
public interface CouponTypeRepository extends JpaRepository<CouponType, BigInteger> {
#Query("from CouponType where active=true and expiryDate > CURRENT_DATE order by priority")
#Cacheable(cacheNames = Constants.CACHE_NAME_COUPONS_TYPES)
List<CouponType> getCoupons();
}
later in my webservice, when trying to get the lookup as follows:
#GET
#Produces(MediaType.APPLICATION_JSON + ";charset=utf-8")
#Path("/getCoupons")
#ApiOperation(value = "")
public ServiceResponse getCoupons(#HeaderParam("token") String token, #HeaderParam("lang") String lang) throws Exception {
try {
List<CouponType> couponsList = couponRepository.getCoupons();
logger.info("###### couponsList: " + couponsList.size());
return new ServiceResponse(ErrorCodeEnum.SUCCESS_CODE, resultList, errorCodeRepository, lang);
} catch (Exception e) {
logger.error("Exception in getCoupons webservice: ", e);
return new ServiceResponse(ErrorCodeEnum.SYSTEM_ERROR_CODE, errorCodeRepository, lang);
}
}
The first call it gets the lookup from the database and the subsequent calls it gets it from the cache, while it should get it from the cache in the first call in the web service?
Why am I having this behavior, and how can I fix it?
The issue was fixed after upgrading to Tomcat 9.0.4
While it's not affecting the scheduled task per se, when refreshAllCaches() is invoked in the CacheService, #CacheEvict on clearCouponsTypesCache() is bypassed since it's invoked from the same class (see this answer). It will lead to cache not being purged before
List<CouponType> couponTypeList = couponTypeRepository.getCoupons();
is invoked. This means that the #Cacheable getCoupons() method will not query the database, but will instead return values from the cache.
This makes the scheduled cache refresh action to do its work properly only once, when the cache is empty. After that it's useless.
The #CacheEvict annotation should be moved to refreshAllCaches() method and add beforeInvocation=true parameter to it, so the cache is purged before being populated, not after.
Also, when using Spring 4 / Spring Boot 1.X, these bugs should be taken into consideration:
https://github.com/spring-projects/spring-boot/issues/8331
https://jira.spring.io/browse/SPR-15271
While this bug doesn't seem to affect this specific program, it might be a good idea to separate #Cacheable annotation from JpaRepository interface until migration to Spring 5 / Spring Boot 2.X.
#CacheEvict won't be invoked when called within the same service. This is because Spring creates a proxy around the service and only calls from "outside" go through the cache proxy.
The solution is to either add
#CacheEvict(cacheNames = Constants.CACHE_NAME_COUPONS_TYPES, allEntries = true)
to refreshAllCaches too, or to move refreshAllCaches into a new service that calls ICacheService.clearCouponsTypeCache.
In my application, I have a scenario where I have to refresh cache each 24hrs.
I'm expecting database downtime so I need to implement a use case to refresh cache after 24hrs only if the database is up running.
I'm using spring-ehache and I did implement simple cache to refresh for each 24 hrs, but unable to get my head around to make the retention possible on database downtime .
Conceptually you could split the scheduling and cache eviction into two modules and only clear your cache if certain condition (in this case, database's healthcheck returns true) is met:
SomeCachedService.java:
class SomeCachedService {
#Autowired
private YourDao dao;
#Cacheable("your-cache")
public YourData getData() {
return dao.queryForData();
}
#CacheEvict("your-cache")
public void evictCache() {
// no body needed
}
}
CacheMonitor.java
class CacheMonitor {
#Autowired
private SomeCachedService service;
#Autowired
private YourDao dao;
#Scheduled(fixedDelay = TimeUnit.DAYS.toMillis(1))
public conditionallyClearCache() {
if (dao.isDatabaseUp()) {
service.evictCache();
}
}
}
Ehcache also allows you to create a custom eviction algorithm but the documentation doesn't seem too helpful in this case.