I am developing one simple Java Spring application and trying to get all the saved Order from database. Application flow is simple though have around 100k record in database. I used streaming at JPA layer from DB to application data is coming fine.
But when this start transferring from Controller to client something it throws 502 response. Following is piece of code for all three layer.
Could someone help to solve this problem? Requirement all this data should come without using any filter or pagination.
#GetMapping(produces = "application/json")
#ResponseStatus(HttpStatus.OK)
public Resources getOrders() {
return userService.getOrders();
}
#Transactional
public Resources getOrders() {
Stream<Orders> orders = streamAll();
List<Order> orderList = new ArrayList<>();
orderList.addAll(orderConverter.createFromEntities(orders));
Resource resources = new Resource();
resources.setResources(orderList);
return resources;
}
#Query("select u from Order")
Stream<Orders> streamAll();
You are getting a 502 response because your JVM is exhausted in memory. Use streaming detached objects at the repository layer.
First, check if your database supports fetching a result set through streaming data. For example, MYSQL supports it using a read-only session and fetch size with value Integer.MIN_VALUE [1].
Later, customize your repository to open a stateless session with StatelessSession interface [2]. Thanks to this interface you are getting objects detached, so Hibernate releases it immediately and no cost memory in cache.
If you need to make changes, remember again, your objects are detached. You need open another session (read-write). Your objects will be saved with these session, and after saving it, you must detach it manually.
This example covers the first requirement:
#PersistenceContext
private EntityManager entityManager;
public void stackoverflowQuestion() {
Session currentSession = entityManager.unwrap(Session.class);
ScrollableResults rows = currentSession
.createQuery("SELECT u FROM Order", Orders.class)
.setFetchSize(Integer.MIN_VALUE)
.scroll(ScrollMode.FORWARD_ONLY);
while (rows.next()) {
Orders order = (Orders) rows.get(0);
...
}
}
[1] https://dev.mysql.com/doc/connector-j/5.1/en/connector-j-reference-implementation-notes.html
[2] https://docs.jboss.org/hibernate/orm/3.5/reference/en-US/html/batch.html
Related
I am building a RESTful web service that can be consumed by a browser or another web service.
I am willing to reduce the bandwidth through caching, however i want the method to be executed and send the actual data only if it's different than the last modified cache.
From my understanding of the #cacheable annotation, the method is only executed once and the output is cached until the cache expires .
Also #CachePut executes everytime and updates the cache but does it send the cache again even if it's not updated?
summary is: i need the client to be able to send the last modified date of it's cache and only get a new data if it has been modified.
Also how does Spring handle the client side caching and if-modified-since headers? does i need to save the last modified time or it is automatically handled ?
No, you need to do it by yourself.
You need to annotate your "fetch" method with #Cacheable(docs) and then, annotate "update" method with #CacheEvict (docs) in order to "drop" your cache. So when you would fetch your data next time after its modification, it will be fresh.
Alternatively, you can create another method with #CacheEvict and manually call it from "update" method.
The cache related annotations (#Cacheable, #CacheEvict etc) will only deal with the cache being maintained by application. Any http response header like last-modified etc has to be managed seperately. Spring MVC provides a handy way to deal with it (docs).
The logic to calculate the last modified time has to be obviously application specific.
An example of its usage would be
MyController {
#Autowire
CacheService cacheService;
#RequestMapping(value = "/testCache", method = RequestMethod.GET)
public String myControllerMethod(WebRequest webRequest, Model model, HttpServletResponse response) {
long lastModified = // calculate as per your logic and add headers to response
if (request.checkNotModified(lastModified)) {
// stop processing
return null;
} else {
return cacheService.getData(model);
}
}
#Component
public class CacheService{
#Cacheable(value = "users", key = "#id")
public String getData(Model model) {
//populate Model
return "dataview";
}
The question is about using multiple transaction in crudrepository, jparepository ext.
In my project, there are two entities. RequestEntity and SendingMailEntity.
WorkFlow in my method:
1) save RequestEntity ,
2) send informationService(it is an rest service that purchased by us. we can't control its any exception.)
3) save SendingMailEntity.
When there is an exception on number 2 or 3, we lost requestEntity because of rollback that is controlled by spring jpa.
The records of requestEntity are never to be lost.
How can I control this issue ? How can I have two independent transaction in spring data ?
Thanks for help.
You need to create a method in your service specifically for managing / saving the requestEntity and annotate it appropriately so that the current transaction is paused, and this code run in a new transaction and commited upon exit from the method:
#Transactional(propagation = Propagation.REQUIRES_NEW)
public void processRequestEntity(...){
// jpa repo actions
}
I'm writing web app with spring security, I have already default security implementation it works fine, but I have question about getting data from database. How can I keep information from database without executing query to database everytime?
Look at this. User is my entity class, but for me it is not effective to retrive data from database everytime. Everytime I refresh that /welcome it will execute query, I'm using Spring data jpa so it's fine but does not make sense since there is no change in database. So what I want to do is to keep user and retrive his data from db once in my app. Is there any way to do it?
#RequestMapping(value = "/userpanel")
public String userpanel(Model model, Principal principal){
String loggedUserName = principal.getName();
Optional<User> user = userService.findByUserName(loggedUserName);
if(user.isPresent()){
model.addAttribute("user", user.get());
}
return "userpanel";
}
That's exactly what a custom UserDetails object is for. Make your User entity implement UserDetails, and make your UserDetailsService return it.
This way, SecurityContextHolder.getContext().getAuthentication().getPrincipal() will return your entity. You will also be able to inject it using #AuthenticationPrincipal.
This of course assumes that you have not set session creation to stateless (i.e. your security provider is not re-authenticating the user for every request).
Just remember that it is bad practice to keep sensitive data in memory for a prolonged period of time.
Spring includes its own caching system: https://spring.io/guides/gs/caching/
To use it, annotate a method that may take while to fetch something from the database with #Cacheable(String)
Then, in the application class, you can add the #EnableCaching annotation.
Google Guava also has its own caching system.
Example usage is like so:
LoadingCache<Key, Graph> graphs = CacheBuilder.newBuilder()
.maximumSize(1000)
.expireAfterWrite(10, TimeUnit.MINUTES)
.build(
new CacheLoader<String, User>() {
public Graph load(User user) throws AnyException {
return loadFromDatabase(user);
}
});
This will store a maximum of 1000 users in memory and deletes entries when it gets too big. It will also delete entries after 10 minutes.
It acts like a map, but calls the load(User) method if the key (user ID) is not present.
I have a Spring Boot webapp connected to a Postgres 9.6 database.
I use Spring's JdbcTemplate for executing SQL statements.
Each table in my database has triggers for INSERT, CREATE and DELETE statments. These triggers copy the affected rows into a history table.
I want the triggers to also save the application user ID of the user who made the change.
According to https://stackoverflow.com/a/13172964/2591231 I can achieve my goal by having the application insert the current user id into a temporary table at the start of each transaction and having the triggers read from the temporary table.
A similar method, mentioned in several other places, is executing:
SET LOCAL application_name = "my_application_user", then reading application_name inside the triggers. Again, this has to be done at the start of each transaction.
I'm looking for way, which is orthogonal to business code (I don't want each DAO to explicitly set user ID), to hook into the start of each transaction in order to run a specific SQL statement before any other statement in the same transaction.
I need this to work for both implicit transactions (single invocations of JdbcTemplate) and transactions demarcated declaratively with Spring's #Transactional annotation.
First of all, JdbcTemplate does not provide transaction support out-of-the-box (see here). So, in order to intercept all #Transaction annotated code AND every call to JdbcTemplate, this could be done at DataSource level, as commented earlier by Serge Bogatyrev.
I have a Spring Web project where I tested this approach. I defined a replacement DataSource #Bean called MyDataSource that extends BasicDataSource, and overwrites its getConnection() method so that it creates the temp table and insert the user_id before returning the connection.
It worked for #Transaction calls and pure JdbcTemplate calls.
If you want to strictly tie this temp table update at the start of each transaction, do this same strategy for defining the PlatformTransactionManager #Bean. You only need to overwrite the doBegin() method. And don't forget to annotate with #Transaction all methods calling JdbcTemplate.
PS1: Make sure to call DROP TABLE IF EXISTS temp_table_name prior creating the temp table, in order to replace the DISCARD ALL on connection returning to pool, as mentioned here.
PS2: This whole solution of creating a temp table doesn't smell well. I wouldn't implement it myself. I would prefer to take a deep breath and add created_by and updated_by columns to all my tables.
You can take advantage of Spring AOP for setting up the user. The aspect will make a call to the database to set up the application user.
In my example, a stored procedure is used to set up the application user responsible for creating, modifying, or deleting a record . You can customize it according to your requirements. Here is the example aspect which retrieves the user from the HTTP request and then makes a call to the stored procedure,
#Component
#Aspect
#Slf4j
public class SetUserAspect {
private final JdbcTemplate jdbcTemplate;
#Autowired
public SetUserAspect(JdbcTemplate jdbcTemplate) {
this.jdbcTemplate = jdbcTemplate;
}
#Before("execution(* com.basaki.example.service.BookService.*(..) )")
public void setUser(JoinPoint jp) {
log.info("In class: " + jp.getSignature().getDeclaringTypeName() +
" - before method: " + jp.getSignature().getName());
HttpServletRequest request =
((ServletRequestAttributes) RequestContextHolder.currentRequestAttributes()).getRequest();
if (request != null) {
String user = request.getHeader("user");
if (user != null) {
log.info("Setting user " + user);
SimpleJdbcCall
jdbcCall = new SimpleJdbcCall(jdbcTemplate)
.withSchemaName("example_book_schema")
.withFunctionName("set_user");
SqlParameterSource
in =
new MapSqlParameterSource().addValue("audit_user",
user);
jdbcCall.executeFunction(String.class, in);
}
}
}
}
All the CRUD operations are performed in BookService, essentially a DAO.
Here is the stored procedure used for setting up the user,
CREATE OR REPLACE FUNCTION example_book_schema.set_user(
audit_user TEXT
) RETURNS BOOLEAN STABLE LANGUAGE SQL AS $$
SELECT set_config('book.audit_user', audit_user, true);
SELECT TRUE;
Restricting Pointcuts to Only Transactional Methods
You can restrict the points cuts to only transactional methods in BookService by adding an additional clause in the Before advice.
#Before("execution(* com.basaki.example.service.BookService.*(..) ) " +
"&& #annotation(annotation)")
public void setUser(final JoinPoint jp, final Transactional annotation) {
...
}
You can use #EntityListeners to listen change of entity in application context, then collect whatever information (entity value, authentication user, etc...) and then insert to your history table. Example you can follow here: http://www.baeldung.com/database-auditing-jpa
You can create view, add user id column, and use your triggers to deal with updates. So that is yet another way to code it at DB side. That way you are supposed to pass it every time, so no other changes are needed.
Going to Java/Spring side.
A bit oldish style: TransactionTemplate - that way you have full control, but your dao needs more code, since transaction management needs to be done there.
Other option is to create proxy on
org.springframework.jdbc.datasource.DataSourceTransactionManager and do your job at doBegin, then your proxy needs to be passed to transaction manager. And that is the way to go for me.
I have a Spring, Hibernate, and Wicket application set up to read internationalized json content items from a database and pass them out via a request on an api url. The codebase responsible for passing out the data is a smaller part of an overall website structure developed for an enterprise client.
The api functions fine in over 90 percent of cases, but the client is experiencing an interesting occassional issue that might be stemming from orphaned hibernate sessions. The request will fail via the php script and give the error:
Warning: file_get_contents( http://client.net/api/attachment_lines?ce=false&language=en®ion=na&ts=1341592326) [function.file-get-contents]: failed to open stream: Redirection limit reached, aborting in client_api->send_request() (line 38 of <sitepath>/api.class.php).
And will spawn the following error in the tomcat server log:
09:15:00,200 ERROR [RequestCycle] failed to lazily initialize a collection of role: com.client.data.AttachmentLineCode.attachmentSublineCodes, no session or session was closed
org.hibernate.LazyInitializationException: failed to lazily initialize a collection of role: com.client.data.AttachmentLineCode.attachmentSublineCodes, no session or session was closed
The application is configured within spring to use the OpenSessionInViewFilter and #Transactional annotation design pattern, so I'm not sure what's causing intermittent request failures. In addition to this, the client states that the api will continue to fail for about 15 minutes following the issue, which seems really wacky given the configuration. Within the web.xml, here is the declaration of the filter:
<filter>
<filter-name>openEntityManagerInView</filter-name>
<filter-class>org.springframework.orm.hibernate3.support.OpenSessionInViewFilter</filter-class>
</filter>
<filter-mapping>
<filter-name>openEntityManagerInView</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>
Within code, here is the transactional annotation on the generic DAO which is extended by the Content Item DAO:
#Transactional(noRollbackFor={javax.persistence.EntityNotFoundException.class, org.springframework.orm.ObjectRetrievalFailureException.class})
public class GenericDaoHibernate<T, PK extends Serializable> implements GenericDao<T, PK> {
#Autowired
private SessionFactory sessionFactory;
Within the generic DAO as well, here is where I retrieve and use sessions:
protected Session getSession() {
return sessionFactory.getCurrentSession();
}
protected Criteria createCacheableCriteria(Class<T> clazz) {
Criteria criteria = createNonCacheableCriteria(clazz);
criteria.setCacheable(true);
criteria.setCacheMode(CacheMode.NORMAL);
return criteria;
}
protected Criteria createCacheableCriteria(Class<?> clazz, String alias) {
Criteria criteria = createNonCacheableCriteria(clazz, alias);
criteria.setCacheable(true);
criteria.setCacheMode(CacheMode.NORMAL);
return criteria;
}
protected Criteria createNonCacheableCriteria(Class<?> clazz) {
Session session = getSession();
Criteria criteria = session.createCriteria(clazz);
criteria.setCacheable(false);
criteria.setCacheMode(CacheMode.IGNORE);
return criteria;
}
protected Criteria createNonCacheableCriteria(Class<?> clazz, String alias) {
Session session = getSession();
Criteria criteria = session.createCriteria(clazz, alias);
criteria.setCacheable(false);
criteria.setCacheMode(CacheMode.IGNORE);
return criteria;
}
Is there some sort of way that the session could get orphaned in this set up? Is there some sort of built in timeout to hibernate sessions that could be causing this issue? Possibly issues with caching? Thanks in advance for your help.
The solution here has nothing to do with Hibernate or Spring, and resides solely in my error not noting the differences between the production environment and our development/staging. The production environment implemented a complex load balancing strategy without sticky sessions.
It turns out that Wicket's Request/Response cycle involves caching a buffered response following a POST. The corresponding GET coming back to pick up that response would throw a 302 occasionally because the load balancing would forward the request to a server without the cached response, and the proxy objects would be lost in oblivion. The relevant piece of code I chose to implement to resolve this is placed within my Application.java under init():
public class ClientApplication extends SpringWebApplication {
...
public void init() {
...
getRequestCycleSettings().setRenderStrategy(IRequestCycleSettings.ONE_PASS_RENDER);
This changes Wicket's rendering strategy configuration to not buffer responses. An issue emerges as a result that allows the classic "refresh double submit" problem. As a result, this isn't necessarily an ideal solution, but the client didn't want to use sticky session enabled load balancing, and didn't mind having the double submit issue.
For more details on this issue, and a far more eloquent/structured answer, see: http://blog.comsysto.com/2011/04/08/lost-in-redirection-with-apache-wicket/