We Keep Coding

How to do Streaming at controller layer Spring

I am developing one simple Java Spring application and trying to get all the saved Order from database. Application flow is simple though have around 100k record in database. I used streaming at JPA layer from DB to application data is coming fine.
But when this start transferring from Controller to client something it throws 502 response. Following is piece of code for all three layer.
Could someone help to solve this problem? Requirement all this data should come without using any filter or pagination.
#GetMapping(produces = "application/json")
#ResponseStatus(HttpStatus.OK)
public Resources getOrders() {
return userService.getOrders();
}
#Transactional
public Resources getOrders() {
Stream<Orders> orders = streamAll();
List<Order> orderList = new ArrayList<>();
orderList.addAll(orderConverter.createFromEntities(orders));
Resource resources = new Resource();
resources.setResources(orderList);
return resources;
}
#Query("select u from Order")
Stream<Orders> streamAll();

You are getting a 502 response because your JVM is exhausted in memory. Use streaming detached objects at the repository layer.
First, check if your database supports fetching a result set through streaming data. For example, MYSQL supports it using a read-only session and fetch size with value Integer.MIN_VALUE [1].
Later, customize your repository to open a stateless session with StatelessSession interface [2]. Thanks to this interface you are getting objects detached, so Hibernate releases it immediately and no cost memory in cache.
If you need to make changes, remember again, your objects are detached. You need open another session (read-write). Your objects will be saved with these session, and after saving it, you must detach it manually.
This example covers the first requirement:
#PersistenceContext
private EntityManager entityManager;
public void stackoverflowQuestion() {
Session currentSession = entityManager.unwrap(Session.class);
ScrollableResults rows = currentSession
.createQuery("SELECT u FROM Order", Orders.class)
.setFetchSize(Integer.MIN_VALUE)
.scroll(ScrollMode.FORWARD_ONLY);
while (rows.next()) {
Orders order = (Orders) rows.get(0);
...
}
}
[1] https://dev.mysql.com/doc/connector-j/5.1/en/connector-j-reference-implementation-notes.html
[2] https://docs.jboss.org/hibernate/orm/3.5/reference/en-US/html/batch.html

Hooking into start of JDBC transaction

I have a Spring Boot webapp connected to a Postgres 9.6 database.
I use Spring's JdbcTemplate for executing SQL statements.
Each table in my database has triggers for INSERT, CREATE and DELETE statments. These triggers copy the affected rows into a history table.
I want the triggers to also save the application user ID of the user who made the change.
According to https://stackoverflow.com/a/13172964/2591231 I can achieve my goal by having the application insert the current user id into a temporary table at the start of each transaction and having the triggers read from the temporary table.
A similar method, mentioned in several other places, is executing:
SET LOCAL application_name = "my_application_user", then reading application_name inside the triggers. Again, this has to be done at the start of each transaction.
I'm looking for way, which is orthogonal to business code (I don't want each DAO to explicitly set user ID), to hook into the start of each transaction in order to run a specific SQL statement before any other statement in the same transaction.
I need this to work for both implicit transactions (single invocations of JdbcTemplate) and transactions demarcated declaratively with Spring's #Transactional annotation.

First of all, JdbcTemplate does not provide transaction support out-of-the-box (see here). So, in order to intercept all #Transaction annotated code AND every call to JdbcTemplate, this could be done at DataSource level, as commented earlier by Serge Bogatyrev.
I have a Spring Web project where I tested this approach. I defined a replacement DataSource #Bean called MyDataSource that extends BasicDataSource, and overwrites its getConnection() method so that it creates the temp table and insert the user_id before returning the connection.
It worked for #Transaction calls and pure JdbcTemplate calls.
If you want to strictly tie this temp table update at the start of each transaction, do this same strategy for defining the PlatformTransactionManager #Bean. You only need to overwrite the doBegin() method. And don't forget to annotate with #Transaction all methods calling JdbcTemplate.
PS1: Make sure to call DROP TABLE IF EXISTS temp_table_name prior creating the temp table, in order to replace the DISCARD ALL on connection returning to pool, as mentioned here.
PS2: This whole solution of creating a temp table doesn't smell well. I wouldn't implement it myself. I would prefer to take a deep breath and add created_by and updated_by columns to all my tables.

You can take advantage of Spring AOP for setting up the user. The aspect will make a call to the database to set up the application user.
In my example, a stored procedure is used to set up the application user responsible for creating, modifying, or deleting a record . You can customize it according to your requirements. Here is the example aspect which retrieves the user from the HTTP request and then makes a call to the stored procedure,
#Component
#Aspect
#Slf4j
public class SetUserAspect {
private final JdbcTemplate jdbcTemplate;
#Autowired
public SetUserAspect(JdbcTemplate jdbcTemplate) {
this.jdbcTemplate = jdbcTemplate;
}
#Before("execution(* com.basaki.example.service.BookService.*(..) )")
public void setUser(JoinPoint jp) {
log.info("In class: " + jp.getSignature().getDeclaringTypeName() +
" - before method: " + jp.getSignature().getName());
HttpServletRequest request =
((ServletRequestAttributes) RequestContextHolder.currentRequestAttributes()).getRequest();
if (request != null) {
String user = request.getHeader("user");
if (user != null) {
log.info("Setting user " + user);
SimpleJdbcCall
jdbcCall = new SimpleJdbcCall(jdbcTemplate)
.withSchemaName("example_book_schema")
.withFunctionName("set_user");
SqlParameterSource
in =
new MapSqlParameterSource().addValue("audit_user",
user);
jdbcCall.executeFunction(String.class, in);
}
}
}
}
All the CRUD operations are performed in BookService, essentially a DAO.
Here is the stored procedure used for setting up the user,
CREATE OR REPLACE FUNCTION example_book_schema.set_user(
audit_user TEXT
) RETURNS BOOLEAN STABLE LANGUAGE SQL AS $$
SELECT set_config('book.audit_user', audit_user, true);
SELECT TRUE;
Restricting Pointcuts to Only Transactional Methods
You can restrict the points cuts to only transactional methods in BookService by adding an additional clause in the Before advice.
#Before("execution(* com.basaki.example.service.BookService.*(..) ) " +
"&& #annotation(annotation)")
public void setUser(final JoinPoint jp, final Transactional annotation) {
...
}

You can use #EntityListeners to listen change of entity in application context, then collect whatever information (entity value, authentication user, etc...) and then insert to your history table. Example you can follow here: http://www.baeldung.com/database-auditing-jpa

You can create view, add user id column, and use your triggers to deal with updates. So that is yet another way to code it at DB side. That way you are supposed to pass it every time, so no other changes are needed.
Going to Java/Spring side.
A bit oldish style: TransactionTemplate - that way you have full control, but your dao needs more code, since transaction management needs to be done there.
Other option is to create proxy on
org.springframework.jdbc.datasource.DataSourceTransactionManager and do your job at doBegin, then your proxy needs to be passed to transaction manager. And that is the way to go for me.

Spring Data Rest transaction boundaries

I've been implementing some business logic/validation with the Spring Data Rest and JPA repositories using repository validators as described in:
http://docs.spring.io/spring-data/rest/docs/current/reference/html/#events.
After digging deeper into the SDR code I noticed that the validators (or more generally, repository listeners) are not invoked within a transaction.
From the source code of org.springframework.data.rest.webmvc.RepositoryEntityController:
private ResponseEntity<ResourceSupport> createAndReturn(Object domainObject, RepositoryInvoker invoker,
PersistentEntityResourceAssembler assembler, boolean returnBody) {
// validation logic is implemented in the listener, no transaction yet
publisher.publishEvent(new BeforeCreateEvent(domainObject));
// invoker calls repository, which is wrapped in the transactional proxy,
// only then transaction begins
Object savedObject = invoker.invokeSave(domainObject);
publisher.publishEvent(new AfterCreateEvent(savedObject));
PersistentEntityResource resource = returnBody ? assembler.toFullResource(savedObject) : null;
HttpHeaders headers = prepareHeaders(resource);
addLocationHeader(headers, assembler, savedObject);
return ControllerUtils.toResponseEntity(HttpStatus.CREATED, headers, resource);
}
As seen in the code, the listeners are not called within a transaction, which could lead to eventual data consistency issues.
Am I missing something? Or the framework simply sets the transactional boundary incorrectly?

In spring data rest the repository method would run its own transaction. I also think that this is problematic in some cases. At least the event handler should be run in the same transaction as the repository method.
There was a similar question here:
Handle spring-data-rest application events within the transaction
Especially this answer provides a workaround that allows you to wrap the whole RepositoryEntityController method in a transaction - I think this is what you need most of the time:
https://stackoverflow.com/a/30713264/5371736

Open Session In View (OSIV) and Hibernate Session flush

Following is a hypothetical situation on Spring 3.x and Hibernate3.x
I have a service layer in spring which invokes 3 DAOs to construct a model.
The DAOs are transactional(#Transactional) and have lazy loaded hibernate collections.
The service method causes a few updates ,along with the fetch of data.
A typical DAO method will be as follows -
public O create(I entity) throws GenericException {
getOrCreateSession().save(entity);
return (O)entity;
}
I have the following questions around OSIV -
1.How many times is this session flushed(database update) in the default AUTO mode?
2.Can OSIV be made to extend the session beyond a single request (to a conversation)?

The AUTO flush mode will execute the pending DML statements when:
the current transaction is committed
when a query might target an entity table, that's current enqueued for flushing
Spring Webflow has support for long conversations.

How can I commit a Play! JPA transaction manually?

Usually, Play! commits the transaction after a request completes successfully.
What is the correct way to commit a transaction manually in Play?
void addPerson() {
Person p = new Person("John", "Doe");
p.save();
// TODO - commit the transaction
// Now p should have an ID
assert p.id != null;
usePersonIdForSomethingNasty(p.id);
}

You can get the Hibernate EntityManager by calling JPA.em(). Then, from there, you have access to the transaction (JPA.em().getTransaction()).
If you intend to manage the transaction yourself, you will want to disable Play!'s transaction handling (there is a #NoTransaction annotation you can use on the method or controller to do that). Otherwise, Play! will try to commit the transaction at the end of the request anyway, and if you have already done that yourself, that will cause an exception.

You don't need to do anything. After the request finishes without any exception, the transaction will be commited for you.
Just ensure to call "save" on all entities you want to persist at the end of transaction.