Spring Boot JPA Hibernate JVM heap is not released - java

The following is a simplification of a more complicated setup. I describe this simple case to show the effect.
I start Spring Boot and call a method. In this method all the contents of a MySQL database table is read via
Iterable<myPojo> myPojos = myPojoRepository.findAll();
Afterwards I leave this method.
After finishing this method I get the message
Started Application in 70.893 seconds (JVM running for 72.899)
So Spring Boot is idling afterwards.
But still the memory is not released.
How can I avoid that the JVM Heap is not released after the application is idling?
This is the result of VisualJM after the application is idling:
char[] 1.080.623.712 (21.0%) 24.040.578 (23.4%)
byte[] 1.034.070.352 (20.1%) 17.280.824 (16.8%)
java.lang.String 768.935.872 (14.9%) 24.029.246 (23.4%)
java.lang.Object[] 556.181.104 (10.8%) 5.320.276 (5.1%)
org.hibernate.engine.internal.MutableEntityEntry
231.287.232 (4.5%) 2.628.264 (2.5%)
org.hibernate.engine.spi.EntityKey
224.752.040 (4.3%) 5.618.801 (5.4%)
byte[][] 212.407.904 (4.1%) 3.318.832 (3.2%)
hello.web.model.MyPojo 185.852.968 (3.6%) 3.318.803 (3.2%)
java.util.HashMap$Node 145.238.976 (2.8%) 3.025.812 (2.9%)
com.mysql.jdbc.ByteArrayRow 132.752.120 (2.5%) 3.318.803 (3.2%)
org.hibernate.engine.internal.EntityEntryContext$ManagedEntityImpl
126.156.720 (2.4%) 2.628.265 (2.5%)
hello.web.model.MyPojoCompoundKey
120.376.680 (2.3%) 3.009.417 (2.9%)
java.util.HashMap$Node[] 108.307.328 (2.1%) 16.558 (0.0%)
java.lang.Float 79.651.320 (1.5%) 3.318.805 (3.2%)
int[] 41.885.056 (0.8%) 54.511 (0.0%)
java.util.LinkedHashMap$Entry 15.519.616 (0.3%) 242.494 (0.2%)
java.io.File 11.323.392 (0.2%) 235.904 (0.2%)
org.springframework.boot.devtools.filewatch.FileSnapshot
10.550.400 (0.2%) 219.800 (0.2%)
java.lang.String[] 8.018.808 (0.1%) 52.031 (0.0%)
java.lang.reflect.Method 6.015.040 (0.1%) 37.594 (0.0%)
java.io.File[] 2.283.528 (0.0%) 16.746 (0.0%)
My effective-pom.xml shows that hibernate 5.0.9.Final is used.
The table my_pojo contains 3.3 million entries.
MyPojoRepository:
package hello.web.model;
import com.querydsl.core.types.Predicate;
import org.springframework.data.querydsl.QueryDslPredicateExecutor;
import org.springframework.data.repository.PagingAndSortingRepository;
import java.util.List;
public interface MyPojoRepository
extends PagingAndSortingRepository<MyPojo, Long>,
QueryDslPredicateExecutor<MyPojo> {
List<MyPojo> findAll(Predicate predicate);
}
MyPojo:
package hello.web.model;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import org.apache.commons.beanutils.BeanComparator;
import org.apache.commons.collections.comparators.ComparatorChain;
import javax.persistence.Entity;
import javax.persistence.Id;
import javax.persistence.IdClass;
import java.io.Serializable;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
#Data
#Entity
#Builder
#AllArgsConstructor
#IdClass(MyPojoCompoundKey.class)
public class MyPojo implements Serializable, Comparable<MyPojo> {
public MyPojo() { }
#Id
private String myId1;
#Id
private String myId2;
#Id
private String myId3;
private Float myId4;
private String myId5;
#Override
public int compareTo(MyPojo o) {
return this.getMyId3().compareTo(o.getMyId3());
}
protected boolean canEqual(Object other) {
return other instanceof MyPojo;
}
public static void sortByMyId1MyId3(List<MyPojo> myPojos) {
ComparatorChain chain = new ComparatorChain(Arrays.asList(
new BeanComparator("myId1"),
new BeanComparator("myId3")
));
Collections.sort(myPojos, chain);
}
}
myId1-3 and myId5 have a length of 10 characters in avarage.
So again:
How can I avoid that the JVM Heap is not released after the application is idling?

Related

Difference between map and Flow

From reading a Google groups post from 2016 : “.map() is converted to a .via()”
src : https://groups.google.com/g/akka-user/c/EzHygZpcCHg
Are the following lines of code equivalent :
Source.repeat(json).take(3).via(mapToDtoFlow).to(printSink).run(actorSystem);
Source.repeat(json).take(3).map(x -> mapper.readValue(x, RequestDto.class)).to(printSink).run(actorSystem);
Are there scenarios when a map should be used instead of flow?
src :
RequestDTO :
import com.fasterxml.jackson.annotation.JsonFormat;
import lombok.Builder;
import lombok.Getter;
import lombok.Setter;
import lombok.ToString;
import lombok.extern.jackson.Jacksonized;
import java.util.Date;
#Getter
#Setter
#Builder
#ToString
#Jacksonized
public class RequestDto {
#JsonFormat(pattern = "yyyy-MM-dd HH:mm:sss")
private final Date datePurchased;
}
StreamManager (contains main method) :
import akka.Done;
import akka.NotUsed;
import akka.actor.typed.ActorSystem;
import akka.actor.typed.javadsl.Behaviors;
import akka.stream.javadsl.Flow;
import akka.stream.javadsl.Sink;
import akka.stream.javadsl.Source;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.concurrent.CompletionStage;
public class StreamManager {
final static ObjectMapper mapper = new ObjectMapper();
private static final Flow<String, RequestDto, NotUsed> mapToDtoFlow = Flow.of(String.class)
.map(input -> mapper.readValue(input, RequestDto.class))
.log("error");
public static void main(String args[]) {
final ActorSystem actorSystem = ActorSystem.create(Behaviors.empty(), "actorSystem");
final Sink<RequestDto, CompletionStage<Done>> printSink = Sink.foreach(System.out::println);
final String json = "{\"datePurchased\":\"2022-03-03 21:32:017\"}";
Source.repeat(json).take(3).via(mapToDtoFlow).to(printSink).run(actorSystem);
Source.repeat(json).take(3).map(x -> mapper.readValue(x, RequestDto.class)).to(printSink).run(actorSystem);
}
}
map is converted to a via, but it's not an exactly syntactically equivalent via as you'd get from Flow.of().map().
The first would translate to a .via(Map(f)), where Map is a GraphStage which implements the map operation.
In the second case, the mapToDtoFlow (ignoring the log) would itself be (in Scala notation) Flow[String].via(Map(f)) so you'd be adding another layer of via: .via(Flow[String].via(Map(f))).
For all intents and purposes, they're the same (I suspect that the materializer, when it comes time to interpret the RunnableGraph you've built, will treat them identically).
Taking the .log into account, mapToDtoFlow is equivalent (again in Scala):
Flow[String]
.via(Map(f))
.via(Log(...))
There are basically three levels of defining streams in Akka Streams, from highest level to lowest level:
the Java/Scala DSLs
the Java/Scala Graph DSLs
GraphStages
The DSLs merely specify succinct ways of building GraphStages and the fundamental way to link GraphStages with Flow shape is through the via operation.

The method getType() is undefined for the type Ingredient

package tacos.web;
import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;
import org.springframework.stereotype.Controller;
import org.springframework.ui.Model;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.ModelAttribute;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.SessionAttributes;
import java.lang.reflect.Field;
import lombok.extern.slf4j.Slf4j;
import tacos.Ingredient;
import tacos.Ingredient.Type;
import tacos.Taco;
#Slf4j
#Controller
#RequestMapping("/design")
#SessionAttributes("tacoOrder")
public class DesignTacoController {
#ModelAttribute
public void addIngredientsToModel(Model model) {
List<Ingredient> ingredients = Arrays.asList(
new Ingredient("FLTO", "Flour Tortilla", Type.WRAP),
new Ingredient("COTO", "Corn Tortilla", Type.WRAP),
new Ingredient("GRBF", "Ground Beef", Type.PROTEIN),
new Ingredient("CARN", "Carnitas", Type.PROTEIN),
new Ingredient("TMTO", "Diced Tomatoes", Type.VEGGIES),
new Ingredient("LETC", "Lettuce", Type.VEGGIES),
new Ingredient("CHED", "Cheddar", Type.CHEESE),
new Ingredient("JACK", "Monterrey Jack", Type.CHEESE),
new Ingredient("SLSA", "Salsa", Type.SAUCE),
new Ingredient("SRCR", "Sour Cream", Type.SAUCE)
);
Type[] types = Ingredient.Type.values();
for (Type type : types) {
model.addAttribute(type.toString().toLowerCase(),
filterByType(ingredients, type));
}
}
#GetMapping
public String showDesignForm(Model model) {
model.addAttribute("taco", new Taco());
return "design";
}
private Iterable<Ingredient> filterByType(
List<Ingredient> ingredients, Type type) {
return ingredients
.stream()
.filter(x -> x.getType().equals(type))
.collect(Collectors.toList());
}
}
I was going through the book Spring in action edition 6 chapter. In that in the filterByType method the '.getType()' is showing the error
The method getType() is undefined for the type Ingredient
I thought it was the error due to lombok but I have installed that as well. I have also import the package 'java.lang.reflect.Field' but still getting the error.
package tacos;
import lombok.Data;
#Data
public class Ingredient {
public Ingredient(String string, String string2, Type wrap) {
// TODO Auto-generated constructor stub
}
private final String id = "";
private final String name = "";
private final Type type = null;
public enum Type {
WRAP, PROTEIN, VEGGIES, CHEESE, SAUCE
}
}
The above class is the Ingredient Class
Seems you are not the first person who faced with this issue https://coderanch.com/t/730026/java/lombok
In addIngredientsToModel of class DesignTacoController the flagged
error is "The constructor Ingredient(String, String, Ingredient.Type)
is undefined". Also, in method filterByType the flagged error is "The
method getType() is undefined for the type Ingredient". It zappears
that lombok is just not working. But I have lombok in the pom:
Answer:
Just adding Lombok as a dependency does not make Eclipse recognize it,
you'll need a plugin for that. See https://www.baeldung.com/lombok-ide
for instructions on installing Lombok into Eclipse (and IntelliJ for
those who prefer it).

Neo4j Plugin Recursive

I'm trying to create my own function for neo4j that recursively goes through a graph and returns any nodes and edges that are connected to an edge with a long value greater than 100.
I know there is a simple CYPHER query for it but by doing this, I can know how to proceed with more complex stuff on my own.
pseudocode
get all relations from node matching Id where the relationship is type 'TypeExample'.
if the relation has a long property "Count" and Count > 100, go to 1.
IF 5 nodes deep, stop. return list of nodes and edges with interface IPath.
package example;
import java.util.Iterator;
import java.util.stream.Stream;
import java.util.stream.StreamSupport;
import org.neo4j.graphdb.GraphDatabaseService;
import org.neo4j.graphdb.Label;
import org.neo4j.graphdb.Node;
import org.neo4j.graphdb.Relationship;
import org.neo4j.graphdb.ResourceIterator;
import org.neo4j.logging.Log;
import org.neo4j.procedure.*;
import org.neo4j.procedure.Description;
import org.neo4j.procedure.Name;
import java.util.List;
import java.util.Map;
import java.util.Set;
import org.neo4j.graphdb.index.Index;
import org.neo4j.graphdb.index.IndexManager;
public class NodeFinder {
#Context
public GraphDatabaseService db;
#Context
public Log log;
#Procedure
#Description("finds Nodes one step away")
public Stream<SomeList> GetRelations(#Name("nodeId") long nodeId, #Name("depth") long depth, #Name("rel") String relType) {
Recursive(nodeId);
//return list of Nodes and Edges
}
private void Recursive(long id) {
Node node = db.getNodeById(nodeId);
Iterable<Relationship> rels = node.getRelationships();
for (Relationship rel : rels) {
long c = (long) rel.getProperty("Count");
if (c > 100) {
Recursive(rel.getEndNodeId());
}
}
}
}

How to use a custom function in a jpa query?

I am new to Spring Jpa and Hibernate. I am trying to fetch data using a custom function from an Oracle db. I could define an entity along with its related service, implementation and repository. In addition, I created a new custom Oracle dialect by using registerFunction as you will see below.
So I have two questions:
1) In my Oracle db, the function sits under a different schema. Do I need to specify its schema? If so how? Or will hibernate find it automatically?
I will be asking my second question at the end of this post after providing my full stacktrace...
Here is my full stack trace:
MyOracle10gDialect
package blog;
import org.hibernate.dialect.Oracle10gDialect;
import org.hibernate.dialect.function.StandardSQLFunction;
public class MyOracle10gDialect extends Oracle10gDialect {
public MyOracle10gDialect() {
super();
registerFunction("my_function", new StandardSQLFunction("my_function"));
}
}
application.properties
...
spring.jpa.database-platform=blog.MyOracle10gDialect
...
Entity:
package blog.models;
import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.Id;
import javax.persistence.Table;
#Entity
#Table(name = "item", schema = "WOS_SOURCE")
public class WosItem {
#Id
#Column(nullable = false)
private String UT;
#Column(nullable = false)
private String TI;
public String getUT() {
return UT;
}
public void setUT(String UT) {
this.UT = UT;
}
public String getTI() {
return TI;
}
public void setTI(String TI) {
this.TI = TI;
}
public WosItem(String UT, String TI) {
this.UT = UT;
this.TI = TI;
}
public WosItem() { }
#Override
public String toString() {
return "WosItem{" +
"UT='" + UT + '\'' +
", TI='" + TI + '\'' +
'}';
}
}
Service:
package blog.services;
import blog.models.WosItem;
import org.springframework.stereotype.Service;
import java.util.List;
#Service
public interface WosItemService {
List<WosItem> findAll();
WosItem findById(String id);
String find_ut(Long ut_seq);
}
Implementation:
package blog.services;
import blog.models.WosItem;
import blog.repositories.WosItemRepository;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import java.util.List;
#Service
public class WosItemServiceJpaImpl implements WosItemService {
#Autowired
private WosItemRepository wosItemRepository;
#Override
public List<WosItem> findAll() {
return this.wosItemRepository.findAll();
}
#Override
public WosItem findById(String id) {
return this.wosItemRepository.findOne(id);
}
#Override
public String find_ut(Long ut_seq) {
return this.wosItemRepository.find_ut();
}
}
Repository:
package blog.repositories;
import blog.models.WosItem;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.data.jpa.repository.Query;
import org.springframework.stereotype.Repository;
#Repository
public interface WosItemRepository extends JpaRepository<WosItem, String> {
#Query("SELECT function('my_function', input) FROM WosItem wos");
String find_ut();
}
So in my Oracle db I can use this function as shown below:
select other_schema.my_function(aa.input) from my_schema.TABLE aa;
For ex. say aa.input is 332708100009 then it returns 000332708100009
As for my second question:
2) How can I carry out this process in jpa? I am aware that my repository is not correct at all. I get an error like "Annotations are not allowed here". I could not find a way to remedy this.
Thanks in advance.
EDIT ON THROWN EXCEPTION:
Caused by: java.lang.IllegalStateException: No data type for node: org.hibernate.hql.internal.ast.tree.MethodNode
\-[METHOD_CALL] MethodNode: 'function (my_function)'
+-[METHOD_NAME] IdentNode: 'my_function' {originalText=my_function}
\-[EXPR_LIST] SqlNode: 'exprList'
\-[NAMED_PARAM] ParameterNode: '?' {name=ut_seq, expectedType=null}
Unfortunately if you want to use the JPA 2.1 feature of the custom function call in your Select statement then you will need to perform some additional actions before you can use it.
When you use it in your where statement then it works without any additional actions, but as i wanted to use it for one of my projects inside the select just as you did then you would need to:
1) Extend the hibernate dialect and register your function(s):
package com.mypkg.dialect;
import org.hibernate.dialect.Oracle10gDialect;
import org.hibernate.dialect.function.StandardSQLFunction;
import org.hibernate.type.StringType;
public class CustomOracle10gDialect extends Oracle10gDialect {
public CustomOracle10gDialect() {
super();
registerFunction("my_function"
, new StandardSQLFunction("my_function", new StringType()));
}
}
2) Edit your hibernate.dialect property of your session factory to point to that custom implementation:
<property name="hibernate.dialect" value="com.mypkg.dialect.CustomOracle10gDialect"/>
Update
If the function needs to be called from a certain schema then this would be suggested:
registerFunction("my_function"
, new StandardSQLFunction("schema.my_function", new StringType()));
Further reading -> native function calls

Play Framework PersistenceException

I'm trying to use the Play framework (Java) to simply read some data from a few Oracle tables, probably even use a few complex queries later on. I'm following a tutorial but I'm having some issues retrieving the data.
My Model class look like this:
package models;
import java.util.ArrayList;
import java.util.List;
import play.libs.F;
import javax.persistence.*;
import com.avaje.ebean.*;
import play.db.ebean.*;
#Entity
#Table(name="TABLESPACE.CAT_BONDS")
public class Cat_Bond extends Model {
#Id
public String symbol;
public String database;
public String tickType;
public String assetClass;
public String sourcePlatform;
public String sourceExchange;
public static Finder<String, Cat_Bond> find = new Finder<String, Cat_Bond>(String.class,Cat_Bond.class);
public Cat_Bond(){}
public Cat_Bond(String symbol, String database, String tickType, String assetClass,
String sourcePlatform, String sourceExchange) {
this.symbol = symbol;
this.database = database;
this.tickType = tickType;
this.assetClass = assetClass;
this.sourcePlatform = sourcePlatform;
this.sourceExchange = sourceExchange;
}
/*
* retrieve all rows from the 'cat_bonds' table
*/
public static List<Cat_Bond> findAll(){
//return new ArrayList<Cat_Bond>(cat_bond);
return find.all();
}
/*
* Find by EAN
*/
public static Cat_Bond findByEan(String symbol){
return find.where().eq("symbol", symbol).findUnique();
}
}
My controller class:
package controllers;
import java.util.List;
import views.html.*;
import models.Cat_Bond;
import play.data.Form;
import play.mvc.*;
public class Cat_Bonds extends Controller {
private static final Form<Cat_Bond> cat_bondForm = Form.form(Cat_Bond.class);
public static Result list(){
List<Cat_Bond> cat_bond = Cat_Bond.findAll();
return ok(list.render(cat_bond));
}
And the application.conf entry looks like:
#Oracle
db.default.driver=oracle.jdbc.OracleDriver
db.default.url="jdbc:oracle:thin:#server.uk.net.intra:port/ALIAS"
db.default.user=user
db.default.password=pass
# Evolutions
# ~~~~~
# You can disable evolutions if needed
evolutionplugin=disabled
Problem is when the call to list is made in the controller then to findAll() in the model I get the error:
**[PersistenceException: Query threw SQLException:ORA-00904: "T0"."SOURCE_EXCHANGE": invalid identifier Bind values:[] Query was: select t0.symbol c0, t0.database c1, t0.tick_type c2, t0.asset_class c3, t0.source_platform c4, t0.source_exchange c5 from TABLESPACE.CAT_BONDS t0 ]**
#Column(name="xx")
Was required above each variable defined in the model class that was to be mapped to the table column.
You can use
clean
compile
~run
If it doesn't work properly, you can use #EntityConcurrencyMode(ConcurrencyMode.NONE) within your model class.

Categories