We Keep Coding

Spring Data JPA: Efficiently Query The Database for A Large Dataset

I have written an application to scrape a huge set of reviews. For each review i store the review itself Review_Table(User_Id, Trail_Id, Rating), the Username (Id, Username, UserLink) and the Trail which is build previously in the code (Id, ...60 other attributes)
for(Element card: reviewCards){
String userName = card.select("expression").text();
String userLink = card.select("expression").attr("href");
String userRatingString = card.select("expression").attr("aria-label");
Double userRating;
if(userRatingString.equals("NaN Stars")){
userRating = 0.0;
}else {
userRating = Double.parseDouble(userRatingString.replaceAll("[^0-9.]", ""));
}
User u;
Rating r;
//probably this is the bottleneck
if(userService.getByUserLink(userLink)!=null){
u = new User(userName, userLink, new HashSet<Rating>());
r = Rating.builder()
.user(u)
.userRating(userRating)
.trail(t)
.build();
}else {
u = userService.getByUserLink(userLink);
r = Rating.builder()
.user(u)
.userRating(userRating)
.trail(t)
.build();
}
i = i +1;
ratingSet.add(r);
userSet.add(u);
}
saveToDb(userSet, t, link, ratingSet);
savedEntities = savedEntities + 1;
log.info(savedEntities + " Saved Entities");
}
The code works fine for small-medium sized dataset but i encounter a huge bottleneck for larger datasets. Let's suppose i have 13K user entities already stored in the PostgresDB and another batch of 8500 reviews comes to be scraped, i have to check for every review if the user of that review is already stored. This is taking forever
I tried to define and index on the UserLink attribute in Postgres but the speed didn't improve at all
I tried to take and collect all the users stored in the Db inside a set and use the contains method to check if a particular user already exists in the set (in this way I thought I could bypass the database bottleneck of 8k write and read but in a risky way because if the users inside the db table were too much i would have encountered a memory overflow). The speed, again, didn't improve
At this point I don't have any other idea to improve this

Well for one, you would certainly benefit from not querying for each user individually in a loop. What you can do is query & cache for only the UserLink or UserName meaning get & cache the complete set of only one of them because that's what you seem to need to differentiate in the if-else.
You can actually query for individual fields with Spring Data JPA #Query either directly or even with Spring Data JPA Projections to query subset of fields if needed and cache & use them for the lookup. If you think the users could run into millions or billions then you could think of using a distributed cache like Apache Ignite where your collection could scale easily.
Btw, the if-else seem to be inversed is it not?
Next you don't store each review individually which the above code appears to imply. You can write in batches. Also since you are using Postgres you can use Postgres CopyManager provided by Postgres for bulk data transfer by using it with Spring Data Custom repositories. So you can keep writing to a new text/csv file locally at a set schedule (every x minutes) and use this to write that batched text/csv to the table (after that x minutes) and remove the file. This would be really quick.
The other option is write a stored procedure that combines the above & invoke it again in a custom repository.
Please let me know which one you had like elaborated..
UPDATE (Jan 12 2022):
One other item i missed is when you querying for UserLink or UserName you can use a very efficient form of select query that Postgres supports instead of using an IN clause like below,
#Select("select u from user u where u.userLink = ANY('{:userLinks}'::varchar[])", nativeQuery = true)
List<Users> getUsersByLinks(#Param("userLinks") String[] userLinks);

How do I query OrientDB Vertex graph object by Record ID in Java?

How do I retrieve an OrientDB Document/Object or Graph object using its Record ID? (Language: Java)
I'm referring to http://orientdb.com/docs/2.0/orientdb.wiki/Tutorial-Record-ID.html and Vertex.getId() / Edge.getId() methods.
It is just like an SQL query "SELECT * from aTable WHERE ID = 1".
Usage/purpose description: I want to store the generated ID after it is created by OrientDB, and later retrieve the same object using the same ID.

(1) I'd suggest using OrientDB 2.1, and its documentation, e.g. http://orientdb.com/docs/2.1/Tutorial-Record-ID.html
(2) From your post, it's unclear to me whether you need help obtaining the RID from the results of a query, or retrieving an object given its RID, so let me begin by mentioning that the former can be accomplished as illustrated by this example (in the case of an INSERT query):
ODocument result=db.command(new OCommandSQL(<INSERTQUERY>)).execute();
System.out.println(result.field("#rid"));
Going the other way around, there are several approaches. I have verified that the following does work using Version 2.1.8:
OrientGraph graph = new OrientGraph("plocal:PATH_TO_DB", "admin", "admin");
Vertex v = graph.getVertex("#16:0");
An alternative and more generic approach is to construct and execute a SELECT query of the form SELECT FROM :RID, along the lines of this example:
List<ODocument> results = db.query(new OSQLSynchQuery<ODocument>("select from " + rid));
for (ODocument aDoc : results) {
System.out.println(aDoc.field("name"));
}
(3) In practice, it will usually be better to use some other "handle" on OrientDB vertices and edges in Java code, or indeed when using any of the supported programming languages. For example, once one has a vertex as a Java Vertex, as in the "Vertex v" example above, one can usually use it.

Spring Data PageImpl not returning page with the correct size?

I am trying to create a new Page using a list of objects retrieved from the database. First I get all the elements from the DB, convert it to a Stream and then use lambda to filter the results. Then I need a Page with a set number of elements, however, instantiating a new PageImpl doesn't seem to return a page with the correct size.
Here is my code:
List<Produtos> listaFinal;
Stream<Produtos> stream = produtosRepository.findAll().stream();
listaFinal = stream.filter(p -> p.getProdNome().contains("uio")).collect(Collectors.toList());
long total = listaFinal.size();
Page<Produtos> imp = new PageImpl<>(listaFinal,pageable,total);
Here's a screenshot from debugging:
Note the size in the Pageable object is set to 20 and it understands that it needs 4 pages to render the 70 elements, but it returns the whole list.
What am I missing?
Edit answering the comment made by Thomas:
I understand how to use Page to return just a slice of the data. The code I showed was my attempt to use a lambda expression to filter my collection. The problem for me is I want to use Java 8's lambda to query the database via Spring Data JPA. Im used to VB.NET's and Entity function(x) query expressions and was wondering how to do the same with Spring JPA.
In my repository, Im using extends JpaRepository<Produtos, Integer>, QueryDslPredicateExecutor<Produtos> which gives me access to findAll(Predicate,Pageable). However, the Predicate is not typed so I cant simply use p -> p.getProdNome().contains("uio") in the query. I'm using SQL Server and Hibernate.

To extend stites' answer, a PagedListHolder is the way to go and here is how:
List<String> list = // ...
// Creation
PagedListHolder page = new PagedListHolder(list);
page.setPageSize(10); // number of items per page
page.setPage(0); // set to first page
// Retrieval
page.getPageCount(); // number of pages
page.getPageList(); // a List which represents the current page
If you need sorting, use another PagedListHolder constructor with a MutableSortDefinition.

PageImpl is not intended to perform any kind of pagination of your list. From the docs you can see that it's just the "basic Page implementation" which almost sounds like what you want, but it's really misleading.
Use PagedListHolder which is a simple state holder for handling lists of objects, separating them into pages.

After learning more about how Spring Data works I ended up using #Query annotations on my methods inside the JpaRepository implementations to properly query the DB and filter the results, eliminating the need to use a stream and then convert back to Page.
Here's how the code above would look in case anyone needs an example:
#Query("select p from Produtos p where p.prodNome = ?1")
public Page<Produtos> productsListByName(String prodNome, Pageable pageable)
Im aware of Spring's findBy methods but sometimes the method names become really difficult to read depending on the amount of parameters so I just stuck to JPQL.
Doing it this way the Page's content will always have up to the maximum amount of elements defined by you in the Spring configuration.
I also use a custom implementation of PageImpl, I'm not at work right now and don't have access to the code, but I'll post it whenever I can.
Edit: Custom implementation can be found here

If I understood your code right, then your intent is to load all records from the database and and split them into x buckets that are collected in the PageImpl, right?
Thats not how it used to work. The actual intent of the Pageable and Page abstraction is NOT
having to query all the data but just the "slice" of data that is needed.
In your case you could query the data via Page<X> page = repository.findAll(pageable); and simply return that.
Page holds the records for the current page alongside some additional information like e.g., the total number of records and whether there is a next page.
In your client code you can use that information to render a list of records and generating next / prev links appropriately.
Note that a query with Page<X> as result type issues 2 queries (1 to determine the overall total count for the query and 1 for the actual page data).
If you don't need the information about the total number of results but still want to be able to generate a next link you should
use Slice<X> as a return type - since it only issues 1 query.

I was also facing the same issue and found the way for it.
The SimpleJpaRepository has the method:
public Page<T> findAll(Specification<T> spec, Pageable pageable) {
TypedQuery<T> query = getQuery(spec, pageable);
return pageable == null ? new PageImpl<T>(query.getResultList())
: readPage(query, getDomainClass(), pageable, spec);
}
Which is used to return Page<T> in case you are extending JpaRepository. So we can use the same functionality here (need to rewrite the code, as Spring doesn't give you public method to have full pagination support).
If you look at the method PageImpl<>(List<T> content, Pageable pageable, long total); it just set's the value whatever you give in pageable. Here you are sending content as full list, but spring doesn't do it for it's internal purpose.
Need to replace Page<Produtos> imp = new PageImpl<>(listaFinal,pageable,total);
Following code:.
TypedQuery<User> query = entityManager.createQuery(criteriaQuery);
// Users type can be replaced with any other entity
query.setFirstResult(pageable.getOffset());
query.setMaxResults(pageable.getPageSize());
List<User> users = query.getResultList();
Page<User> result = PageableExecutionUtils.getPage(users,pageable,
() -> getCountForQuery(User.class));
Method getCountForQuery:
private Long getCountForQuery(Class<?> t) {
CriteriaBuilder criteriaBuilder=entityManager.getCriteriaBuilder();
CriteriaQuery<Long> countQuery = criteriaBuilder
.createQuery(Long.class);
countQuery.select(criteriaBuilder.count(
countQuery.from(t)));
Long count = entityManager.createQuery(countQuery)
.getSingleResult();
return count;
}
You can find the usage of PageableExecutionUtils.getPage in:
readPage(TypedQuery<S> query, final Class<S> domainClass,
Pageable pageable, final Specification<S> spec)
Method in SimpleJpaRepository which is mostly used by findAll internal method.

After applying a lot of methodes, this was the working solution in my case:
int pageSize = pageable.getPageSize();
long pageOffset = pageable.getOffset();
long total = pageOffset + list.size() + (list.size() == pageSize ? pageSize : 0);
Page<listType> page = new PageImpl<listType>(list, pageable,total)

previously my code was written like this
Pageable pageable = PageRequest.of(pageNo,size);
Query query = new Query().with(pageable);
and i was getting this for
"pageNumber": 0,
"pageSize": 5,
"size": 5,
"numberOfElements": 5,
"pageNumber": 0,
"pageSize": 10,
"size": 10,
"numberOfElements": 8,
8 is the total actual element i have in my DB
I changed that to this
Pageable pageable = PageRequest.of(pageNo,size);
Query query = new Query();
now i'm getting the actual number of item for any size of page
"pageNumber": 0,
"pageSize": 5,
"size": 5,
"numberOfElements": 8,

Activiti BPM get Variables within Task

is it possible to get all process or task variables using TaskService:
processEngine.getTaskService.createTaskQuery().list();
I know there is an opportunity to get variables via
processEngine.getTaskService().getVariable()
or
processEngine.getRuntimeService().getVariable()
but every of operation above goes to database. If I have list of 100 tasks I'll make 100 queries to DB. I don't want to use this approach.
Is there any other way to get task or process related variables?

Unfortunately, there is no way to do that via the "official" query API! However, what you could do is writing a custom MyBatis query as described here:
https://app.camunda.com/confluence/display/foxUserGuide/Performance+Tuning+with+custom+Queries
(Note: Everything described in the article also works for bare Activiti, you do not need the fox engine for that!)
This way you could write a query which selects tasks along with the variables in one step. At my company we used this solution as we had the exact same performance problem.
A drawback of this solution is that custom queries need to be maintained. For instance, if you upgrade your Activiti version, you will need to ensure that your custom query still fits the database schema (e.g., via integration tests).

If it is not possible to use the API as elsvene says, you can query yourself the database. Activiti has several tables on the database.
You have act_ru_variable, were the currently running processes store the variables. For the already finished processess you have act_hi_procvariable. Probably you can find a detailed explanation on what is on each table in activiti userguide.
So you just need to make queries like
SELECT *
FROM act_ru_variable
WHERE *Something*

The following Test, sends a value object (Person) to a process which just adds a few tracking infos for demonstration.
I had the same problem, to get the value object after execution the service to do some validation in my test.
The following piece of code shows the execution and the gathering of the task varaible after the execution was finished.
#Test
public void justATest() {
Map<String, Object> inVariables = new HashMap<String, Object>();
Person person = new Person();
person.setName("Jens");
inVariables.put("person", person);
ProcessInstance processInstance = runtimeService.startProcessInstanceByKey("event01", inVariables);
String processDefinitionId = processInstance.getProcessDefinitionId();
String id = processInstance.getId();
System.out.println("id " + id + " " + processDefinitionId);
List<HistoricVariableInstance> outVariables =
historyService.createHistoricVariableInstanceQuery().processInstanceId(id).list();
for (HistoricVariableInstance historicVariableInstance : outVariables) {
String variableName = historicVariableInstance.getVariableName();
System.out.println(variableName);
Person person1 = (Person) historicVariableInstance.getValue();
System.out.println(person1.toString());
}
}

Cleanest way to build an SQL string in Java

I want to build an SQL string to do database manipulation (updates, deletes, inserts, selects, that sort of thing) - instead of the awful string concat method using millions of "+"'s and quotes which is unreadable at best - there must be a better way.
I did think of using MessageFormat - but its supposed to be used for user messages, although I think it would do a reasonable job - but I guess there should be something more aligned to SQL type operations in the java sql libraries.
Would Groovy be any good?

First of all consider using query parameters in prepared statements:
PreparedStatement stm = c.prepareStatement("UPDATE user_table SET name=? WHERE id=?");
stm.setString(1, "the name");
stm.setInt(2, 345);
stm.executeUpdate();
The other thing that can be done is to keep all queries in properties file. For example
in a queries.properties file can place the above query:
update_query=UPDATE user_table SET name=? WHERE id=?
Then with the help of a simple utility class:
public class Queries {
private static final String propFileName = "queries.properties";
private static Properties props;
public static Properties getQueries() throws SQLException {
InputStream is =
Queries.class.getResourceAsStream("/" + propFileName);
if (is == null){
throw new SQLException("Unable to load property file: " + propFileName);
}
//singleton
if(props == null){
props = new Properties();
try {
props.load(is);
} catch (IOException e) {
throw new SQLException("Unable to load property file: " + propFileName + "\n" + e.getMessage());
}
}
return props;
}
public static String getQuery(String query) throws SQLException{
return getQueries().getProperty(query);
}
}
you might use your queries as follows:
PreparedStatement stm = c.prepareStatement(Queries.getQuery("update_query"));
This is a rather simple solution, but works well.

For arbitrary SQL, use jOOQ. jOOQ currently supports SELECT, INSERT, UPDATE, DELETE, TRUNCATE, and MERGE. You can create SQL like this:
String sql1 = DSL.using(SQLDialect.MYSQL)
.select(A, B, C)
.from(MY_TABLE)
.where(A.equal(5))
.and(B.greaterThan(8))
.getSQL();
String sql2 = DSL.using(SQLDialect.MYSQL)
.insertInto(MY_TABLE)
.values(A, 1)
.values(B, 2)
.getSQL();
String sql3 = DSL.using(SQLDialect.MYSQL)
.update(MY_TABLE)
.set(A, 1)
.set(B, 2)
.where(C.greaterThan(5))
.getSQL();
Instead of obtaining the SQL string, you could also just execute it, using jOOQ. See
http://www.jooq.org
(Disclaimer: I work for the company behind jOOQ)

One technology you should consider is SQLJ - a way to embed SQL statements directly in Java. As a simple example, you might have the following in a file called TestQueries.sqlj:
public class TestQueries
{
public String getUsername(int id)
{
String username;
#sql
{
select username into :username
from users
where pkey = :id
};
return username;
}
}
There is an additional precompile step which takes your .sqlj files and translates them into pure Java - in short, it looks for the special blocks delimited with
#sql
{
...
}
and turns them into JDBC calls. There are several key benefits to using SQLJ:
completely abstracts away the JDBC layer - programmers only need to think about Java and SQL
the translator can be made to check your queries for syntax etc. against the database at compile time
ability to directly bind Java variables in queries using the ":" prefix
There are implementations of the translator around for most of the major database vendors, so you should be able to find everything you need easily.

I am wondering if you are after something like Squiggle (GitHub). Also something very useful is jDBI. It won't help you with the queries though.

I would have a look at Spring JDBC. I use it whenever I need to execute SQLs programatically. Example:
int countOfActorsNamedJoe
= jdbcTemplate.queryForInt("select count(0) from t_actors where first_name = ?", new Object[]{"Joe"});
It's really great for any kind of sql execution, especially querying; it will help you map resultsets to objects, without adding the complexity of a complete ORM.

I tend to use Spring's Named JDBC Parameters so I can write a standard string like "select * from blah where colX=':someValue'"; I think that's pretty readable.
An alternative would be to supply the string in a separate .sql file and read the contents in using a utility method.
Oh, also worth having a look at Squill: https://squill.dev.java.net/docs/tutorial.html

I second the recommendations for using an ORM like Hibernate. However, there are certainly situations where that doesn't work, so I'll take this opportunity to tout some stuff that i've helped to write: SqlBuilder is a java library for dynamically building sql statements using the "builder" style. it's fairly powerful and fairly flexible.

I have been working on a Java servlet application that needs to construct very dynamic SQL statements for adhoc reporting purposes. The basic function of the app is to feed a bunch of named HTTP request parameters into a pre-coded query, and generate a nicely formatted table of output. I used Spring MVC and the dependency injection framework to store all of my SQL queries in XML files and load them into the reporting application, along with the table formatting information. Eventually, the reporting requirements became more complicated than the capabilities of the existing parameter mapping frameworks and I had to write my own. It was an interesting exercise in development and produced a framework for parameter mapping much more robust than anything else I could find.
The new parameter mappings looked as such:
select app.name as "App",
${optional(" app.owner as "Owner", "):showOwner}
sv.name as "Server", sum(act.trans_ct) as "Trans"
from activity_records act, servers sv, applications app
where act.server_id = sv.id
and act.app_id = app.id
and sv.id = ${integer(0,50):serverId}
and app.id in ${integerList(50):appId}
group by app.name, ${optional(" app.owner, "):showOwner} sv.name
order by app.name, sv.name
The beauty of the resulting framework was that it could process HTTP request parameters directly into the query with proper type checking and limit checking. No extra mappings required for input validation. In the example query above, the parameter named serverId
would be checked to make sure it could cast to an integer and was in the range of 0-50. The parameter appId would be processed as an array of integers, with a length limit of 50. If the field showOwner is present and set to "true", the bits of SQL in the quotes will be added to the generated query for the optional field mappings. field Several more parameter type mappings are available including optional segments of SQL with further parameter mappings. It allows for as complex of a query mapping as the developer can come up with. It even has controls in the report configuration to determine whether a given query will have the final mappings via a PreparedStatement or simply ran as a pre-built query.
For the sample Http request values:
showOwner: true
serverId: 20
appId: 1,2,3,5,7,11,13
It would produce the following SQL:
select app.name as "App",
app.owner as "Owner",
sv.name as "Server", sum(act.trans_ct) as "Trans"
from activity_records act, servers sv, applications app
where act.server_id = sv.id
and act.app_id = app.id
and sv.id = 20
and app.id in (1,2,3,5,7,11,13)
group by app.name, app.owner, sv.name
order by app.name, sv.name
I really think that Spring or Hibernate or one of those frameworks should offer a more robust mapping mechanism that verifies types, allows for complex data types like arrays and other such features. I wrote my engine for only my purposes, it isn't quite read for general release. It only works with Oracle queries at the moment and all of the code belongs to a big corporation. Someday I may take my ideas and build a new open source framework, but I'm hoping one of the existing big players will take up the challenge.

Why do you want to generate all the sql by hand? Have you looked at an ORM like Hibernate Depending on your project it will probably do at least 95% of what you need, do it in a cleaner way then raw SQL, and if you need to get the last bit of performance you can create the SQL queries that need to be hand tuned.

You can also have a look at MyBatis (www.mybatis.org) . It helps you write SQL statements outside your java code and maps the sql results into your java objects among other things.

Google provides a library called the Room Persitence Library which provides a very clean way of writing SQL for Android Apps, basically an abstraction layer over underlying SQLite Database. Bellow is short code snippet from the official website:
#Dao
public interface UserDao {
#Query("SELECT * FROM user")
List<User> getAll();
#Query("SELECT * FROM user WHERE uid IN (:userIds)")
List<User> loadAllByIds(int[] userIds);
#Query("SELECT * FROM user WHERE first_name LIKE :first AND "
+ "last_name LIKE :last LIMIT 1")
User findByName(String first, String last);
#Insert
void insertAll(User... users);
#Delete
void delete(User user);
}
There are more examples and better documentation in the official docs for the library.
There is also one called MentaBean which is a Java ORM. It has nice features and seems to be pretty simple way of writing SQL.

Read an XML file.
You can read it from an XML file. Its easy to maintain and work with.
There are standard STaX, DOM, SAX parsers available out there to make it few lines of code in java.
Do more with attributes
You can have some semantic information with attributes on the tag to help do more with the SQL. This can be the method name or query type or anything that helps you code less.
Maintaince
You can put the xml outside the jar and easily maintain it. Same benefits as a properties file.
Conversion
XML is extensible and easily convertible to other formats.
Use Case
Metamug uses xml to configure REST resource files with sql.

If you put the SQL strings in a properties file and then read that in you can keep the SQL strings in a plain text file.
That doesn't solve the SQL type issues, but at least it makes copying&pasting from TOAD or sqlplus much easier.

How do you get string concatenation, aside from long SQL strings in PreparedStatements (that you could easily provide in a text file and load as a resource anyway) that you break over several lines?
You aren't creating SQL strings directly are you? That's the biggest no-no in programming. Please use PreparedStatements, and supply the data as parameters. It reduces the chance of SQL Injection vastly.