Java For loop with batch SQL - java

I have a problem. Right now I am using JOOQ to insert arround 100.000 records in my database using the following code:
try (Connection conn = DriverManager.getConnection(SqlConn.getURL(), SqlConn.getUSERNAME(), SqlConn.getPASSWORD())) {
DSLContext create = DSL.using(conn, SQLDialect.MYSQL);
for (String key : trendlines.keySet()) {
for (Trendline trendline : trendlines.get(key)) {
String sql = createTrendlineQuery(trendline);
create.fetch(sql);
}
}
}
catch (Exception e) {
e.printStackTrace();
}
With the function createTrendlineQuery():
private String createTrendlineQuery(Trendline trendline) {
return "INSERT INTO Trendline (openTime, market, coin, period, metric, number, slope, interceptY, percentage, formula, data) VALUES (" +
trendline.getOpenTime() + ", '" +
trendline.getMarket() + "', '" +
trendline.getCoin() + "', '" +
trendline.getPeriod() + "', '" +
trendline.getFormula() + "') " +
"ON DUPLICATE KEY UPDATE " +
"openTime = " + trendline.getOpenTime() + ", " +
"market = '" + trendline.getMarket()+ "', " +
"coin = '" + trendline.getCoin() + "', " +
"period = '" + trendline.getPeriod() + "', " +
"formula = '" + trendline.getFormula() + "';";
}
But this gives a lot of load on my internet/database, so I found out you can do batch inserts for big data. I found the following page of JOOQ about batch inserts: https://www.jooq.org/doc/3.14/manual/sql-execution/batch-execution/. Now I think this is what I need, but I have a problem. The example looks in my case like this:
try (Connection conn = DriverManager.getConnection(SqlConn.getURL(), SqlConn.getUSERNAME(), SqlConn.getPASSWORD())) {
DSLContext create = DSL.using(conn, SQLDialect.MYSQL);
create.batch(create.insertInto(DSL.table("Trendline"), DSL.field("openTime"), DSL.field("market"), DSL.field("coin") ).values((Integer) null, null, null))
.bind( trendline.getOpenTime() , trendline.getMarket() , trendline.getCoin() )
.bind( trendline.getOpenTime() , trendline.getMarket() , trendline.getCoin() )
.bind( trendline.getOpenTime() , trendline.getMarket() , trendline.getCoin() )
.bind( trendline.getOpenTime() , trendline.getMarket() , trendline.getCoin() )
.execute();
}
catch (Exception e) {
e.printStackTrace();
}
Except that I need to put the 2 for-loops between the create.batch() to programmatically create the inserts. How can I insert the for loops and am I using the batch insert the right way to reduce internet traffic and database pressure?

Using BatchedConnection as a quick fix
The simplest solution to turning existing jOOQ code (or any JDBC based code, for that matter) into a batched JDBC interaction is to use jOOQ's BatchedConnection:
create.batched((Connection c) -> {
// Now work with this Connection c, instead of your own Connection and all the statements
// will be buffered and batched, e.g.
DSL.using(c).insertInto(...).values(...).execute();
});
Using the batch API that you've tried using
You just have to assign the BatchBindStep to a local variable in your loop and you're set:
BatchBindStep step = create.batch(query);
for (...)
step = step.bind(...);
step.execute();
Using the import API
Use the import API. Assuming you're using the code generator and you have the usual static imports
import static org.jooq.impl.DSL.*;
import static com.example.generated.Tables.*;
Write this:
create.loadInto(TRENDLINE)
.onDuplicateKeyUpdate()
.loadArrays(trendlines
.values()
.stream()
.map(t -> new Object[] {
t.getOpenTime(),
t.getMarket(),
t.getCoin(),
t.getPeriod(),
t.getFormula()
/* And the other fields which you partially omitted */
})
.toArray(Object[][]::new)
)
.fields(
TRENDLINE.OPENTIME,
TRENDLINE.MARKET,
TRENDLINE.COIN,
TRENDLINE.PERIOD,
TRENDLINE.FORMULA
/* And the other fields which you partially omitted */
)
.execute();
See also the sections about:
Throttling (you may want to play with these values to find what's optimal for your system)
Error handling
Which may be of interest. If the input Object[][] gets too large, you can chunk your input trendlines.values() collection manually on your side. If sorting your map by key is really essential (it shouldn't be from what I can tell from your question), then write this instead:
trendlines
.keySet()
.stream()
.flatMap(k -> trendlines.get(k).stream())
.map(t -> new Object[] { ... })
...
A few remarks on your own attempts
You're calling create.fetch(sql), when in fact your statement is a query with an update count, so in that case, you would have wanted to use create.execute(sql) instead.
Please never concatenate SQL strings when using jOOQ! Even when using plain SQL templating, there is never a need for concatenating SQL strings. You'll run into syntax errors and SQL injection. Please always use bind variables.
I really recommend you use jOOQ's code generator. Most benefits of using jOOQ arise when you use the code generator. Valid reasons to avoid code generation include when your schema is dynamic and not known at runtime. That's almost the only reason not to use code generation.

Related

NamedParameterJdbcTemplate not updating when using batchUpdate

I have a list of objects provided by another service which I use to update my own data. When I try to use NamedParameterJdbcTemplate.batchUpdate, all returned values are zero.
public void updateWeather(List<Weather> weatherList) {
String query = "UPDATE weather \n" +
"SET rain_probability = ROUND(:rainProbability, 4), \n" +
"wind_speed = :windSpeed \n" +
"WHERE city_id = :cityId AND date = :date;";
List<MapSqlParameterSource> batchList = new ArrayList<>();
for(Weather weather : weatherList) {
MapSqlParameterSource params = new MapSqlParameterSource();
params.addValue("rainProbability", weather.getRainProbability());
params.addValue("windSpeed", weather.getWindSpeed());
params.addValue("cityId", weather.getCityId());
params.addValue("date", weather.getDate());
batchList.add(params);
}
this.namedParameterJdbcParameter
.batchUpdate(query, batchList.toArray(new MapSqlParameterSource[] {});
}
If I run this UPDATE directly in the database, it works fine. Futhermore, if I run it one by one, that is, replacing values (instead of adding the parameter source to batchList) it also works.
For example:
for (Weather weather : weatherList) {
String query = String.format("UPDATE weather \n" +
"SET rain_probability = ROUND('%d', 4), \n" +
" wind_speed = %d \n" +
" WHERE city_id = :cityId AND date = :date;",
weather.getRainProbability(),
weather.getWindSpeed(),
weather.getCityId(),
weather.getDate()
);
this.namedParameterJdbcTemplate.update(query, Collections.emptyMap());
}
Any suggestions of what I'm doing wrong?
Is it the use of "\n" or the ";" at the end of the statement within the String? (I'm surprised you don't get a SQL Syntax exception with the ; inside the actual query string)
Also dates are always a bit tricky and if that isn't converting properly then your WHERE clause isn't going to match and is possibly why 0 rows are returned. Could you temporarily try converting dates to Strings and see if the count is correct (e.g. for Oracle: AND date = TO_DATE(:dateStr, 'DD/MM/YYYY') )

JdbcTemplate returns empty list

JdbcTemplate is returning an empty list when executing the "query" method.
public List<Loan> getLoanDAO(Employee employee, String s) {
final String SQL = "SELECT CTLPCODCIA, CTLPCODSUC, CTLPCODTRA, EMPNOMBRE, EMPAPATERN, EMPAMATERN, CTLPCODPRE, "
+ "CTLPTIPPRE, TIPDESPRE, CTLPMONEDA, CTLPESTADO, CTLPMONTOP, CTLPNROCUO, CTLPCUOTA, FLAGTIPOCUOTA, CTLGLOSA, CTLDIASFR, "
+ "CTLDOCADJ, CTLUSUCREA, CTLFECCREA "
+ "FROM HR_CTLPREC_SS INNER JOIN HR_EMPLEADO ON CTLPCODCIA=EMPCODCIA AND CTLPCODSUC=EMPCODSUC AND CTLPCODTRA=EMPCODTRA "
+ "INNER JOIN HR_TIPPRE ON CTLPCODCIA=TIPCODCIA AND CTLPCODSUC=TIPCODSUC AND CTLPTIPPRE=TIPCODPRE "
+ "WHERE TIPFLGEST = '1' AND TIPSELFSERVICE = '1' "
+ "AND CTLPCODCIA = ? AND CTLPCODSUC = ? AND EMPCODTRAJEF = ? AND CTLPESTADO = ? ";
List<Loan> loans = jdbcTemplate.query(SQL, new Object[] {
employee.getCTLPCODCIA(), employee.getCTLPCODSUC(), employee.getCTLPCODTRA(), s }, loanMapper);
return loans;
}
However, when replacing the "?" with the same parameters used in execution and executing in sqldeveloper, it returns 4 rows. I don't know what is wrong since I've been doing de data access code in the same way for all the other entities.
Problem solved
As stated by #Julian:
JdbcTemplate is a proved spring component used by a huge number of applications so in my opinion it must be a bug in your code.
It was not a problem from JdbcTemplate, neither my code. It was an issue from the IDE. I just build my project from scratch using maven console commands and the code worked as intended.
Thanks folks.
JdbcTemplate is a proved spring component used by a huge number of applications so in my opinion it must be a bug in your code.
Not sure what version of Spring you are using but jdbcTemplate.query would expect a Loan Mapper class as one of its arguments. There is no such a mapper present in your code.
I suggest you put a breakpoint just before the query and inspect the employee fields and see if they match the values you are playing in the sqldeveloper.
One other thing that it attracts my attention is the third one where u have EMPCODTRAJEF = ? in the query definition but you use employee.getCTLPCODTRA() as the argument. Obviously I don't know your data model but should it rather be employee.getEMPCODTRAJEF() or the other way around?
If this won't work double check your arguments.
final String SQL = "SELECT CTLPCODCIA, CTLPCODSUC, CTLPCODTRA, EMPNOMBRE, EMPAPATERN, EMPAMATERN, CTLPCODPRE, "
+ "CTLPTIPPRE, TIPDESPRE, CTLPMONEDA, CTLPESTADO, CTLPMONTOP, CTLPNROCUO, CTLPCUOTA, FLAGTIPOCUOTA, CTLGLOSA, CTLDIASFR, "
+ "CTLDOCADJ, CTLUSUCREA, CTLFECCREA "
+ "FROM HR_CTLPREC_SS INNER JOIN HR_EMPLEADO ON CTLPCODCIA=EMPCODCIA AND CTLPCODSUC=EMPCODSUC AND CTLPCODTRA=EMPCODTRA "
+ "INNER JOIN HR_TIPPRE ON CTLPCODCIA=TIPCODCIA AND CTLPCODSUC=TIPCODSUC AND CTLPTIPPRE=TIPCODPRE "
+ "WHERE CTLPCODCIA=? AND CTLPCODSUC = ? AND EMPCODTRAJEF = ? AND CTLPESTADO = ? "
+ "AND TIPFLGEST='1' AND TIPSELFSERVICE='1'";
Add this to application.properties to debug your query.
logging.level.org.springframework.jdbc.core = TRACE

How to parse a table name from CREATE statement using Hibernate?

This is a native create statement for some unknown database carrier
String createStatement = "CREATE TABLE test_database.test_table " +
"AS " +
"( " +
"var1, " +
"var2 " +
") " +
"; "
);
I need to parse this String test_database.test_table
I don't know in advance what SQL flavor this CREATE statement is. If I knew that, I would simply use something like
String table = createStatement.split(" ")[2];
But the above solution might not work in all databases. What if some database allows for blanks in table name? So I have to use Hibernate.
How?
In general, I don't think you can do this without certain assumptions or considering each and every SQL dialect you want to support.
Hibernate itself supportes a number of SQL dialects and you can infer a lot of things from the used dialect. However, org.hibernate.dialect.Dialect does not provide enough information for parse all the possible native CREATE TABLE statements in the selected dialect.
I don't think that Hibernate can take care of all situations especially when dealing with something like Transact-SQL or CREATE GLOBAL TEMPORARY TABLE or even CREATE TEMPORARY TABLESPACE and then you have the AS, AS SELECT, and even PARALLEL COMPRESS AS SELECT after the table name to consider.
As an alternative however you can create a method which can retrieve the Table Name from a supplied CREATE TABLE SQL string which I believe will cover most (if not all) of these issues. Below is such a method:
public String getTableNameFromCreate(final String sqlString) {
// Always rememeber...we're only trying to get the table
// name from the SQL String. We really don't care anything
// about the rest of the SQL string.
String tableName;
String wrkStrg = sqlString.replace("[", "").replace("]", "").trim();
// Is "CREATE TABLE" only
if (wrkStrg.startsWith("CREATE TABLE ")) {
wrkStrg = wrkStrg .substring(13).trim();
}
else if (wrkStrg.startsWith("CREATE GLOBAL TEMPORARY TABLE ")) {
wrkStrg = wrkStrg .substring(30).trim();
}
else if (wrkStrg.startsWith("CREATE TEMPORARY TABLESPACE ")) {
wrkStrg = wrkStrg .substring(28).trim();
}
// Is it Create Table ... AS, AS SELECT, PARALLEL COMPRESS AS,
// or PARALLEL COMPRESS AS SELECT?
if (wrkStrg.toUpperCase().contains(" PARALLEL COMPRESS ")) {
wrkStrg = wrkStrg.replace(" parallel compress ", " PARALLEL COMPRESS ");
tableName = wrkStrg.substring(0, wrkStrg.indexOf(" PARALLEL COMPRESS ")).trim();
}
else if (wrkStrg.toUpperCase().contains(" AS ")) {
wrkStrg = wrkStrg.replace(" as ", " AS ");
tableName = wrkStrg.substring(0, wrkStrg.indexOf(" AS ")).trim();
}
// Nope...none of that in the SQL String.
else {
tableName = wrkStrg.substring(0, wrkStrg.indexOf("(")).trim();
}
// return but remove quotes first if any...
return tableName.replace("\"","").replace("'", "");
}
If the database name is attached to the table name as in your example (test_database.test_table) then of course you will need to further parse off the actual table name.

How do I form this complex query in Hibernate?

I'm building a REST service using Hibernate, Spring HATEOAS and Jackson. I am adding a method which returns a JSON representation of the results of a query like the one below;
SELECT ERRORS.DMN_NAM, CODES.MSG_TXT,
FROM SERV_ERR ERRORS, EVENT_CD CODES
WHERE ERRORS.SERV_RESP_CD_TXT = CODES.CD_TXT
GROUP BY ERRORS.DMN_NAM, ERRORS.SERV_NAM, CODES.MSG_TXT,
ERRORS.SERV_ERR_CNT, ERRORS.ERR_TS_NUM
ORDER BY ERRORS.DMN_NAM, CODES.MSG_TXT
I currently have two objects defined (ErrorsEntity and EventCodeEntity) which map to the tables SERV_ERR and EVENT_CD.
So the results of this query will be a list, but not of ErrorsEntity or EventCodeEntity but rather an amalgamation of the two entities.
Up to now, my queries have all returned objects that map directly to one table like so:
public List<ErrorsEntity> getErrors(double daysPrevious, double hoursToShow);
What's the best way to handle this in Hibernate where the results of a query aren't objects that are mapped to a single table and how can I write this query in HQL?
It's better to stick to an SQL query then, since HQL makes sense only when you plan on changing states from the resulted entities. In your case, the SQL is a better alternative, since it doesn't really follow the standard and you only want a projection anyway. You could remove the group by with distinct but it will require a derived table, which can be done in plain SQL anyway.
List dtos = s.createSQLQuery(
"SELECT " +
" ERRORS.DMN_NAM AS dmnNam, " +
" CODES.MSG_TXT AS msgTxt " +
"FROM SERV_ERR ERRORS, EVENT_CD CODES " +
"WHERE ERRORS.SERV_RESP_CD_TXT = CODES.CD_TXT " +
"GROUP BY " +
" ERRORS.DMN_NAM, " +
" ERRORS.SERV_NAM, " +
" CODES.MSG_TXT, " +
" ERRORS.SERV_ERR_CNT, " +
" ERRORS.ERR_TS_NUM " +
"ORDER BY " +
" ERRORS.DMN_NAM, " +
" CODES.MSG_TXT "
.addScalar("dmnNam")
.addScalar("msgTxt")
.setResultTransformer( Transformers.aliasToBean(MyDTO.class))
.list();
Make sure YourDTO has a matching constructor, and the types are the exactly like ee.dmn.nam and ece msgTxt.
Instead of group by I'd choose:
SELECT dmnNam, msgTxt
FROM (
SELECT DISTINCT
ERRORS.DMN_NAM AS dmnNam,
ERRORS.SERV_NAM,
CODES.MSG_TXT AS msgTxt,
ERRORS.SERV_ERR_CNT,
ERRORS.ERR_TS_NUM
FROM SERV_ERR ERRORS, EVENT_CD CODES
WHERE ERRORS.SERV_RESP_CD_TXT = CODES.CD_TXT
ORDER BY
dmnNam,
msgTxt
) as DATA

Generating queries by string concatenation in java -- variable values are not substituted

I have the following code as part of a method which reads from CSV file and store its contents into DB by generating insert statements and executing them.
....
try
{
while( (strLine = br.readLine()) != null)
{
query = baseQuery;
st = new StringTokenizer(strLine, ",");
while(st.hasMoreTokens())
{
token = st.nextToken();
if("TIMESTAMP".equals(values.get(tokenNumber)))
query += "'" + GeneralMethods.dateFormat(token) + "' , ";
else
query += "'" + token + "' , ";
}
query = query.substring(0, query.length()-2) + ")";
ds.insertData(query );
}
} catch (IOException e)
{
logger.error("IOException Occured while trying to read lines of CSV file: \n " + e.getMessage());
status = false;
}
Everything works fine, except that the query is generated with empty values. This is printed into the logs:
Failed to execute query: INSERT INTO c1_ds1 ( TIME , USER ) VALUES ('' , '' )
I believe the problem is in these two lines:
if("TIMESTAMP".equals(values.get(tokenNumber)))
query += "'" + GeneralMethods.dateFormat(token) + "' , ";
else
query += "'" + token + "' , ";
I have printed the (token) variable in the logs, and it's getting the values from the CSV file as it should do.
Anyone knows where is the problem?
I tried this as well:
query += "'"; query += token;
Look into PreparedStatement instead of concatenating your query via Strings. This will let the DB driver handle escaping and converting values for you.
There are exactly two possibilities:
either token is empty; or
GeneralMethods.dateFormat(token) is getting called and returns an empty string.
To find out which it is, either print out more diagnostics, or use a debugger.
P.S. It is a very insecure practice to build SQL queries bit by bit, since you open yourself up for SQL injection attacks and other problems. I would strongly encourage you to rewrite your code to use PreparedStatement.

Categories