I have the following code as part of a method which reads from CSV file and store its contents into DB by generating insert statements and executing them.
....
try
{
while( (strLine = br.readLine()) != null)
{
query = baseQuery;
st = new StringTokenizer(strLine, ",");
while(st.hasMoreTokens())
{
token = st.nextToken();
if("TIMESTAMP".equals(values.get(tokenNumber)))
query += "'" + GeneralMethods.dateFormat(token) + "' , ";
else
query += "'" + token + "' , ";
}
query = query.substring(0, query.length()-2) + ")";
ds.insertData(query );
}
} catch (IOException e)
{
logger.error("IOException Occured while trying to read lines of CSV file: \n " + e.getMessage());
status = false;
}
Everything works fine, except that the query is generated with empty values. This is printed into the logs:
Failed to execute query: INSERT INTO c1_ds1 ( TIME , USER ) VALUES ('' , '' )
I believe the problem is in these two lines:
if("TIMESTAMP".equals(values.get(tokenNumber)))
query += "'" + GeneralMethods.dateFormat(token) + "' , ";
else
query += "'" + token + "' , ";
I have printed the (token) variable in the logs, and it's getting the values from the CSV file as it should do.
Anyone knows where is the problem?
I tried this as well:
query += "'"; query += token;
Look into PreparedStatement instead of concatenating your query via Strings. This will let the DB driver handle escaping and converting values for you.
There are exactly two possibilities:
either token is empty; or
GeneralMethods.dateFormat(token) is getting called and returns an empty string.
To find out which it is, either print out more diagnostics, or use a debugger.
P.S. It is a very insecure practice to build SQL queries bit by bit, since you open yourself up for SQL injection attacks and other problems. I would strongly encourage you to rewrite your code to use PreparedStatement.
Related
I have a problem. Right now I am using JOOQ to insert arround 100.000 records in my database using the following code:
try (Connection conn = DriverManager.getConnection(SqlConn.getURL(), SqlConn.getUSERNAME(), SqlConn.getPASSWORD())) {
DSLContext create = DSL.using(conn, SQLDialect.MYSQL);
for (String key : trendlines.keySet()) {
for (Trendline trendline : trendlines.get(key)) {
String sql = createTrendlineQuery(trendline);
create.fetch(sql);
}
}
}
catch (Exception e) {
e.printStackTrace();
}
With the function createTrendlineQuery():
private String createTrendlineQuery(Trendline trendline) {
return "INSERT INTO Trendline (openTime, market, coin, period, metric, number, slope, interceptY, percentage, formula, data) VALUES (" +
trendline.getOpenTime() + ", '" +
trendline.getMarket() + "', '" +
trendline.getCoin() + "', '" +
trendline.getPeriod() + "', '" +
trendline.getFormula() + "') " +
"ON DUPLICATE KEY UPDATE " +
"openTime = " + trendline.getOpenTime() + ", " +
"market = '" + trendline.getMarket()+ "', " +
"coin = '" + trendline.getCoin() + "', " +
"period = '" + trendline.getPeriod() + "', " +
"formula = '" + trendline.getFormula() + "';";
}
But this gives a lot of load on my internet/database, so I found out you can do batch inserts for big data. I found the following page of JOOQ about batch inserts: https://www.jooq.org/doc/3.14/manual/sql-execution/batch-execution/. Now I think this is what I need, but I have a problem. The example looks in my case like this:
try (Connection conn = DriverManager.getConnection(SqlConn.getURL(), SqlConn.getUSERNAME(), SqlConn.getPASSWORD())) {
DSLContext create = DSL.using(conn, SQLDialect.MYSQL);
create.batch(create.insertInto(DSL.table("Trendline"), DSL.field("openTime"), DSL.field("market"), DSL.field("coin") ).values((Integer) null, null, null))
.bind( trendline.getOpenTime() , trendline.getMarket() , trendline.getCoin() )
.bind( trendline.getOpenTime() , trendline.getMarket() , trendline.getCoin() )
.bind( trendline.getOpenTime() , trendline.getMarket() , trendline.getCoin() )
.bind( trendline.getOpenTime() , trendline.getMarket() , trendline.getCoin() )
.execute();
}
catch (Exception e) {
e.printStackTrace();
}
Except that I need to put the 2 for-loops between the create.batch() to programmatically create the inserts. How can I insert the for loops and am I using the batch insert the right way to reduce internet traffic and database pressure?
Using BatchedConnection as a quick fix
The simplest solution to turning existing jOOQ code (or any JDBC based code, for that matter) into a batched JDBC interaction is to use jOOQ's BatchedConnection:
create.batched((Connection c) -> {
// Now work with this Connection c, instead of your own Connection and all the statements
// will be buffered and batched, e.g.
DSL.using(c).insertInto(...).values(...).execute();
});
Using the batch API that you've tried using
You just have to assign the BatchBindStep to a local variable in your loop and you're set:
BatchBindStep step = create.batch(query);
for (...)
step = step.bind(...);
step.execute();
Using the import API
Use the import API. Assuming you're using the code generator and you have the usual static imports
import static org.jooq.impl.DSL.*;
import static com.example.generated.Tables.*;
Write this:
create.loadInto(TRENDLINE)
.onDuplicateKeyUpdate()
.loadArrays(trendlines
.values()
.stream()
.map(t -> new Object[] {
t.getOpenTime(),
t.getMarket(),
t.getCoin(),
t.getPeriod(),
t.getFormula()
/* And the other fields which you partially omitted */
})
.toArray(Object[][]::new)
)
.fields(
TRENDLINE.OPENTIME,
TRENDLINE.MARKET,
TRENDLINE.COIN,
TRENDLINE.PERIOD,
TRENDLINE.FORMULA
/* And the other fields which you partially omitted */
)
.execute();
See also the sections about:
Throttling (you may want to play with these values to find what's optimal for your system)
Error handling
Which may be of interest. If the input Object[][] gets too large, you can chunk your input trendlines.values() collection manually on your side. If sorting your map by key is really essential (it shouldn't be from what I can tell from your question), then write this instead:
trendlines
.keySet()
.stream()
.flatMap(k -> trendlines.get(k).stream())
.map(t -> new Object[] { ... })
...
A few remarks on your own attempts
You're calling create.fetch(sql), when in fact your statement is a query with an update count, so in that case, you would have wanted to use create.execute(sql) instead.
Please never concatenate SQL strings when using jOOQ! Even when using plain SQL templating, there is never a need for concatenating SQL strings. You'll run into syntax errors and SQL injection. Please always use bind variables.
I really recommend you use jOOQ's code generator. Most benefits of using jOOQ arise when you use the code generator. Valid reasons to avoid code generation include when your schema is dynamic and not known at runtime. That's almost the only reason not to use code generation.
I have a list of objects provided by another service which I use to update my own data. When I try to use NamedParameterJdbcTemplate.batchUpdate, all returned values are zero.
public void updateWeather(List<Weather> weatherList) {
String query = "UPDATE weather \n" +
"SET rain_probability = ROUND(:rainProbability, 4), \n" +
"wind_speed = :windSpeed \n" +
"WHERE city_id = :cityId AND date = :date;";
List<MapSqlParameterSource> batchList = new ArrayList<>();
for(Weather weather : weatherList) {
MapSqlParameterSource params = new MapSqlParameterSource();
params.addValue("rainProbability", weather.getRainProbability());
params.addValue("windSpeed", weather.getWindSpeed());
params.addValue("cityId", weather.getCityId());
params.addValue("date", weather.getDate());
batchList.add(params);
}
this.namedParameterJdbcParameter
.batchUpdate(query, batchList.toArray(new MapSqlParameterSource[] {});
}
If I run this UPDATE directly in the database, it works fine. Futhermore, if I run it one by one, that is, replacing values (instead of adding the parameter source to batchList) it also works.
For example:
for (Weather weather : weatherList) {
String query = String.format("UPDATE weather \n" +
"SET rain_probability = ROUND('%d', 4), \n" +
" wind_speed = %d \n" +
" WHERE city_id = :cityId AND date = :date;",
weather.getRainProbability(),
weather.getWindSpeed(),
weather.getCityId(),
weather.getDate()
);
this.namedParameterJdbcTemplate.update(query, Collections.emptyMap());
}
Any suggestions of what I'm doing wrong?
Is it the use of "\n" or the ";" at the end of the statement within the String? (I'm surprised you don't get a SQL Syntax exception with the ; inside the actual query string)
Also dates are always a bit tricky and if that isn't converting properly then your WHERE clause isn't going to match and is possibly why 0 rows are returned. Could you temporarily try converting dates to Strings and see if the count is correct (e.g. for Oracle: AND date = TO_DATE(:dateStr, 'DD/MM/YYYY') )
I have 2 datebox to make a filter. Their value will determine the 'from' and 'to' in my query (I'm using Oracle now), here is the code.
#Listen("onClick=#btnSaveFilter")
public void saveFilter() throws Exception {
SimpleDateFormat formatter = new SimpleDateFormat("yyyy-MM-dd");
c_main_filter.detach();
cc.refreshFilter(""
+"[Date of Birth] between '" + formatter.format(dbDateBirthFrom.getValue()).toString() + "' "
+ "and '" + formatter.format(dbDateBirthto.getValue()).toString() + "'");
}
when both datebox has value, the query work. but when they have no value the query is giving no data.
when datebox has value, it's giving data
"[Registration Date] between '2010-09-23' and '2010-09-23' "
when datebox has no value, it's giving no data
"[Registration Date] between '' and '' "
like another filter I expect if the value is '' then all data will appear, but not :D hahaha. the condition is more than this actually, the filter has a lot of parameter one of them is this condition, and some of them use date format so there will be more condition like this.
do you know how to elegantly fix this problem, I've been thinking to use 'if' to determine the datebox has value or no then I will append the text to query text if both of them has value, but then I found another problem how I can add 'and' in query to give another condition,
let say I have 5 conditions so then
"select * from xxxx where 1stcondition and 2ndcondition and
3rdcondition and 4thcondition and 5th condition"
so when the dateboxes of the 5thcondition has no value the query will be wrong like this
"select * from xxxx where 1stcondition and 2ndcondition and
3rdcondition and 4thcondition and"
if I want to use 'if' how can I play with the 'and'? but if you have alternative it will be great cause I don't have to deal with 'if' :D
You can use String.isEmpty() to determine whether you need to put an and:
String where = "";
if (from != null && to != null) {
where += <yourDateCondition>;
}
if (<needToAddSecondCondtion>) {
if (!where.isEmpty()) {
where += " and ";
}
where += <secondCondition>;
}
// continue with other conditions
String query = "select * from xxxx where " + where;
I don't know if you use plain JDBC or ORM framework like hibernate to querying to the database, but you can try something like this :
public void saveFileter(){
StringBuilder sb = new StringBuilder();
sb.append("select * from table ");
if(dbDateBirthFrom.getValue() != null{
sb.append(determineFilterWord(sb.toString()));
sb.append("your condition ");
}
if(dbDateBirthTo.getValue() != null{
sb.append(determineFilterWord(sb.toString()));
sb.append("your condition ")
}
session.createQuery(sb.toString()); //if you using hibernate
}
private string determineFilterWord(String query){
if(query.toLowerCase().indexOf("where") != -1){
return "and ";
}else{
return "where ";
}
}
String sqlInsertBeacon = "INSERT INTO `beacon` (zone_id, location) VALUE ('(SELECT id FROM zone WHERE GeographicalID = '" + geometry3 + "')', Point(" + x_coordinate + "," + y_coordinate + "))";
System.out.println("The SQL query is: " + sqlInsertBeacon); // Echo for debugging
int countInserted3 = stmt.executeUpdate(sqlInsertBeacon);
System.out.println(countInserted3 + " records inserted.\n");
When I run the above code, the build is successful but the program stops when it reaches the execute line. I am entering using this sql query to insert data into a mysql database. I am not sure where the error is in my query? Can anyone suggest an alternative way or find the mistake?
The output of the program is this, as you can see the program, stops running after the second line:
The SQL query is: INSERT INTO table
(zone_id, location)
VALUES
((SELECT id FROM zone WHERE GeographicalID = '6311599'), Point(-121.9453802,37.3256131) )
;
BUILD SUCCESSFUL (total time: 6 seconds)
For additional information incase it helps:
The stmt, is created like this:
try (
// Step 1: Allocate a database 'Connection' object
Connection conn = DriverManager.getConnection(
"jdbc:mysql://localhost:3306/adhwere2?useSSL=false", "root", "your_new_password"); // MySQL
// Step 2: Allocate a 'Statement' object in the Connection
Statement stmt = conn.createStatement();) {
and the catch exception is :
} catch (SQLException ex) {
}
Try something like this:
String sqlInsertBeacon = "INSERT INTO `beacon` (zone_id, location)" +
" VALUES ( (SELECT id FROM zone WHERE GeographicalID = '" + geometry3 + "'), Point(" +
x_coordinate + "," + y_coordinate + "))";
Just removed the apostrophes aroung the inner SELECT and replaced VALUE with VALUES...
The problem was because the sub-query was returning more than one result, and printing out a stack trace helped debug this error. Using Limit 1 in the sub query also solved this issue.
please use query according to this syntax:
INSERT INTO table
(column1, column2, ... )
VALUES
(expression1, expression2, ... ),
(expression1, expression2, ... ),
...;
your table name is in single quotes and its VALUES not value mind these small things
Correct INSERT INTO SELECT statement looks like this:
INSERT INTO table2
SELECT * FROM table1
WHERE condition;
And you can use PreparedStatement to set parameters in your query.
I am new to using SQL2O with MySQL, but I am having a weird problem, where different queries return same results. Is SQL2O returning me cached results?
My code looks like this:
String sql = "SELECT * " +
"FROM report_A" +
"ORDER BY :order :sequence "+
"LIMIT :from, :limit";
int limit = 5;
int startIndex = (page-1)*limit;
String sequence = "DESC";
try(Connection con = sql2o.open()) {
if(order.contains("-")){
order = order.replace("-", "");
sequence= " ASC";
}
Query query= con.createQuery(sql)
.addParameter("from", startIndex)
.addParameter("limit", limit)
.addParameter("order", order)
.addParameter("sequence", sequence);
List<ReportA> result = query.executeAndFetch(ReportA.class);
con.close();
The 4 parameters always change, but the output remains the same. I have verified the queries in mysql workbench, the data is different, but SQL2O returns me the same set of data. Am I missing something?
Your query is invalid. It wont compile and throw an Sql2oException on execution.
The problem is, basically, that you can use parameters only for values, not for table names, column names or other keywords like "ASC". Changing those would change the structure of the query.
It's possible to construct queries with variable structure by good old string concatenation, i.e.
String sql = "SELECT * " +
"FROM report_A" +
"ORDER BY " + order " " + SEQUENCE +
"LIMIT :from, :limit";
and then
query(sql)
.addParameter("from", from)
.addParameter("limit", limit)
.executeAndFetch(...)