I'm trying to use jOOQ for batch inserts into my postgres database.
What I'm trying to do is:
BatchBindStep bbs = context.batch(context.insertInto(TABLENAME,FIELD1,FIELD2,....).values("?","?",...));
bbs = bbs.bind(v1a,v2a).bind(v1b,v2b)....;
bbs.execute();
as described at http://www.jooq.org/doc/3.1/manual-single-page/#batch-execution
To make it clear, I want to insert thousands of rows in one query, not by using a batch with thousands of queries :
// 2. a single query
// -----------------
create.batch(create.insertInto(AUTHOR, ID, NAME).values("?", "?"))
.bind(1, "Erich Gamma")
.bind(2, "Richard Helm")
.bind(3, "Ralph Johnson")
.bind(4, "John Vlissides")
.execute();
The problem is:
To get to the point where the BatchBindStep accepts a .bind() call, one need to have called
context.batch with an argument, that has .values(...) as the last call.
In the documentation is stated, that "?" has to be used. This is typed as String, and may work only for tables where alls columns are varchars, since jOOQ does static typing.
This irritates me. I tried my luck with arbitrary default values (null,0...) just to go through the values(...) step, hoping that since these "values" are not really
values that I want to batch insert, they get overwritten later by the binds.
As a matter of fact, they will.
TWICE for the first row. Which completly baffles me.
To repeat, I CAN do batch inserts, but the first row gets inserted TWICE. I have the intuition that It has to do with the "values" call (at least there is a conceptual problem in the DSL with the typing).
Has anyone tried to use jOOQ for batch inserts, and how does one that without inserting the first row twice ?
P.S. This happens when I try to use
.values("?", "?", "?", "?", "?", "?", "?", "?", "?","?","?","?","?","?","?","?","?","?","?","?","?","?")
:
"The method values(Integer, String, String, String, String, String, String, String, String, String, Double, Double, String, String, String, String, Timestamp, String, String, String,
String, String) in the type
InsertValuesStep22 is not applicable for the arguments (String, String, String, String, String, String, String, String, String, String, String, String, String, String,
String, String, String, String, String, String, String, String)"
So clearly, the typing is wrong, when I try to adapt the example from the documentation.
The example from the documentation was wrong. It has now been fixed:
http://www.jooq.org/doc/latest/manual/sql-execution/batch-execution
In principle, as you've noticed, it doesn't matter what dummy bind values you're passing to the insert statement, as those values will be replaced when binding the values specified by the various .bind() calls. So in principle, some correct solutions would be:
// Passing in null
create.insertInto(AUTHOR, ID, NAME).values((Integer) null, null);
// Passing in a dummy value (even with a wrong type)
create.insertInto(AUTHOR, ID, NAME).values(Arrays.asList("?", "?"))
jOOQ integration tests suggest that batch insertion works correctly. The issue you have been experiencing with double-inserts of the first record would be surprising. Either this is a subtle bug that is not visible from your current question, or you might have called .bind() one too many times?
Related
Having searched unsuccessfully for days, I thought I would ask.
I am retrieving a value from a table, whose columns are all varchar. How do we go about identifying the type of value. For e.g.
accountnumber may have values such as 123456, 123456:78910
...
try(ResultSet rs = stmt.executeQuery("select * from table")){
if(rs.next()){
//Cannot use anything other than 'getString'
String var = rs.getString("accountnumber");
//Determine if var is an Integer
System.out.println(var instanceof Integer) //returns false because 'var' is of type String.
}
}
...
Any pointers are much appreciated.
The thing works as intended. The columns are typed as varchar in the schema, so the database and the Java SQL interface can't know if you actually have numbers or names or dates or whatever stored inside. Only getString should work.
It's up to you to try to parse the string results in ways that make sense.
For example, the Integer.parseInt method would transform a string to an integer, or throw an exception if it cannot. There's equivalent versions for Long or Double etc.
However, looking at the example accountnumbers, they aren't really integers, but rather have a more complex structure. You might want to model that with a class with two fields (primaryAccountNumber, theOtherAccountNumber) etc, and it should take care of parsing.
On a different note, you should never select * from a database, but rather be explicit about the fields that you need. This protects against schema changes which remove fields you were dependent on (the select will fail, instead of some code later down the line), or pulling in too much data and just using a bit of it.
ResultSet#getMetaData(); will give you the required informations.
ResultMetaData has some methods that will help eg. getColumnType(int) or getColumnClassName(int)
You can use pattern matching to check its a number or a string.
String var = rs.getString("accountnumber");
Matcher ss1 = Pattern.compile("([0-9]*)").matcher(var);
if (ss1.matches()) {
// its a number
} else {
// its a string
}
What could be the reason that some strings are not inserted into PostgreSQL table from Java?
This happens after an update of source of data from API v1 to API v2. The inserted data structure is almost the same, float values are inserted, but some strings are not. And without any error: the fields are just empty.
It is probably some escape character but I'm not able to figure out which one. And how to fix it as well. The string length is about 6k characters and its PostgreSQL representation is text.
Here is the stored procedure:
CREATE OR REPLACE FUNCTION add_data_string(in_ts bigint, in_ids integer[], in_string_values text[])
RETURNS integer AS
$BODY$
DECLARE
tmp_id integer;
tmp_index integer;
BEGIN
tmp_index := 0;
FOREACH tmp_id IN ARRAY in_ids LOOP
tmp_index := tmp_index +1;
INSERT INTO data (ts, id, string_value) VALUES (in_ts, tmp_id, in_string_values[tmp_index]);
END LOOP;
RETURN tmp_index;
END;
The strings are passed into the procedure like this:
cs.setArray(3, con.createArrayOf("varchar", (String[]) values));
There is one more thing: if the stored procedure is modified, for the sake of debug, that only first five characters of the string are inserted, e.g. like this:
INSERT INTO data (ts, id, string_value) VALUES (in_ts, tmp_id, substring(0, 5, in_string_values[tmp_index]));
the first five characters are inserted as expected.
Check the in_ids and in_string_values arrays before you pass them to PostgreSQL. My guess is that their sizes do not match (the string_values array probably has a few extra empty string values you do not expect) like this
ids string_values
1 "string that makes it to the database"
2 "" //unexpected empty string
"string which does not get inserted in the database"
The problem is in pgAdmin III which for some strange reason does show empty fields for strings longer than 3640 characters even though the value Max. characters per column is set to 4096.
My database is Postgres 8. I need to cast data type to another. That means, one of columns data type is varchar and need to cast it into int with Postgres in a SELECT statement.
Currently, I get the string value and cast it into int in Java.
Is there any way to do it? Sample code would be highly appreciated.
cast(varchar_col AS int) -- SQL standard
or
varchar_col::int -- Postgres syntax shorthand
Theses syntax variants are valid (almost) anywhere. The second may require nesting parentheses in special situations:
PostgreSQL: Create index on length of all table fields
And the first may be required where only functional notation is allowed by syntax restrictions:
PostgreSQL - CAST vs :: operator on LATERAL table function
There are two more variants:
int4(varchar_col) -- only works for some type names
int '123' -- must be an untyped, quoted string literal
Note how I wrote int4(varchar_col). That's the internal type name and there is also a function defined for it. Wouldn't work as integer() or int().
Note also that the last form does not work for array types. int[] '{1,2,3}' has to be '{1,2,3}'::int[] or cast('{1,2,3}' AS int[]).
Details in the manual here and here.
To be valid for integer, the string must be comprised of an optional leading sign (+/-) followed by digits only. Leading / trailing white space is ignored.
I have a method insert() which inserts a list of values into a table which is chosen by the user.
The problem is that since the user gets to choose the table, the method does not know how many values that are to be inserted and of which type they are. I've solved the variable amount of values with a loop that uses a stringbuilder to insert the correct amount of "?"-chars into the values part of the query.
I also have a loop that splits the values received into a String array, but I then have a problem with ints being processed like strings. Can I get around this using some trick with sql-syntax, or do I need to fetch information about which kind of data type each value is?
And if I have to fetch info about the data types, how do I do that? (Preferably an sql query that returns nothing but the types since I want to use the result directly in my java code).
Firstly, what I suspect you are doing is wrong, or certainly sub-optimal.
Assuming you were adamant this is how you want to go about it, you need to retrieve a row from your table and then call getMetadata() on the ResultSet.
You would end up with something like:
rs.getMetaData().getColumnTypeName(int column)
Once you know the column type, you can parse/sanitize your user entered data accordingly.
I need to parse the value of a database column that generally contains integers, based on the result Set generated from a JDBC call. However, one particular row of the column has two integers in it (ie, "48, 103"). What will be the return value of resultSet.getInt() on that column?
It will throw an exception.
I think you are taking the wrong approach here. The getXXX() is supposed to match the data type of the table. Is the data type on the table listed as VARCHAR? If that case you should use getString() to get the data and then parse it with the String.spilt(",") if the , exists (you can use String.indexOf() to verify is the comma is there or not).
You'll almost certainly get a SQLException (or possibly a NumberFormatException). The actual interface just says that the result set will return "the value of the designated column... as an int". The exact details will be implementation-specific, but I doubt you'll get anything sensible from a value of "48, 103".
(Personally I think it's an error if the driver lets you call getInt on that column in any case, even for "sensible" values. A string is not an int, even if it's a string representation of an int, and the conversion should be done manually by the developer.)
I'd expect it to throw an exception. If it does give you a value, it won't be what you want. I'd get the values as strings and parse them, splitting on commas and trimming spaces.
I believe it's a NumberFormatException.