I need to insert multiple rows (about 1 mln.) containing random numbers into a Postgresql database. This code generates one row with random numbers into a database. How can i make the statement loop itself for any amount of times?
Random rand = new Random();
for (int j=0;j < 1;j++);
stmt = c.createStatement();
String sql = "INSERT INTO COMPANY (ID,NAME,AGE,ADDRESS,SALARY) "
+ "VALUES ('" + rand.nextInt() + "', '" + rand.nextInt() + "', '" + rand.nextInt() + "', '" + rand.nextInt() + "'," +
" '" + rand.nextInt() + "')";
stmt.executeUpdate(sql);
stmt.close();
c.commit();
c.close();
You basically have two options to do that
Have the database do all the work
As suggested by #a_horse_with_no_name: use just one INSERT, and let the database compute all the random values:
INSERT INTO COMPANY
(ID
,NAME
,AGE
,ADDRESS
,SALARY)
SELECT
i
,random() * 10000 + 1
,random() * 80 + 1
,random() * 10000 + 1
,random() * 1000000 + 1
FROM
generate_series(1,1000000) i
This would be the usual way of filling a table with random values (to simulate data, for instance) when working with PostgreSQL. Note one thing: the ID column, assuming it is a PRIMARY KEY (i.e.: UNIQUE and NOT NULL) should never be assigned a random value, that could be repeated.
Have all the values computed by your program, but generate just one statement:
If, for some reason, the randomness of PostgreSQL is not good enough for your application, or you want to control through your program how the (pseudo)random values are generated, you could take advantage of the fact that you can have several rows stated in VALUES.
That is, the following statement is valid:
INSERT INTO some_table(a_column) VALUES (101), (102), (103), (104) ;
and would insert four rows into some_table.
You would change your program to generate values this way:
Random rand = new Random();
StringBuilder sql =
new StringBuilder("INSERT INTO COMPANY (ID,NAME,AGE,ADDRESS,SALARY) VALUES ;");
for (int j=0; j < 1000000; j++)
{
if (j) sql.append (",")
sql.append ("(" + j.toString() + /* ID should NOT be random() */
",'" + rand.nextInt().toString() + "'" + /* NAME */
",'" + rand.nextInt().toString() + "'" + /* AGE */
",'" + rand.nextInt().toString() + "'" + /* ADDRESS */
",'" + rand.nextInt().toString() + "'" + /* SALARY */
")") ;
}
stmt = c.createStatement();
stmt.executeUpdate(sql.toString());
stmt.close();
c.commit();
c.close();
NOTE 1: the SQL statement generated this way is not "dangerous" because you are completely controlling the data used to generate it. If you would use user input, or some information whose origin or format cannot be trusted, use PREPARED STATEMENTS, to avoid risks of SQL injection.
NOTE 2: Use a StringBuilder (not a String) to generate such a large String.
NOTE 3: As the SQL string can be too large to be handled by either the JDBC or the database itself (as pointed out by #dsp_user), it might be necessary to limit the number of iterations within the loop; and have a second loop over it (obviously, the use of the j variable should change in this scenario).
Related
(Edit: This is a question that I had and I answer it in the hopes of helping someone else who had a similar question.)
I am trying to clean up geographic data in my Google Fusion Table and would like to write a Java program to read in select Fusion Table rows, modify columns in each row and write out the modified rows back to the original fusion table.
I have found Christian Junk's example code in the Google API Client Libraries documentation: "fusiontables-cmdline-sample" that shows how to: Authorize access to a users fusion tables, list tables, create a table, insert data into a table, show rows, delete a table.
How do I modify this example to make updates to selected rows in a table? (see answer with code below)
[edit]: I didn't find any good solutions on the Net. I have written the solution in Java and will answer in the answers in the hope that it can help someone else how is trying to do this. I am a novice Java programmer so the code reflects that. I also needed to get nearby big cities based on a gps location and used GeoNames api (citiesJSON) creating a bounding box to do that. This solution uses JSON to access items returned from REST calls.
I have written a Java program that does the row data modification described in the question. It uses Christian Junk's example noted in the question and also calls GeoNames citiesJSON webservice as described in the question (sending bounding box coordinates in the parameters). I'm a novice in Java so the code is what it is. I do a lot of commenting in order to reuse code, like SQL queries, later.
You can find my solution on Github at: FusionTableModifyJava
The primary module of interest is: FusionTableSample.java
Here are the functions that do the getRows and updateRows. Everything else can be seen at github (by Microsoft$):
private static void getRows(String tableId) throws IOException {
View.header("Updating Rows From Table");
/*Sql sql = fusiontables.query().sql("SELECT RowID, 'Area Name', Notes, Number FROM " + tableId +
" Where Manager = '' AND 'Review 1' CONTAINS IGNORING CASE '.fs.' Order by Number ASC LIMIT 3000");*/
/*Sql sql = fusiontables.query().sql("SELECT RowID, 'Area Name', Notes, Number FROM " + tableId +
" Where 'Area Name' CONTAINS IGNORING CASE 'Tioga George' Order by Number ASC LIMIT 3000");*/
/*Sql sql = fusiontables.query().sql("SELECT RowID, 'Area Name', Notes, Number FROM " + tableId +
" Where 'Area Name' ='' Order by Number DESC LIMIT 2000");*/
/*Sql sql = fusiontables.query().sql("SELECT RowID, 'Area Name', Notes, Number FROM " + tableId +
" Where 'Area Name' CONTAINS 'X01' Order by Number DESC LIMIT 1"); */
/*AND 'City (nearest)' DOES NOT CONTAIN IGNORING CASE 'Mexico'*/
/*Sql sql = fusiontables.query().sql("SELECT RowID, 'Area Name', Notes, Number, Location FROM " + tableId +
" Where State = '' Order by Number DESC LIMIT 100");*/
/*Sql sql = fusiontables.query().sql("SELECT RowID, 'Area Name', Notes, Number, Location FROM " + tableId +
" Where State = 'BCS' Order by Number DESC LIMIT 100");*/
Sql sql = fusiontables.query().sql("SELECT RowID, 'Area Name', Notes, Number, Location, State, Codes FROM " + tableId +
" Where State = 'ID' AND 'City (nearest)' = '' Order by Number DESC LIMIT 100");
try {
Sqlresponse response = sql.execute();
// System.out.println(response.toPrettyString());
mylist = response.getRows();
} catch (IllegalArgumentException e) {
// For google-api-services-fusiontables-v1-rev1-1.7.2-beta this exception will always
// been thrown.
// Please see issue 545: JSON response could not be deserialized to Sqlresponse.class
// http://code.google.com/p/google-api-java-client/issues/detail?id=545
}
}
private static void updateRows(String tableId) throws IOException {
// IOException needed ParseException
count = 1;
mylist.forEach((myRow) -> {
try {
// modify fields in table...
//newAreaName = kt.firstpart(myRow.get(NOTES).toString()); //get Notes first sentence
//newAreaName = newAreaName.replace("'", "''");
//newAreaName += " X01";
//String state = getStateFrmLoc(myRow.get(LOCATION).toString());
//String state = "MX-BCS";
float km;
if ( "AK,MT,NV".contains(myRow.get(STATE).toString()) ) {
km = 180f; // 111.85 miles
} else {
km = 80.5f; // 50 miles
}
BigCity big = new BigCity(myRow.get(LOCATION).toString(), km);
String cityState = big.cityName +", "+big.state;
if (big.population < 10000f) {
System.out.println("Skip for low population :"+myRow.get(NUMBER));
} else {
sqlupdate = "UPDATE " + tableId + " " +
"SET 'City (nearest)' = '" + cityState + "' " +
",'Codes' = '" + myRow.get(CODES).toString() + ",#U1' " +
"WHERE ROWID = " + myRow.get(ROW_ID);
System.out.println("[" + count + "]" + myRow.get(NUMBER) + ": " + sqlupdate);
// do the update...
if (!mtest) { // if testing then don't update
sql_doupdate(sqlupdate);
}
count++;
if ((count % 30) == 0) {
System.out.println("waiting 60 seconds");
TimeUnit.SECONDS.sleep(60); //Fusion Tables allows 30 updates then must wait 1 minute.
}
}
} catch(Exception e){
System.out.println(e.getMessage());
}
});
}
I am trying to populate one table in my database with pretty complex data. For this, I am using a generator API (which gives me random data).
public void populateCrackers(){
PreparedStatement psm;
String queryJoke = "(SELECT jid FROM Jokes WHERE jid=?)";
String queryHat = "(SELECT hid FROM Hats WHERE hid=?)";
String queryGift = "(SELECT gid FROM Gifts WHERE gid=?)";
String query = "INSERT INTO Crackers(cid, name, jid, hid, gid, quantity) VALUES(" +
"?, " +
"?, " +
queryJoke + ", " +
queryHat + ", " +
queryGift + ", " +
"?)";
System.out.println(query);
String cracker_String = utils.JSONUtils.getJSON(crackerAPI, client);
JSONObject crackerJSON = new JSONObject(cracker_String);
JSONArray crackers = crackerJSON.getJSONArray("results");
for(int j=0; j<crackers.length(); j++){
try{
psm = connection.prepareStatement(query);
psm.setInt(1,crackers.getJSONObject(j).getInt("cid"));
psm.setString(2, crackers.getJSONObject(j).getString("cname"));
psm.setInt(3, crackers.getJSONObject(j).getInt("rjoke"));
psm.setInt(4, crackers.getJSONObject(j).getInt("rhat"));
psm.setInt(5, crackers.getJSONObject(j).getInt("rgift"));
psm.setInt(6, crackers.getJSONObject(j).getInt("cquantity"));
psm.execute();
System.out.println(crackers.getJSONObject(j).get("cid") + " "
+ crackers.getJSONObject(j).get("cname") + " "
+ crackers.getJSONObject(j).get("cquantity") + " "
+ crackers.getJSONObject(j).get("rjoke") + " "
+ crackers.getJSONObject(j).get("rhat") + " "
+ crackers.getJSONObject(j).get("rgift"));
}catch (Exception e){
e.printStackTrace();
}
}
}
This is the method that populates my "Crackers" tab. I am wondering if this be accepted as a prepared statement. When I run it in psql interactive command line tool, exactly that statement with some chosen ids (e.g INSERT INTO Crackers (cid, name, hid, jid, gid, quantity) VALUES('cid', 'name', (SELECT hid FROM Hats WHERE hid=11), (SELECT jid FROM Jokes where jid=99), (SELECT gid FROM Gifts WHERE gid=13), 5) it works flawlessly.
Does my preparedstatement break the Constraint?
Any ideas?
LATER EDIT: The inconsistency is the form of that null values can reach my Crackers table (e.g. Cracker(1, "hello", null, null, 3, 123) appears in the table.
There is nothing about Prepared statement. Constraint can be broken by parameters you set to it. And you can run your PLSQL statement as anonimous block in PreparedStatement as well.
Just surround it with BEGIN ... END. only one thing is different - for JDBC parameters are ? mark not :parameter as for PLSQL and there is no way to use named parameter.
That means if you need to use parameter more than once for JDBC you have to have that many ? marks and set all of them.
So, focus on parameters you pass to and their sequence.
The code is correct, though the prepared statement must be closed, and it would be better to create the statement once, before the for loop.
Now there is crackers.length() times a statement created but not closed. That might give problems.
Use the try-with-resouce syntax for automatic closing, irrespective of any exception or return.
try (PreparedStatement psm = connection.prepareStatement(query)) {
for (int j = 0; j < crackers.length(); j++) {
...
psm.executeUpdate();
And call executeUpdate instead of the more general execute. The resulting update count might be of interest (1/0).
I realised I had the wrong constraints on my table. I was letting null values in. There was nothing wrong with the prepared statement.
The right query to create the table is this one:
String createCrackersQuery = "CREATE TABLE Crackers(" +
" cid INTEGER," +
" name VARCHAR NOT NULL," +
" jid INTEGER NOT NULL," +
" hid INTEGER NOT NULL," +
" gid INTEGER NOT NULL," +
" quantity INTEGER NOT NULL," +
" CONSTRAINT Cracker_Primary PRIMARY KEY (cid)," +
" CONSTRAINT Cracker_Name_Unique UNIQUE(name)," +
" CONSTRAINT Joke_Foreign FOREIGN KEY (jid) REFERENCES Jokes(jid)," +
" CONSTRAINT Hat_Foreign FOREIGN KEY (hid) REFERENCES Hats(hid), " +
" CONSTRAINT Gift_Foreign FOREIGN KEY (gid) REFERENCES Gifts(gid)" +
")";
"MERGE INTO NT_PROPERTY ntProp USING ( " +
"SELECT * FROM NT_PROPERTY ) " +
"VALUES " +
"('minDPTObjectId'," + minDPTObjectId + ", 'Starting DPT Object Id') " +
"('maxDPTObjectId', " + maxDPTObjectId + ", 'Ending DPT Object Id') " +
"vt (NAME, VALUE, NOTE) " +
"ON ( ntProp.NAME = vt.NAME ) " +
"WHEN MATCHED THEN " +
"UPDATE SET VALUE = vt.VALUE "+
"WHEN NOT MATCHED THEN " +
"INSERT (NAME, VALUE, NOTE) VALUES (vt.NAME, vt.VALUE, vt.NOTE)";
Well I'm getting a missing ON keyword error and with no clue what so ever, also is there any other way to make it less clumsy
Help is very much appreciated.
The problem is that your MERGE syntax is incorrect. Your statement takes the form of:
MERGE INTO nt_property ntprop
USING (SELECT * FROM nt_property)
VALUES (...)
vt (...)
ON (ntprop.name = vt.name)
WHEN MATCHED THEN
UPDATE ...
WHEN NOT MATCHED THEN
INSERT ...;
but it should be of the form:
MERGE INTO target_table tgt_alias
USING source_table_or_subquery src_alias
ON (<JOIN conditions>)
WHEN MATCHED THEN
UPDATE ...
WHEN NOT MATCHED THEN
INSERT ...;
Why do you have the VALUES and vt clauses between your using and your on clauses? That's the incorrect syntax. Also, whilst you can use select * from tablename in the using clause, you could just use the tablename directly, since you're selecting all columns and all rows.
MERGE INTO NT_PROPERTY D
USING (SELECT * FROM DUAL ) S
ON (D.NAME = 'minDPTObjectId')
WHEN MATCHED THEN UPDATE SET D.VALUE = '1234'
WHEN NOT MATCHED THEN INSERT (NAME, VALUE, NOTE)
VALUES ('maxDPTObjectId', '1111', 'Ending DPT Object Id') ;
I have a table X which has a column BALANCE. I have two row ids and a value amount. There are two constraint variables - MAX_BALANCE and MIN_BALANCE.
I need to write an update query which updates the column BALANCE. The first row id's BALANCE is added with amount and the amount is subtracted from the second row id's BALANCE. I need to ensure that the BALANCE always stays within the range. That is, MIN_BALANCE <= BALANCE <= MAX_BALANCE.
I am not supposed to update one row and then roll back if the number of rows updated is not equal to 2. The update query should either update two rows (success case) or it should not update any row at all.
I am using Hibernate in Java and here is the query which I have tried. It doesn't work for the success case.
String sql = "UPDATE X x "
+ "SET x.balance = CASE "
+ "WHEN x.id = :rowId1 THEN (x.balance + :amount) "
+ "WHEN x.id = :rowId2 THEN (x.balance - :amount) "
+ "END "
+ "WHERE x.id IN :ids "
+ "AND ((x.id = :rowId1 AND x.balance + :amount <= :MAX_BALANCE) "
+ "OR (x.id = :rowId2 AND x.balance - :amount >= :MIN_BALANCE))";
Query query = entityManager.createQuery(sql);
List<BigInteger> ids = Arrays.asList(new BigInteger(rowId1), new BigInteger(rowId2));
int rows = query.setParameter("amount", amount)
.setParameter("ids", ids)
.setParameter("rowId1", new BigInteger(rowId1))
.setParameter("rowId2", new BigInteger(rowId2))
.setParameter("MAX_BALANCE", new Float(MAX_BALANCE))
.setParameter("MIN_BALANCE", new Float(MIN_BALANCE))
.executeUpdate();
I don't want to check if rows == 1 and throw an exception. The update query should always ensure that rows will take the value either 0 or 2.
Or is there a way to perform this operation based on Criteria Update in Hibernate?
You need to self join your table on itself. The first instance should have the record for rowid1 and the 2nd instance should have the record for rowid2. This way you can check the balances of both records in one go and make the decision whether to update or not.
update x x1 join x x2
set x1.balance=if(x1.balance+:amount <= :MAX_BALANCE,x1.balance+:amount, x1.balance),
x2.balance=if(x2.balance-:amount >= :MIN_BALANCE,x2.balance-:amount, x2.balance)
where x1.id=:rowid1 and x2.id=rowid2
try this ...
String sql = "UPDATE X x "
+ "SET x.balance = CASE "
+ "WHEN x.id = :rowId1 AND x.balance + :amount <= :MAX_BALANCE THEN (x.balance + :amount) "
+ "WHEN x.id = :rowId2 AND x.balance - :amount >= :MIN_BALANCE THEN (x.balance - :amount) "
+ "END "
+ "WHERE x.id = :rowId1 OR x.id = :rowId2";
public static void updateHistory(Client winner) {
try {
if (winner != null) {
getDatabase().newQuery("INSERT INTO tournaments (`players`, `winner`, `winnerKills`, `map`, `prize`) VALUES ('" + tournyPlayers.size() +"', '" + winner.playerName + "', '" + winner.killCount + "', 'Wilderness', '" + winner.gameScore + "')");
}
else {
getDatabase().newQuery("INSERT INTO tournaments (`players`, `winner`, `winnerKills`, `map`, `prize`) VALUES ('" + tournyPlayers.size() +"', 'None', 'None', 'Wilderness', 'None')");
}
ArrayList<Client> player = tournyPlayers;
for (int i = 0; i < player.size(); i++) {
if (player.get(i).killedByPlayer == "") {
getDatabase().newQuery("INSERT INTO tournament_players (`name`, `kills`, `killedBy`, `score`, `points`) VALUES ('" + player.get(i).playerName + "', '" + player.get(i).killCount + "', 'N/A', '" + player.get(i).gameScore + "', '" + player.get(i).gamePoints + "')");
}
else {
getDatabase().newQuery("INSERT INTO tournament_players (`name`, `kills`, `killedBy`, `score`, `points`) VALUES ('" + player.get(i).playerName + "', '" + player.get(i).killCount + "', '" + player.get(i).killedByPlayer + "', '" + player.get(i).gameScore + "', '" + player.get(i).gamePoints + "')");
}
}
} catch (SQLException e) {
e.printStackTrace();
}
}
Well what I am trying to do is, insert a new tournament row, which will display the information about the tournament, now I want to add the players to the tournament (Who was playing it), so I now will add each player, into a new row, with his data.
But now I want to make sure that the player matches the current tournament I inserted. I can't make a unique number of the current server, server-sided because the server may go down, and lose the count.
therefore I've thought of using the inserted tournaments table's auto_increment ID, and add it along with the inserted player, so his id will be tournament's auto increment id.
Is there a way to fetch the auto increment ID without using the exact same data I inserted into the tournaments (cause there may be 2 tournaments with the same status, who knows).
Is there a way to do so?
I was thinking of adding a MD5 hash with the date + time, to the tournaments row and then add it to the player's row, is that a good solution?
Take a look at Statement#getGeneratedKeys() which returns a ResultSet that you can use to retrieve any generated primary keys as
ResultSet rsKeys = statement.getGeneratedKeys();
if (rsKeys.next()) {
tournament.setId(rsKeys.getLong(1));
}
Please, note that you need to give a hint to the JDBC driver that you would like to retrieve the generated keys while preparing a PreparedStatement as
connection.prepareStatement(strSQL, Statement.RETURN_GENERATED_KEYS);
or, at the time of executing a Statement as
statement.executeUpdate(strSQL, Statement.RETURN_GENERATED_KEYS);
Reference:
Statement#getGeneratedKeys()
Retrieves any auto-generated keys created as a result of executing this Statement object. If this Statement object did not generate any keys, an empty ResultSet object is returned.