We Keep Coding

How to read and update rows of a Google Fusion Table using JAVA code

(Edit: This is a question that I had and I answer it in the hopes of helping someone else who had a similar question.)
I am trying to clean up geographic data in my Google Fusion Table and would like to write a Java program to read in select Fusion Table rows, modify columns in each row and write out the modified rows back to the original fusion table.
I have found Christian Junk's example code in the Google API Client Libraries documentation: "fusiontables-cmdline-sample" that shows how to: Authorize access to a users fusion tables, list tables, create a table, insert data into a table, show rows, delete a table.
How do I modify this example to make updates to selected rows in a table? (see answer with code below)
[edit]: I didn't find any good solutions on the Net. I have written the solution in Java and will answer in the answers in the hope that it can help someone else how is trying to do this. I am a novice Java programmer so the code reflects that. I also needed to get nearby big cities based on a gps location and used GeoNames api (citiesJSON) creating a bounding box to do that. This solution uses JSON to access items returned from REST calls.

I have written a Java program that does the row data modification described in the question. It uses Christian Junk's example noted in the question and also calls GeoNames citiesJSON webservice as described in the question (sending bounding box coordinates in the parameters). I'm a novice in Java so the code is what it is. I do a lot of commenting in order to reuse code, like SQL queries, later.
You can find my solution on Github at: FusionTableModifyJava
The primary module of interest is: FusionTableSample.java
Here are the functions that do the getRows and updateRows. Everything else can be seen at github (by Microsoft$):
private static void getRows(String tableId) throws IOException {
View.header("Updating Rows From Table");
/*Sql sql = fusiontables.query().sql("SELECT RowID, 'Area Name', Notes, Number FROM " + tableId +
" Where Manager = '' AND 'Review 1' CONTAINS IGNORING CASE '.fs.' Order by Number ASC LIMIT 3000");*/
/*Sql sql = fusiontables.query().sql("SELECT RowID, 'Area Name', Notes, Number FROM " + tableId +
" Where 'Area Name' CONTAINS IGNORING CASE 'Tioga George' Order by Number ASC LIMIT 3000");*/
/*Sql sql = fusiontables.query().sql("SELECT RowID, 'Area Name', Notes, Number FROM " + tableId +
" Where 'Area Name' ='' Order by Number DESC LIMIT 2000");*/
/*Sql sql = fusiontables.query().sql("SELECT RowID, 'Area Name', Notes, Number FROM " + tableId +
" Where 'Area Name' CONTAINS 'X01' Order by Number DESC LIMIT 1"); */
/*AND 'City (nearest)' DOES NOT CONTAIN IGNORING CASE 'Mexico'*/
/*Sql sql = fusiontables.query().sql("SELECT RowID, 'Area Name', Notes, Number, Location FROM " + tableId +
" Where State = '' Order by Number DESC LIMIT 100");*/
/*Sql sql = fusiontables.query().sql("SELECT RowID, 'Area Name', Notes, Number, Location FROM " + tableId +
" Where State = 'BCS' Order by Number DESC LIMIT 100");*/
Sql sql = fusiontables.query().sql("SELECT RowID, 'Area Name', Notes, Number, Location, State, Codes FROM " + tableId +
" Where State = 'ID' AND 'City (nearest)' = '' Order by Number DESC LIMIT 100");
try {
Sqlresponse response = sql.execute();
// System.out.println(response.toPrettyString());
mylist = response.getRows();
} catch (IllegalArgumentException e) {
// For google-api-services-fusiontables-v1-rev1-1.7.2-beta this exception will always
// been thrown.
// Please see issue 545: JSON response could not be deserialized to Sqlresponse.class
// http://code.google.com/p/google-api-java-client/issues/detail?id=545
}
}
private static void updateRows(String tableId) throws IOException {
// IOException needed ParseException
count = 1;
mylist.forEach((myRow) -> {
try {
// modify fields in table...
//newAreaName = kt.firstpart(myRow.get(NOTES).toString()); //get Notes first sentence
//newAreaName = newAreaName.replace("'", "''");
//newAreaName += " X01";
//String state = getStateFrmLoc(myRow.get(LOCATION).toString());
//String state = "MX-BCS";
float km;
if ( "AK,MT,NV".contains(myRow.get(STATE).toString()) ) {
km = 180f; // 111.85 miles
} else {
km = 80.5f; // 50 miles
}
BigCity big = new BigCity(myRow.get(LOCATION).toString(), km);
String cityState = big.cityName +", "+big.state;
if (big.population < 10000f) {
System.out.println("Skip for low population :"+myRow.get(NUMBER));
} else {
sqlupdate = "UPDATE " + tableId + " " +
"SET 'City (nearest)' = '" + cityState + "' " +
",'Codes' = '" + myRow.get(CODES).toString() + ",#U1' " +
"WHERE ROWID = " + myRow.get(ROW_ID);
System.out.println("[" + count + "]" + myRow.get(NUMBER) + ": " + sqlupdate);
// do the update...
if (!mtest) { // if testing then don't update
sql_doupdate(sqlupdate);
}
count++;
if ((count % 30) == 0) {
System.out.println("waiting 60 seconds");
TimeUnit.SECONDS.sleep(60); //Fusion Tables allows 30 updates then must wait 1 minute.
}
}
} catch(Exception e){
System.out.println(e.getMessage());
}
});
}

Unnecessary Foreign Key Constraint Failure

So I have a mock database for a conference where I'm creating tables for the authors, papers, reviewers, etc.
The reviewers provide an email which refers to the Program Committee emails. This is the key constraint I put in place. Then I add data to the PC table and then attempt to add data to the reviewer table. This is the error I get:
Exception encountered com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Cannot add or update a child row: a foreign key constraint fails (`sampledb`.`review`, CONSTRAINT `review_ibfk_2` FOREIGN KEY (`email`) REFERENCES `pcmember` (`email`))
Here are the other functions:
public int loadPCMember(){
String tablename = "pcmember";
String create = "CREATE TABLE IF NOT EXISTS pcmember(email VARCHAR(100), name VARCHAR(50), PRIMARY KEY (email));";
makeTable(create);
System.out.println("made table pcmember");
//CSV Reader
String[][] content = CSVReader(tablename,2);
for(int i = 0 ; i < content.length; i++){
try{
String query = "INSERT INTO pcmember(email,name) VALUES (?,?)";
PreparedStatement ps2 = net.prepareStatement(query);
ps2.setString(1, content[i][0]);
ps2.setString(2, content[i][1]);
ps2.executeUpdate();
System.out.println((i+1)+ " done");
// Throw exception
}catch (SQLException e){System.out.println("Exception encountered");return 0;}
}
System.out.println("PC Member Done");
return 1;
}
//Load next
public int loadReview(){
String tablename = "review";
String create = "CREATE TABLE IF NOT EXISTS review(reportid INTEGER, sdate DATE, comment VARCHAR(250), recommendation VARCHAR(6), paperid INTEGER NOT NULL, email VARCHAR(100) NOT NULL, PRIMARY KEY(reportid), FOREIGN KEY (paperid) REFERENCES paper(paperid), FOREIGN KEY(email) REFERENCES pcmember(email));";
makeTable(create);
System.out.println("made table review");
//CSV Reader
String[][] content = CSVReader(tablename,6);
for(int i = 0 ; i < content.length; i++){
System.out.println("" + content[i][0] + "\t" +content[i][1] + "\t" + content[i][2] + "\t" +content[i][3] + "\t" +content[i][4] + "\t" +content[i][5]);
try{
//SimpleDateFormat sdf = new SimpleDateFormat("dd/MM/yy");
//java.util.Date date = sdf.parse(content[i][1]);
//System.out.println(""+date);
//Date newDate = new Date(date.getTime());
//System.out.println(""+newDate);
String query = "INSERT INTO review(reportid,sdate,comment,recommendation,paperid,email) VALUES (?,?,?,?,?,?)";
PreparedStatement ps2 = net.prepareStatement(query);
ps2.setInt(1, Integer.parseInt(content[i][0]));
ps2.setDate(2, java.sql.Date.valueOf(content[i][1]));
ps2.setString(3, content[i][2]);
ps2.setString(4, content[i][3]);
ps2.setInt(5, Integer.parseInt(content[i][4]));
ps2.setString(6, content[i][5]);
ps2.executeUpdate();
System.out.println((i+1)+ " done");
// Throw exception
}catch (SQLException e){System.out.println("Exception encountered "+ e);return 0;
}//catch (ParseException e){System.out.println("Parse Exception encountered "+e);}
}
System.out.println("Review Done");
return 1;
}
I have a decent understanding of the key constraints and I'm pretty spot on with the CSV files having the same exact emails, so what could be causing this error?

I'm a SQL guy, not a Java guy. If I ask or suggest anything that doesn't make sense because of that - you know why. To be clear - based on that error message and the sql you have included it looks like you trying to insert a record into the review table that has an email address that doesn't exist in the pcmember table thus violating the foreign key constraint on the review table. This seems like a data problem with the CVSs you are using. However since you mentioned that you are confident in the data files is it possible that the it's trying to INSERT the rows into the review table before the pcmember INSERT has successfully completed?

Does the prepared-statement work this way?

I am trying to populate one table in my database with pretty complex data. For this, I am using a generator API (which gives me random data).
public void populateCrackers(){
PreparedStatement psm;
String queryJoke = "(SELECT jid FROM Jokes WHERE jid=?)";
String queryHat = "(SELECT hid FROM Hats WHERE hid=?)";
String queryGift = "(SELECT gid FROM Gifts WHERE gid=?)";
String query = "INSERT INTO Crackers(cid, name, jid, hid, gid, quantity) VALUES(" +
"?, " +
"?, " +
queryJoke + ", " +
queryHat + ", " +
queryGift + ", " +
"?)";
System.out.println(query);
String cracker_String = utils.JSONUtils.getJSON(crackerAPI, client);
JSONObject crackerJSON = new JSONObject(cracker_String);
JSONArray crackers = crackerJSON.getJSONArray("results");
for(int j=0; j<crackers.length(); j++){
try{
psm = connection.prepareStatement(query);
psm.setInt(1,crackers.getJSONObject(j).getInt("cid"));
psm.setString(2, crackers.getJSONObject(j).getString("cname"));
psm.setInt(3, crackers.getJSONObject(j).getInt("rjoke"));
psm.setInt(4, crackers.getJSONObject(j).getInt("rhat"));
psm.setInt(5, crackers.getJSONObject(j).getInt("rgift"));
psm.setInt(6, crackers.getJSONObject(j).getInt("cquantity"));
psm.execute();
System.out.println(crackers.getJSONObject(j).get("cid") + " "
+ crackers.getJSONObject(j).get("cname") + " "
+ crackers.getJSONObject(j).get("cquantity") + " "
+ crackers.getJSONObject(j).get("rjoke") + " "
+ crackers.getJSONObject(j).get("rhat") + " "
+ crackers.getJSONObject(j).get("rgift"));
}catch (Exception e){
e.printStackTrace();
}
}
}
This is the method that populates my "Crackers" tab. I am wondering if this be accepted as a prepared statement. When I run it in psql interactive command line tool, exactly that statement with some chosen ids (e.g INSERT INTO Crackers (cid, name, hid, jid, gid, quantity) VALUES('cid', 'name', (SELECT hid FROM Hats WHERE hid=11), (SELECT jid FROM Jokes where jid=99), (SELECT gid FROM Gifts WHERE gid=13), 5) it works flawlessly.
Does my preparedstatement break the Constraint?
Any ideas?
LATER EDIT: The inconsistency is the form of that null values can reach my Crackers table (e.g. Cracker(1, "hello", null, null, 3, 123) appears in the table.

There is nothing about Prepared statement. Constraint can be broken by parameters you set to it. And you can run your PLSQL statement as anonimous block in PreparedStatement as well.
Just surround it with BEGIN ... END. only one thing is different - for JDBC parameters are ? mark not :parameter as for PLSQL and there is no way to use named parameter.
That means if you need to use parameter more than once for JDBC you have to have that many ? marks and set all of them.
So, focus on parameters you pass to and their sequence.

The code is correct, though the prepared statement must be closed, and it would be better to create the statement once, before the for loop.
Now there is crackers.length() times a statement created but not closed. That might give problems.
Use the try-with-resouce syntax for automatic closing, irrespective of any exception or return.
try (PreparedStatement psm = connection.prepareStatement(query)) {
for (int j = 0; j < crackers.length(); j++) {
...
psm.executeUpdate();
And call executeUpdate instead of the more general execute. The resulting update count might be of interest (1/0).

I realised I had the wrong constraints on my table. I was letting null values in. There was nothing wrong with the prepared statement.
The right query to create the table is this one:
String createCrackersQuery = "CREATE TABLE Crackers(" +
" cid INTEGER," +
" name VARCHAR NOT NULL," +
" jid INTEGER NOT NULL," +
" hid INTEGER NOT NULL," +
" gid INTEGER NOT NULL," +
" quantity INTEGER NOT NULL," +
" CONSTRAINT Cracker_Primary PRIMARY KEY (cid)," +
" CONSTRAINT Cracker_Name_Unique UNIQUE(name)," +
" CONSTRAINT Joke_Foreign FOREIGN KEY (jid) REFERENCES Jokes(jid)," +
" CONSTRAINT Hat_Foreign FOREIGN KEY (hid) REFERENCES Hats(hid), " +
" CONSTRAINT Gift_Foreign FOREIGN KEY (gid) REFERENCES Gifts(gid)" +
")";

Convert PostgreSQL Serial Primary Key to Oracle Statement

I have a java file that connects to a database and this is the code:
package movies;
import java.sql.*;
public class CreateTable {
public static void main(String args[]) {
Connection c = null;
Statement stmt = null;
String sql;
try {
Class.forName("oracle.jdbc.OracleDriver");//driver
c = DriverManager.getConnection("jdbc:oracle:thin:#localhost:1521/XE", "username", "password");//PUT DATABASE CONNECTION INFO
System.out.println("Opened database successfully from within CreateTable.java");
stmt = c.createStatement();
sql = "CREATE TABLE MOVIES "
//+ "(ID SERIAL PRIMARY KEY,"//can't figure out this statement and it works if I comment it out.
+ " (NAME NVARCHAR2(255) PRIMARY KEY,"//I don't want this to be the primary key.
+ " YEAR NVARCHAR2(255),"
+ " RATING NVARCHAR2(16),"
+ " ACTORS NVARCHAR2(1024))";
stmt.executeUpdate(sql);
stmt.close();
c.close();
} catch (Exception e) {
System.err.println(e.getClass().getName() + ": " + e.getMessage());
System.exit(0);
}
System.out.println("Table created successfully");
}
}
The commented part is where I'm running into problems. I'm trying to convert a PostgreSQL statement to an Oracle 11g XE statement. I would like to have an ID with a sequential number to identify it. (ie. 0001, 0002, 0003, etc.) How can I do this? Could you provide an example? I'm at a loss right now. I'm using Netbeans 8.02, Oracle 11g XE, and Apache Tomcat 8.0.15.0.
edit I'm trying to have the ID column as the only primary key. I would eliminate the NAME column as a primary key and make the ID column the new primary key.

I don't think you can have two separate columns with primary key like that, if you want a primary key on two columns use,
edit :
sql = "CREATE TABLE MOVIES "
+ "(ID INT PRIMARY KEY," -- CHANGES
+ " NAME NVARCHAR2(255),"
+ " YEAR NVARCHAR2(255),"
+ " RATING NVARCHAR2(16),"
+ " ACTORS NVARCHAR2(1024))";
i have created the table to show the syntax works in oracle now..
SQL> create table movies
2 (id int primary key,
3 name NVARCHAR2(255),
4 year NVARCHAR2(255),
5 rating NVARCHAR2(255),
6 actors NVARCHAR2(1024));
Table created.

This code worked. See the comments below to view the changes:
try {
Class.forName("oracle.jdbc.OracleDriver");//driver
c = DriverManager.getConnection("jdbc:oracle:thin:#localhost:1521/XE", "username", "password");//PUT DATABASE CONNECTION INFO
System.out.println("Opened database successfully from within CreateTable.java");
stmt = c.createStatement();
sql = "CREATE TABLE MOVIES "
//THE FOLLOWING LINE WAS CHANGED:
+ "(\"ID\" INTEGER PRIMARY KEY, \"TYPE\" VARCHAR2(32),"
+ " NAME NVARCHAR2(255),"
+ " YEAR NVARCHAR2(255),"
+ " RATING NVARCHAR2(16),"
+ " ACTORS NVARCHAR2(1024))";
//THE FOLLOW TWO STATEMENTS WERE ADDED:
String sql1 = "CREATE SEQUENCE MOVIES_SEQ START WITH 1 INCREMENT BY 1 NOMAXVALUE";
String sql2 = "CREATE TRIGGER MOVIES_TRIGGER BEFORE INSERT ON MOVIES FOR EACH ROW BEGIN SELECT MOVIES_SEQ.NEXTVAL INTO :NEW.ID FROM DUAL; END;";
stmt.executeUpdate(sql);
//THESE TWO WERE ALSO ADDED:
stmt.executeUpdate(sql1);
stmt.executeUpdate(sql2);
stmt.close();
c.close();
} catch (Exception e) {
System.err.println(e.getClass().getName() + ": " + e.getMessage());
System.exit(0);
}

JPA: How to INSERT setting PK to MAX(PK) + 1

Scenario: I came across some code that is mixing JPA with JDBC within a transaction. The JDBC is doing an INSERT into a table with basically a blank row, setting the Primary Key to (SELECT MAX(PK) + 1) and the middleName to a temp timestamp. The method is then selecting from that same table for max(PK) + that temp timestamp to check if there was a collision. If successful, it then nulls out the middleName and updates. The method returns the newly created Primary Key.
Question:
Is there a better way to insert an entity into the database, setting the PK to max(pk) + 1 and gaining access to that newly created PK (preferably using JPA)?
Environment:
Using EclipseLink and need to support several versions of both Oracle and MS SqlServer databases.
Bonus Background: The reason I'm asking this question is because I run into a java.sql.BatchUpdateException when calling this method as part of a chain when running integration tests. The upper part of the chain uses JPA EntityManager to persist some objects.
Method in question
#Override
#TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
public int generateStudentIdKey() {
final long now = System.currentTimeMillis();
int id = 0;
try {
try (final Connection connection = dataSource.getConnection()) {
if (connection.getAutoCommit()) {
connection.setAutoCommit(false);
}
try (final Statement statement = connection.createStatement()) {
// insert a row into the generator table
statement.executeUpdate(
"insert into student_demo (student_id, middle_name) " +
"select (max(student_id) + 1) as student_id, '" + now +
"' as middle_name from student_demo");
try (final ResultSet rs = statement.executeQuery(
"select max(student_id) as student_id " +
"from student_demo where middle_name = '" + now + "'")) {
if (rs.next()) {
id = rs.getInt(1);
}
}
if (id == 0) {
connection.rollback();
throw new RuntimeException("Key was not generated");
}
statement.execute("update student_demo set middle_name = null " +
"where student_id = " + id);
} catch (SQLException statementException) {
connection.rollback();
throw statementException;
}
}
} catch (SQLException exception) {
throw new RuntimeException(
"Exception thrown while trying to generate new student_ID", exception);
}
return id;
}

First off: it hurts to answer this. But I know, sometimes you have to deal with the devil :(
So technically, it's not JPA, but if you are using Hibernate as JPA-Provider, you can go with
#org.hibernate.annotations.GenericGenerator(
name = “incrementGenerator”,
strategy = “org.hibernate.id.IncrementGenerator”)
#GeneratedValue(generator="incrementGenerator")
private Long primaryKey;
The Hibernate solution is "thread-safe", but not "cluster-safe", i.e. if you run your application on several hosts, this may fail. You may catch the appropriate exception and try again.
If you stick with your solution: close the ResultSet, Statement and the Connection. Sorry, didn't catch the try-with-resources initially.

The JDBC code is pathological, makes no sense, and will not work in a multi user environment.
I would strongly recommend fixing the code to use a sequence object, or sequence table.
In JPA you can just use sequencing.
See,
http://en.wikibooks.org/wiki/Java_Persistence/Identity_and_Sequencing#Sequencing
If you really want to do your own sequencing, you can either assign the Id yourself, use PrePersist to assign your own id, or in EclipseLink implement your own Sequence subclass that does whatever you desire. You will need to register this Sequence object using a SessionCustomizer.
See,
http://wiki.eclipse.org/EclipseLink/Examples/JPA/CustomSequencing