Hive auto increment - java

I want to create an auto_increment column in Hive.
I didn't see anything on the hive doc about that but I found that we can used :
UDFRowSequence to make that.
Is there the most recent way to do that or is there a new way, most "easier" ?
I already tried it : so in my Java project, I had created the function like that :
private static void createAutoIncrFunction() throws SQLException {
Statement stmt = conn.createStatement();
String sql = "create function autoincr as \"org.apache.hadoop.hive.contrib.udf.UDFRowSequence\"";
stmt.execute(sql);
}
The creation of the function is working.
But now I don't how to create my table with this function I tried :
private static void createTableLine() throws SQLException {
String sql = "CREATE TABLE IF NOT EXISTS line(id_line INT autoincr(), "
+ "uid_ticket VARCHAR(64), "
+ "number INT, "
+ "kind INT)";
Statement stmt = conn.createStatement();
stmt.execute(sql);
}
But It's not working, so my question is: how can I create a table with an auto_increment column and how can I insert data in it ?

The table is created normally. But while adding, you might use the function created by you.
hive> CREATE TABLE increment_table1 (id INT, c1 STRING, c2 STRING, c3 STRING);
hive> INSERT OVERWRITE TABLE increment_table1 SELECT incr() AS inc, id, c1, c2 FROM t1;
You can use this link for more information

Related

select scope_identity() using createSQLQuery in Hibernate

I am forced to use createSQLQuery to insert values into tables with an Identity column (the first column and the primary key) using hibernate. Using hibernate classes are not an option since the tables are created on the fly for each customer that is added to the system. I have run the query and it successfully inserts into the table. I then execute a "select scope_identity()" and it always returns null. "select ##Identity" works but that is not guaranteed to be the correct one. I have also tried to append "select scope_identity()" to the insert query. Then I tried query.list() and query.uniqueResult() both of which throw the hibernate exception of "No Results ..."
Session session = DatabaseEngine.getSessionFactory().openSession();
String queryString = "insert into table1 (dataid) values (1)"
SQLQuery query = session.createSQLQuery(insertQueryString);
query.executeUpdate();
query = session.createSQLQuery("select scope_identity()");
BigDecimal entryID = (BigDecimal)query.uniqueResult();
The simple example table is defined as follows:
"CREATE TABLE table1 (EntryID int identity(1,1) NOT NULL," +
"DataID int default 0 NOT NULL, " +
"PRIMARY KEY (EntryID))";
Is there a way I am missing to use scope_identity() with createSQLQuery?
Actually the SQLServerDialect class used by Hibernate uses the same "scope_identity()" too.
The reason why it's not working is because you need to execute those in the same statement or stored procedure.
If you execute the scope_identity() call in a separate statement, SQL Server will not be able to give you last inserted identity value.
You cannot do it with the SQLQuery, even Hibernate uses JDBC to accomplish this task. I wrote a test on GitHub to emulate this and it works like this:
Session session = entityManager.unwrap(Session.class);
final AtomicLong resultHolder = new AtomicLong();
session.doWork(connection -> {
try(PreparedStatement statement = connection.prepareStatement("INSERT INTO post VALUES (?) select scope_identity() ") ) {
statement.setString(1, "abc");
if ( !statement.execute() ) {
while ( !statement.getMoreResults() && statement.getUpdateCount() != -1 ) {
// do nothing until we hit the resultset
}
}
try (ResultSet rs = statement.getResultSet()) {
if(rs.next()) {
resultHolder.set(rs.getLong(1));
}
}
}
});
assertNotNull(resultHolder.get());
The code uses Java 8 lambdas instead of anonymous classes, but you can easily port it to Java 1.7 too.

JDBC "Error in MySQL Syntax" when trying to switch to (USE) a database

I've been trying to use this code to create a table in a database that the code before this created successfully.
//Creates "Parts" Table if it's not already created.
public void createTableIfNecessary() throws SQLException{
String createString =
"USE inventory; " +
"CREATE TABLE IF NOT EXISTS parts " +
"(part_name TEXT NOT NULL, remaining_amt INT NOT NULL, " +
"restock_amt INT, barcode TEXT, location TEXT, part_id INT NOT NULL AUTO_INCREMENT);";
Statement createTable = this.con.createStatement();
createTable.execute(createString);
System.out.println("Table created or already existed: Parts");
}
My code to connect to the Database server:
try{Class.forName("com.mysql.jdbc.Driver");} catch(Exception e){
e.printStackTrace();
}
this.con = null;
Properties connectionProps = new Properties();
connectionProps.put("user", this.userName);
connectionProps.put("password", this.password);
this.con = DriverManager.getConnection("jdbc:mysql://" + this.serverName + ":" + this.portNumber + "/", connectionProps);
System.out.println("Connected to MySQL Database Server");
(This works, but it may be helpful to see)
My Error:
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'CREATE TABLE IF NOT EXISTS parts (part_name TEXT NOT NULL, remaining_amt INT NOT' at line 1
I Assume it is just my incompetence with MySQL, but maybe someone here can see my syntax error. Thanks!
The USE inventory; statement is not appropriate in the context of a JDBC connection. You need to use Connection#setCatalog() to switch to a particular db. For more details, see
Java, how to change current database to another?
So do that and remove the USE clause from your SQL.
For auto_increment column in mysql, it has to be indexed. So you have to index part_id by making it a primary key or unique.
CREATE TABLE IF NOT EXISTS parts
(part_name TEXT NOT NULL,
remaining_amt INT NOT NULL,
restock_amt INT,
barcode TEXT,
location TEXT,
part_id INT NOT NULL AUTO_INCREMENT,
primary key (part_id));

Preventing from Multiple primary key error

This is my code for executing in my java program:
public static void createBooksTablesAndSetPK() {
String selectDB = "Use lib8";
String createBooksTable = "Create table IF NOT EXISTS books (ID int,Name varchar(20),ISBN varchar(10),Date date )";
String bookTablePK = "ALTER TABLE BOOKS ADD PRIMARY KEY(id)";
Statement st = null;
try (
Connection con = DriverManager.getConnection(dbUrl, "root", "2323");) {
st = con.createStatement();
st.execute(selectDB);
st.execute(createBooksTable);
st.execute(bookTablePK);
} catch (SQLException sqle) {
sqle.printStackTrace();
}
}
I cat use IF NOT EXIST for creating databasesand tables to prevent creating duplicate database and tables and corresponding errors.
But i don't know how prevent Multiple primary key error, because program may call createBooksTablesAndSetPK() multiple times.
Error:
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Multiple primary key defined
The column Book_id is not existing in your case. You are creating a table with ID as the column and then updating the table with a PRIMARY KEY constraint on a column that is not existing.
Create table IF NOT EXISTS books (ID int,Name varchar(20),ISBN varchar(10),Date date )
ALTER TABLE BOOKS ADD PRIMARY KEY(BOOK_id)
Try running these statements on a MySQL command prompt (or MySql Workbench) and the see the error.
You need change the alter table command like this.
ALTER TABLE BOOKS ADD BOOK_id VARCHAR( 255 ), ADD PRIMARY KEY(BOOK_id);

Getting column metadata from jdbc/postgresql for newly created table

I'm trying to get the column list from newly created table(it is created in the java code).
The thing is that I do not get the columns.
The code works for tables that are already in the database, but if i create a new one and try to get the column info immediately it does not find any...
Update:
Here is full code that I used for testing:
#Test
public void testtest() throws Exception {
try (Connection conn = dataSource.getConnection()) {
String tableName = "Table_" + UUID.randomUUID().toString().replace("-", "");
try (Statement statement = conn.createStatement()) {
statement.executeUpdate(String.format("create table %s (id int primary key,name varchar(30));", tableName));
}
DatabaseMetaData metaData = conn.getMetaData();
try (ResultSet rs = metaData.getColumns(null, null, tableName, null)) {
int colsFound = 0;
while (rs.next()) {
colsFound++;
}
System.out.println(String.format("Found %s cols.", colsFound));
}
System.out.println(String.format("Autocommit is set to %s.", conn.getAutoCommit()));
}
}
The and the output:
Found 0 cols.
Autocommit is set to true.
The problem is with the case of your tablename:
String tableName = "Table_"
As that is an unquoted identifier (a good thing) the name is converted to lowercase when Postgres stores its name in the system catalog.
The DatabaseMetaData API calls are case sensitive ( "Table_" != "table_"), so you need to pass the lowercase tablename:
ResultSet rs = metaData.getColumns(null, null, tableName.toLowerCase(), null))
More details on how identifiers are using are in the manual: http://www.postgresql.org/docs/current/static/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS
I have made simple test and it seems to work. I can create new table and show its columns using PostgreSQL JDBC (I use Jython):
conn = db.createStatement()
conn.execute("CREATE TABLE new_table (id SERIAL, txt VARCHAR(200))")
db_meta_data = db.getMetaData()
for tbl_name in ('date_test', 'new_table'):
print('\n-- %s --' % (tbl_name))
rs = db_meta_data.getColumns(None, None, tbl_name, None)
while (rs.next()):
print('%s:%s' % (rs.getString(3), rs.getString(4)))
conn.close()
This code shows columns for both already existing table: date_test and for just created new_table. I also added some code to close connection after CREATE TABLE but my results are always the same and correct.
Maybe it is problem with your JDBC driver. I use driver from postgresql-9.3-1100.jdbc41.jar.
It may be also problem with user permissions. Do you use the same user for both creating table and getting metadata? Is new table visible in psql, pgAdmin or other tool?
Other reason is that PostgreSQL uses transactions also for schema changes. So if you disabled default autocommit and closed connection your schema changes will be lost. Do you use db.setAutoCommit(false)?
You can also query PostgreSQL schema directly:
SELECT DISTINCT table_name, column_name
FROM information_schema.columns
WHERE table_schema='public'
AND table_name = 'new_table'
ORDER BY 1, 2
Strangely giving passing table name in lower case to getColumns method does work...thanks for the query MichaƂ Niklas it got me on the right track.

Creating a database table if it does not exist in Java production code and confirming in JUnit

I am writing a database program in Java and want to create a table if it does not already exist. I learned about DatabaseMetaData.getTables() from How can I detect a SQL table's existence in Java? and I am trying to use it:
private boolean tableExists() throws SQLException {
System.out.println("tableExists()");
DatabaseMetaData dbmd = conn.getMetaData();
ResultSet rs = dbmd.getTables(null, null, this.getTableName(), null);
System.out.println("TABLE_NAME: " + rs.getString("TABLE_NAME"));
return rs.getRow() == 1;
}
The problem is that rs.getRow() always returns 0, even after the table has been created. Using rs.getString("TABLE_NAME") throws an exception stating that the result set is empty.
One possible solution I thought of is to execute the CREATE TABLE statement and catch any exceptions that are thrown. However, I don't like the idea of using exceptions for control flow of my program.
FWIW, I am using HSQLDB. However, I would like write Java code that is independent of the RDMS engine. Is there another way to use DatabaseMetaData.getTables() to do what I want? Or is there some other solution to write my tableExists() method?
Added:
Using the suggestions given here, I found a solution that seems to work in my production code:
private void createTable() throws SQLException {
String sqlCreate = "CREATE TABLE IF NOT EXISTS " + this.getTableName()
+ " (brand VARCHAR(10),"
+ " year INTEGER,"
+ " number INTEGER,"
+ " value INTEGER,"
+ " card_count INTEGER,"
+ " player_name VARCHAR(50),"
+ " player_position VARCHAR(20))";
Statement stmt = conn.createStatement();
stmt.execute(sqlCreate);
}
Now I am also writing a JUnit test to assert that the table does indeed get created:
public void testConstructor() throws Exception {
try (BaseballCardJDBCIO bcdb = new BaseballCardJDBCIO(this.url)) {
String query = "SELECT count(*) FROM information_schema.system_tables WHERE table_name = '" + bcdb.getTableName() + "'";
Connection conn = DriverManager.getConnection(this.url);
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(query);
Assert.assertTrue(rs.next());
Assert.assertEquals(1, rs.getInt(1));
Assert.assertFalse(rs.next());
}
}
This test fails on the assertEquals() with the following message:
FAILED: expected: <1> but was: <0>
The solution I found seems to work:
private void createTable() throws SQLException {
String sqlCreate = "CREATE TABLE IF NOT EXISTS " + this.getTableName()
+ " (brand VARCHAR(10),"
+ " year INTEGER,"
+ " number INTEGER,"
+ " value INTEGER,"
+ " card_count INTEGER,"
+ " player_name VARCHAR(50),"
+ " player_position VARCHAR(20))";
Statement stmt = conn.createStatement();
stmt.execute(sqlCreate);
}
I had to place the IF NOT EXISTS in the correct location in my SQL statement.
From the ResultSet definition at Java docs:
A ResultSet object maintains a cursor pointing to its current row of
data. Initially the cursor is positioned before the first row. The
next method moves the cursor to the next row, and because it returns
false when there are no more rows in the ResultSet object, it can be
used in a while loop to iterate through the result set.
So, you must always call the next() method otherwise getRow() will always return zero as the cursor is positioned before the first row.
There is build in mysql functionality for what you seek: http://dev.mysql.com/doc/refman/5.0/en/create-table.html
In short: Just append IF NOT EXISTS at the end of your table creation query.
Edit:
There is no general way of doing this. Most databases have an information_scheme table though, a query to determine the information could look like this:
SELECT count(*)
FROM information_schema.system_tables
WHERE table_schem = 'public' AND table_name = 'user';
This works with sqlite, mysql, msql, mariadb and postgres + probably a lot of others.
I don't know if this will necessarily help towards your goals, but when I ran into this problem using Python and MySQL, I just added a "Drop Table" statement before each "Create Table" statement, such that just running the script automatically deletes the existing table, and then rebuilds each table. That may not work for your needs, but it's one solution that I found successful for my similar problem.

Categories