We Keep Coding

Inserting rows in sql table problem, some columns inserted NULL value

I have 2 DB with multiple tables. All tables have the same syntax. Here I have a method that takes name of the table as an argument. The table that I try to insert is with 3 columns (int, varchar, int). The problem is, only the first row is inserted, and the 2 and 3 row is NULL, I don't know what is the problem. Any suggestions, please?
public void getAndInsertData(String nameOfTable) {
try {
Class.forName("com.mysql.jdbc.Driver");
Connection con1 = DriverManager.getConnection(urlDB1, user1, password1);
Statement s1 = con1.createStatement();
Connection con2 = DriverManager.getConnection(urlDB2, user2, password2);
Statement s2 = con2.createStatement();
ResultSet rs1 = s1.executeQuery("SELECT * FROM " + nameOfTable);
ResultSetMetaData rsmd1 = rs1.getMetaData();
int columnCount = rsmd1.getColumnCount();
for (int column = 1; column <= columnCount; column++) {
String columnName = rsmd1.getColumnName(column);
int columnType = rsmd1.getColumnType(column);
while (rs1.next()) {
switch (columnType) {
case Types.INTEGER:
case Types.SMALLINT:
case Types.BIGINT:
case Types.TINYINT:
int integerValue = rs1.getInt(column);
String integerQuery = "insert into " + nameOfTable + " (" + columnName + ") VALUES("
+ integerValue + ");";
s2.executeUpdate(integerQuery);
break;
case Types.VARCHAR:
case Types.NVARCHAR:
case Types.LONGNVARCHAR:
case Types.LONGVARCHAR:
String varcharValue = rs1.getString(column);
String varcharQuery = "insert into " + nameOfTable + " (" + columnName + ") VALUES("
+ varcharValue + ");";
s2.executeUpdate(varcharQuery);
default:
System.out.println("Default");
break;
}
}
}
} catch (Exception e) {
e.printStackTrace();
}
}

Your integerQuery and varcharQuery both insert into datebase table a record with one filled column and blank other columns. Because you provide value to one column only.

A few issues:
Use try-with-resources to make sure the JDBC resources are cleaned up correctly.
No need for a switch statement, because we don't actually need to know the types of the columns. The JDBC driver will handle that if you use getObject() and setObject().
Only execute one INSERT per row from the source table.
Use batching when inserting a lot of records, for better performance.
Here is how to do it:
try (
Connection conSource = DriverManager.getConnection(urlDB1, user1, password1);
Connection conTarget = DriverManager.getConnection(urlDB2, user2, password2);
Statement stmtSource = conSource.createStatement();
ResultSet rsSource = stmtSource.executeQuery("SELECT * FROM " + nameOfTable);
) {
// Build insert statement
ResultSetMetaData metaData = rsSource.getMetaData();
int columnCount = metaData.getColumnCount();
StringBuilder sql = new StringBuilder("INSERT INTO " + nameOfTable + " (");
for (int column = 1; column <= columnCount; column++) {
if (column != 1)
sql.append(", ");
sql.append(metaData.getColumnName(column));
}
sql.append(") VALUES (");
for (int column = 1; column <= columnCount; column++) {
if (column != 1)
sql.append(", ");
sql.append("?");
}
sql.append(")");
// Copy data
conTarget.setAutoCommit(false);
try (PreparedStatement stmtTarget = conTarget.prepareStatement(sql.toString())) {
int batchSize = 0;
while (rsSource.next()) {
for (int column = 1; column <= columnCount; column++) {
// Copy row here. Use switch statement to control the mapping
// if source and target table columns don't have compatible types.
// The following statement should work for most types, so switch
// statement only needs to cover the exceptions.
stmtTarget.setObject(column, rsSource.getObject(column), metaData.getColumnType(column));
}
stmtTarget.addBatch();
if (++batchSize == 1000) { // Flush every 1000 rows to prevent memory overflow
stmtTarget.executeBatch();
batchSize = 0;
}
}
if (batchSize != 0)
stmtTarget.executeBatch();
}
conTarget.commit();
}

As The Impaler already mentioned, your loops are at the wrong place.
For every record of rs1, you want to insert one record using s2.
You can build a prepared statement first using the metadata and then inject the values:
ResultSetMetaData rsmd1 = rs1.getMetaData();
int columnCount = rsmd1.getColumnCount();
StringBuffer sql=new StringBuffer("insert into "+nameOfTable+" (");
for (int column = 1; column <= columnCount; column++) {
String columnName = rsmd1.getColumnName(column);
if(column>1)
sql.append(",");
sql.append(columnName);
}
sql.append(") values (");
for(int i=1;i<=columnCount;i++)
{
sql.append((i==1?"":",")+ "?");
}
sql.append(")");
System.out.println("Prepared SQL:"+sql.toString());
// sql = insert into nameOfTable (col1,col2,col3) values (?,?,?)
PreparedStatement s2= con2.prepareStatement(sql.toString());
while (rs1.next()) {
s2.clearParameters();
for (int column = 1; column <= columnCount; column++) {
int columnType = rsmd1.getColumnType(column);
switch (columnType) {
case Types.INTEGER:
case Types.SMALLINT:
case Types.BIGINT:
case Types.TINYINT:
s2.setInt(column, rs1.getInt(column));
break;
case Types.VARCHAR:
case Types.NVARCHAR:
case Types.LONGNVARCHAR:
case Types.LONGVARCHAR:
s2.setString(column, rs1.getString(column));
break;
default:
System.err.println("Not supported type for column "+column+" with type:"+columnType);
s2.setNull(column, columnType);
break;
}
} // end of for loop
// execute statement once per record in rs1
s2.executeUpdate();
} // end of while

out of memory when trying to select data from oracle database and insert it into sqlite

we get out of memory (the process go up and consumes %100 of ram) trying to select data and insert it into SQLite DB, the program written in java, and the data are so big, we even made pagination for it, it selects 100000 rows and inserts it into SQLite database, to figure out the problem, we commented out all line from the code that insert the data, and we saw that consumption of ram stop's at %6
package dbex;
import java.sql.*;
import java.util.ArrayList;
public class DBEX {
String url = "jdbc:oracle:thin:#192.168.120.46:1521:db";
String user = "dbuser";
String password = "dbpass";
ArrayList<Table> Tales = new ArrayList<>();
public static void main(String[] args) {
DBEX db = new DBEX();
db.CreateDB();
}
public int CountRecords(String table) {
int result = 0;
try {
Class.forName("oracle.jdbc.driver.OracleDriver");
DriverManager.registerDriver(new oracle.jdbc.driver.OracleDriver());
Connection getCountCon = DriverManager.getConnection(url, user, password);
Statement CountSt = getCountCon.createStatement();
ResultSet countRs = CountSt.executeQuery("SELECT COUNT(*) as num FROM " + table);
while (countRs.next()) {
result = countRs.getInt("num");
}
} catch (Exception ex) {
}
return result;
}
public void dumpData() {
try {
int incrementRecord = 100000;
Class.forName("org.sqlite.JDBC");
Class.forName("oracle.jdbc.driver.OracleDriver");
DriverManager.registerDriver(new oracle.jdbc.driver.OracleDriver());
Table table = null;
ArrayList<String> columns = new ArrayList<>();
Connection getDataCon = DriverManager.getConnection(url, user, password);
getDataCon.setAutoCommit(false);
Statement dataSt = getDataCon.createStatement();
dataSt.setFetchSize(incrementRecord);
ResultSet dataRs = null;
Connection LiteInsertCon = DriverManager.getConnection("jdbc:sqlite:" + "DBEX" + ".db");
LiteInsertCon.setAutoCommit(false);
PreparedStatement insertStatement = null;
int numRecord = 0;
for (int i = 0; i < Tales.size(); i++) {
table = Tales.get(i);
String tableName = table.name;
System.out.println("table::" + tableName);
System.out.println("Dumping data from " + tableName);
String InsertSQL = "INSERT INTO " + tableName + " ( ";
String vals = "VALUES (";
String selectSQL = "SELECT * FROM ( SELECT b.*, ROWNUM RN FROM ( SELECT ";
System.out.println("geting column name...");
columns = table.Columns;
for (int j = 0; j < columns.size(); j++) {
String columnName = columns.get(j);
if (j == 0) {
vals += " ? ";
InsertSQL += columnName + " ";
selectSQL += columnName + " ";
}else {
InsertSQL += " , " + columnName + " ";
vals += ", ? ";
selectSQL += " , " + columnName + " ";
}
}
System.out.println("Number of column of table " + tableName + " is " + columns.size());
vals += ")";
InsertSQL += ")" + vals;
selectSQL += "FROM " + tableName + " ORDER BY "+table.primary_key+" ASC ) b WHERE ROWNUM <= :TO ) WHERE RN > :FROM ";
System.out.println(selectSQL);
System.out.println(InsertSQL);
int parIndex = 0;
String colName = null;
String data = null;
numRecord = CountRecords(tableName);
int from = 0;
int to = incrementRecord;
if (to > numRecord) {
System.out.println("Table data is less than "+incrementRecord+" geting all data "+numRecord+" " );
dataRs = dataSt.executeQuery(selectSQL.replace(":TO", numRecord + "").replace(":FROM", "0"));
while (dataRs.next()) {
insertStatement = LiteInsertCon.prepareStatement(InsertSQL);
insertStatement.setFetchSize(incrementRecord);
for (; parIndex < columns.size(); parIndex++) {
colName = columns.get(parIndex);
data = dataRs.getString(colName);
if (data != null) {
insertStatement.setString(parIndex + 1, data);
} else {
insertStatement.setString(parIndex + 1, " ");
}
parIndex++;
}
parIndex = 0;
insertStatement.executeUpdate();
insertStatement = null;
}
} else {
while (numRecord >= to) {
System.out.println("page "+(to/incrementRecord)+" of "+(numRecord/incrementRecord) );
dataRs = dataSt.executeQuery(selectSQL.replace(":TO", to + "").replace(":FROM", from+""));
while (dataRs.next()) {
insertStatement = LiteInsertCon.prepareStatement(InsertSQL);
insertStatement.setFetchSize(incrementRecord);
for (; parIndex < columns.size(); parIndex++) {
colName = columns.get(parIndex);
data = dataRs.getString(colName);
if (data != null) {
insertStatement.setString(parIndex + 1, data);
} else {
insertStatement.setString(parIndex + 1, " ");
}
parIndex++;
}
parIndex = 0;
insertStatement.executeUpdate();
insertStatement = null;
}
dataRs=null;
from = to;
to += incrementRecord;
System.gc();
}
}
}
} catch (Exception ex) {
System.out.println(ex.toString());
}
}
class Table {
public String name;
public String primary_key;
ArrayList< String> Columns = new ArrayList<>();
public Table(String name, String primary_key, ArrayList< String> Columns) {
this.name = name;
this.primary_key = primary_key;
this.Columns = Columns;
}
}
public void CreateDB() {
try {
Class.forName("oracle.jdbc.driver.OracleDriver");
System.out.println("Oracle JDBC driver found");
DriverManager.registerDriver(new oracle.jdbc.driver.OracleDriver());
Connection getTableCon = DriverManager.getConnection(url, user, password);
Connection getColumnCon = DriverManager.getConnection(url, user, password);
Connection getKeyCon = DriverManager.getConnection(url, user, password);
System.out.println("connected to database");
Statement getTableSt = getTableCon.createStatement();
PreparedStatement getColumnSt = getColumnCon.prepareStatement("SELECT DISTINCT column_name FROM all_tab_cols WHERE table_name = ? ");
PreparedStatement getKeySt = getKeyCon.prepareStatement("SELECT cols.table_name, cols.column_name, cols.position, cons.status, cons.owner FROM all_constraints cons, all_cons_columns cols WHERE cols.table_name = ? AND cons.constraint_type = 'P' AND cons.constraint_name = cols.constraint_name AND cons.owner = cols.owner ORDER BY cols.table_name, cols.position");
ResultSet getTableRs = getTableSt.executeQuery("SELECT * FROM all_tables WHERE OWNER not in('SYS','OUTLN','SYSTEM')");
ResultSet getColumnRs;
ResultSet getKeyRs;
Class.forName("org.sqlite.JDBC");
System.out.println("Sqlite JDBC driver found");
Connection LiteCreateTableCon = null;
LiteCreateTableCon = DriverManager.getConnection("jdbc:sqlite:" + "DBEX" + ".db");
Statement statement = LiteCreateTableCon.createStatement();
String keyName = "";
while (getTableRs.next()) {
String tableName = getTableRs.getString("table_name");
System.out.println("table: " + tableName);
getColumnSt.setString(1, tableName);
System.out.println("SELECT * FROM all_tab_cols WHERE table_name = " + tableName + "");
getColumnRs = getColumnSt.executeQuery();
String createTableSQL = "create table if not exists " + tableName + " ( ";
int index = 0;
ArrayList<String> Columns = new ArrayList<>();
while (getColumnRs.next()){
String columnName = getColumnRs.getString("column_name");
Columns.add(columnName);
System.out.println("column: " + columnName);
if (index == 0) {
createTableSQL += columnName + " string";
} else {
createTableSQL += " , " + columnName + " string";
}
index++;
}
createTableSQL += ")";
System.out.println(createTableSQL);
statement.executeUpdate(createTableSQL);
getKeySt.setString(1, tableName);
getKeyRs = getKeySt.executeQuery();
keyName = "1";
while (getKeyRs.next()) {
keyName = getKeyRs.getString("column_name");
}
Tales.add(new Table(tableName, keyName, Columns));
}
getTableCon.close();
getColumnCon.close();
getKeyCon.close();
System.out.println("All table created...");
}catch (Exception ex) {
System.out.println(ex.toString());
}
dumpData();
}
}

You need to issue/excute commit on SQLite after inserting every batch of records.
And try to come-up with suitable batch size (may be 100).

How to add double quotes to the output values in a Resultset when the column datatype is String?

I am trying to run the below query and insert the output of it into another table (postgres db):
String ssnQuery = "SELECT period_year, period_name, period_num, NULL as count_of_issues,
ledger_id,
balancing_segment,
Count(*) AS count_of_account_segments,
Sum(accounted_period_net_dr) AS balance_accounted_period_net_dr,
Sum(accounted_period_net_cr) AS balance_accounted_period_net_cr,
Round(Sum(accounted_period_net_dr_cr)) AS balance_accounted_period_net_dr_cr_diff,
Sum(entered_period_net_dr) AS balance_entered_period_net_dr,
Sum(entered_period_net_cr) AS balance_entered_period_net_cr,
Round(Sum(entered_period_net_dr_cr)) AS balance_entered_period_net_dr_cr_diff,
Sum(begin_balance_dr) AS begin_balance_dr,
Sum(begin_balance_cr) AS begin_balance_cr,
Round(Sum(net_beginning_balance)) AS net_beginning_balance,
Round(Sum(net_closing_balance)) AS net_closing_balance
FROM schema.tablename";
try {
pstmnt = financeConnection.prepareStatement(ssnQuery);
rs = pstmnt.executeQuery();
rsmd = rs.getMetaData();
for(int i=1; i<=rsmd.getColumnCount(); i++) {
columnNames.add(rsmd.getColumnName(i));
if(i == 1) {
queryColumns = rsmd.getColumnName(i);
} else if(i<7) {
queryColumns += ", " + rsmd.getColumnName(i);
} else {
queryColumns += ", value_" + (i-7);
}
}
while (rs.next()) {
queryValues = " ";
for(String colname: columnNames) {
if(queryValues.isEmpty()) {
queryValues = rs.getString(colname);
} else {
queryValues += rs.getString(colname) + ", ";
}
}
remCommas = queryValues.replaceAll(", $", "");
insertQuery = "INSERT INTO bdmerge.gen_audit_func_hive_results (run_id, run_date, run_date_ist" + queryColumns + ") VALUES (" + runid +"," + utcTimeStamp + "," + istTimeStamp + "," + remCommas + ")";
System.out.println("Final Insert query: " + insertQuery);
}
} catch (SQLException e) {
e.printStackTrace();
} catch(Exception e) {
e.printStackTrace();
}
To insert the output of above query, I formed the insert query according the column names in the destination table as below:
INSERT INTO schema.destinationTable (run_id, run_date, run_date_ist, source_system_type, source_system, module, source_table_name, period_year, period_name, period_num, count_of_issues, count_of_accounted_issues, count_of_entered_issues, value_0, value_1, value_2, value_3, value_4, value_5, value_6, value_7, value_8, value_9, value_10, value_11, value_12) VALUES (781,2018-11-12 08:15:32.0,2018-11-12 13:45:32.0, 2018, OCT-18, 10, null, 1, 1, 2073, ATRS, 6135, 6.2778220466107E11, 6.277946274560101E11, -1.2422795E7, 5.929031383587201E11, 5.9291556115366E11, -1.2422795E7, 3.931397937759116E13, 3.9313979377591164E13, 0.0)
But the destination table's columns:
run_id, count_of_issues, count_of_accounted_issues, count_of_entered_issues
are in numeric format (working on postgres db) and all others are varchar(1000).
In order to insert the varchar data, I need to enclose the column values from value_0 till value_12 in double quotes.
Without properly modifying the data, I am getting SQLException while inserting which is expected.
Is there any way I can just enclose those varchar column values from the resultSet in double quotes and insert them into destination table?

You need to add double quotes to the string values; Use Escape characters, like this:
String quotedStr = "Non Quoted," + " \"Quoted\".";
Modifying your code to address the problem:
while (rs.next()) {
queryValues = " ";
int i = 0;
for(String colname: columnNames) {
if(queryValues.isEmpty()) {
queryValues = rs.getString(colname);
} else {
if (i >= 7)
queryValues += "\"" + rs.getString(colname) +"\"" + ", ";
}
else
queryValues += rs.getString(colname) + ", ";
i++;
}

how to correctly MySQL "desc table" via java connectivity

I am actually building a project where the names of tables present in databases of mysql are displayed in a list in java.
user selects a table from a list and the description of that table in given using "desc tablename" command.
The problem is, it is supposed to get every field in table but it only gets first field. Below i have explained it more, but first heres my code-:
try {
int rowCount = tableModel.getRowCount();
for (int i = 0; i < rowCount; i++) {
tableModel.removeRow(i);
}
String z = jList2.getSelectedValue().toString();
try {
Class.forName("java.sql.DriverManager");
} catch (ClassNotFoundException e) {
timeget();
jTextArea4.append(now + ": " + "/ Failed in getting Driver \n Error Message: " + e.getMessage() + " / \n \n");
JOptionPane.showMessageDialog(this, e.getMessage());
}
DriverManager.getConnection("jdbc:mysql://localhost:" + GlobalParams.portvar + "/", "" + k, "" + j);
stmnt = (Statement) con.createStatement();
String query = "desc " + z;
jTextArea5.append(now + ": " + "/ desc " + z + "; / \n \n");
ResultSet rs = stmnt.executeQuery(query);
String[] cnames = {"Field", "Type", "Null", "Key", "Extra"};
tableModel.setColumnIdentifiers(cnames);
jTable1.setAutoResizeMode(JTable.AUTO_RESIZE_ALL_COLUMNS);
jTable1.setFillsViewportHeight(true);
if (rs.next()) {
String field = rs.getString("Field");
String type = rs.getString("type");
String nullinfo = rs.getString("null");
String key = rs.getString("key");
String extra = rs.getString("extra");
tableModel.addRow(new Object[]{field, type, nullinfo, key, extra});
}
catch(SQLException e){//some blabla}
Now for detailing into problem->
Say from the list, i select a table called "city". Originally, it has four fields- ID, Name, Population, CountryCode. But in my jTable, only "ID" appears.
the code...
int rowCount = tableModel.getRowCount();
for (int i = 0; i < rowCount; i++) {
tableModel.removeRow(i);
}
...is simply to remove fields of old table when a new table is selected from list.
Hope i clarified my prob.
Please help regarding this. Thanks.

the problem is in this
if (rs.next()) {
String field = rs.getString("Field");
String type = rs.getString("type");
String nullinfo = rs.getString("null");
String key = rs.getString("key");
String extra = rs.getString("extra");
tableModel.addRow(new Object[]{field, type, nullinfo, key, extra});
}
you are not continuing the loop
change the if loop to while loop
while(rs.next()) {
String field = rs.getString("Field");
String type = rs.getString("type");
String nullinfo = rs.getString("null");
String key = rs.getString("key");
String extra = rs.getString("extra");
tableModel.addRow(new Object[]{field, type, nullinfo, key, extra});
}

MySQL command SELECT values in column - JDBC

I'm using MySQL commands via JDBC (Java) to make changes to my database. I have implemented the following method to return the values of a column. The goal is to have the location in the column (row) correspond with their location in the array (index). This works with String columns, but with numerical columns, the ResultSet seems to place them in ascending order, thus making their positioning in the returned String array not reflect their positioning in the column. 'rs' is a ResultSet reference variable.
public String[] getColumnContents(String tableName, String columnName) {
String sql = "SELECT " + columnName + " FROM " + tableName;
String[] results = new String[SQLManager.getColumnLength(tableName, columnName)];
try {
rs = statement.executeQuery(sql);
for (int counter = 0; rs.next(); counter++) {
results[counter] = rs.getString(columnName);
}
} catch (SQLException e) {
e.printStackTrace();
}
return results;
}

It's as simple as adding an ORDER BY clause to the SQL command. Here's my working method:
public String[] getColumnContents(String tableName, String columnName) {
String sql = "SELECT " + columnName + " FROM " + tableName + " ORDER BY " + columnName1 + " ASC, " + columnName2 + " ASC";
String[] results = new String[SQLManager.getColumnLength(tableName, columnName)];
try {
rs = statement.executeQuery(sql);
for (int counter = 0; rs.next(); counter++) {
results[counter] = rs.getString(columnName);
}
} catch (SQLException e) {
e.printStackTrace();
}
return results;
}