I currently have a very large file which contains a few million lines of entries, and want them inserted into a database. The connection established from java to SQL works as I have tried inserting the data singularly and it works, however, when I switched to using executeBatch and addBatch, it seems to loop though but not populating anything into my database.
Code is as follows:
import java.io.BufferedReader;
import java.io.FileReader;
import java.sql.Connection;
import java.sql.DatabaseMetaData;
import java.sql.DriverManager;
import java.sql.SQLException;
public class fedOrganiser6 {
private static String directory = "C:\\Users\\x\\Desktop\\Files\\";
private static String file = "combined.fed";
private static String mapperValue = "";
public static void main(String[] args) throws Exception {
Connection conn = null;
try {
BufferedReader mapper = new BufferedReader(new FileReader(directory + file));
String dbURL = "jdbc:sqlserver://localhost\\SQLExpress;database=TIMESTAMP_ORGANISER;integratedSecurity=true";
String user = "sa";
String pass = "password";
conn = DriverManager.getConnection(dbURL, user, pass);
if (conn != null) {
DatabaseMetaData dm = (DatabaseMetaData) conn.getMetaData();
System.out.println("Driver name: " + dm.getDriverName());
System.out.println("Driver version: " + dm.getDriverVersion());
System.out.println("Product name: " + dm.getDatabaseProductName());
System.out.println("Product version: " + dm.getDatabaseProductVersion());
System.out.println("clearing database");
conn.createStatement().executeUpdate("truncate table TimestampsStorage");
System.out.println("bulk insert into database");
System.out.println("complete");
int i = 0;
int records = 0;
String query = "INSERT INTO TimestampsStorage " + "values(" + "'" + mapperValue.toString() + "'"+ ")";
conn.prepareStatement(query);
for (mapperValue = mapper.readLine(); mapperValue != null; mapperValue = mapper.readLine()) {
i++;
records++;
System.out.println("Batching " + records + " records...");
conn.createStatement().addBatch(query);
if (i == 100000) {
conn.createStatement().executeBatch();
i = 0;
}
}
}
conn.createStatement().executeBatch();
conn.createStatement().close();
System.out.print("Done");
} catch (SQLException ex) {
ex.printStackTrace();
} finally {
try {
if (conn != null && !conn.isClosed()) {
conn.close();
}
} catch (SQLException ex) {
ex.printStackTrace();
}
}
}
}
createStatement() creates a new statement object, so you're execute a different statement than the one you're batching on. You should create the PreparedStatement once, add several batches to it, and then execute on the same object:
String query = "INSERT INTO TimestampsStorage VALUES (?)";
PreparedStatement ps = conn.prepareStatement(query);
for (mapperValue = mapper.readLine();
mapperValue != null;
mapperValue = mapper.readLine()) {
i++;
records++;
System.out.println("Batching " + records + " records...");
ps.setString(1, mapperValue);
ps.addBatch();
if (i == 100000) {
ps.executeBatch();
i = 0;
}
}
I think you are a bit mistaken on how batch processing for JDBC works.
You are creating a new Statement each time you call conn.createStatement().
Instead, you will want to use a PreparedStatement. First, change your query to include a ? where you want your values to go.
String query = "INSERT INTO TimestampsStorage VALUES(?)";
Then, when you call conn.prepareStatement(query), store the returned PreparedStatement.
PreparedStatement ps = conn.prepareStatement(query);
This PreparedStatement will then 'remember' your query, and you can simply change the values you want where the ? is on each iteration of your loop.
ps.setString(1, mapperValue);
The setString method will take your mapperValue and use it instead of the first ? it finds in your query (since you pass in the index 1).
Then, instead of calling conn.createStatement().addBatch(), you would call ps.addBatch().
Then, outside of your loop, you can call ps.executeBatch(). (There is no need to call this inside your loop, so you can remove your if (i == 100000) condition).
Finally, if you are using Java 7+, you can use a try with resources, so that you don't need to worry about closing the PreparedStatement or Connection in a finally block.
Here is what your end result should look like.
String query = "INSERT INTO TimestampsStorage VALUES (?)";
try (Connection con = DriverManager.getConnection(dbURL, user, pass); PreparedStatement ps = con.prepareStatement(query);) {
for (mapperValue = mapper.readLine(); mapperValue != null; mapperValue = mapper.readLine()) {
records++;
ps.setString(1, mapperValue);
ps.addBatch();
}
System.out.println("Executing batch of " + records + " records...");
ps.executeBatch();
} catch (SQLException ex) {
//handle exception
}
you are throwing away the prepared statement
String query = "INSERT INTO TimestampsStorage VALUES (?)";
PreparedStatement statement = conn.prepareStatement(query);
for (mapperValue = mapper.readLine(); mapperValue != null; mapperValue = mapper.readLine()) {
i++;
records++;
System.out.println("Batching " + records + " records...");
statement.setString(1,mapperValue);
statement.addBatch();
if (i == 100000) {
statement.executeBatch();
i = 0;
}
Related
i have built quiz web project.
i want to insert below mentioned test question into database. i am writing this question in html textarea, then getdata from html with inner.html :
but when i select this question from database it looks like that, it is hard readable, carelessly written:
help me make above mentioned tests readable.
my sql insert code :
public void addtest(TestModel test) throws Exception {
Connection c = null;
PreparedStatement ps = null;
String sql = "INSERT INTO TEST_TABLE (QUESTION,A,B,C,D,E,QUESTION_TYPE,SCORE,SUBJECT, CORRECT, Variant) " +
" VALUES (?,?,?,?,?,?,?,?,?,?,?) ";
try {
c= DbHelper.getConnection();
if(c != null) {
ps = c.prepareStatement(sql);
ps.setString(1,test.getQuestion());
ps.setString(2,test.getOptionA());
ps.setString(3,test.getOptionB());
ps.setString(4,test.getOptionC());
ps.setString(5,test.getOptionD());
ps.setString(6,test.getOptionE());
ps.setString(7,test.getQuestionType());
ps.setLong(8,test.getScore());
ps.setLong(9,test.getSubjectId());
ps.setString(10,test.getCorrectOption());
ps.setInt(11,test.getVariant());
ps.execute();
} else {
System.out.println("Connection is null");
}
} catch (Exception ex) {
ex.printStackTrace();
} finally {
JdbcUtility.close(c,ps,null);
}
}
my sql select code :
public TestModel getquestionlist(long firstpage,int variant) throws Exception {
TestModel testdata = new TestModel();
Connection c = null;
PreparedStatement ps = null;
ResultSet rs = null;
String sql =" SELECT TE.ID,TE.QUESTION,TE.A,TE.B,TE.C,TE.D,TE.E,TE.F,TE.G,TE.QUESTION_TYPE,TE.SCORE,S.NAME as Subject,TE.CREATE_DAY,TE.CORRECT, d.value, TE.Variant FROM TEST_TABLE TE " +
"INNER JOIN SUBJECT S ON S.ID = TE.SUBJECT " +
"inner join dictionary d on d.ID = TE.Variant " +
"WHERE TE.ACTIVE =1 AND S.ACTIVE =1 AND TE.Variant = ? " +
"LIMIT ?,1; ";
try {
c = DbHelper.getConnection();
if (c != null) {
ps = c.prepareStatement(sql);
ps.setInt(1,variant);
ps.setLong(2, firstpage);
rs = ps.executeQuery();
while (rs.next()) {
testdata.setId(rs.getLong("ID"));
testdata.setQuestion(rs.getString("QUESTION"));
testdata.setOptionA(rs.getString("A"));
testdata.setOptionB(rs.getString("B"));
testdata.setOptionC(rs.getString("C"));
testdata.setOptionD(rs.getString("D"));
testdata.setOptionE(rs.getString("E"));
testdata.setOptionF(rs.getString("F"));
testdata.setOptionG(rs.getString("G"));
testdata.setQuestionType(rs.getString("QUESTION_TYPE"));
testdata.setScore(rs.getLong("SCORE"));
testdata.setTestSubject(rs.getString("Subject"));
testdata.setCreateDate(rs.getDate("CREATE_DAY"));
testdata.setCorrectOption(rs.getString("CORRECT"));
testdata.setVariant(rs.getInt("Variant"));
}
}
} catch (Exception ex) {
ex.printStackTrace();
} finally {
JdbcUtility.close(c, ps, rs);
}
return testdata;
}
You can use <br/> to break line :
"1: public class WaterBottle { <br/>"
So in the moment of getting the result you can concatenate your result with <br/> for example :
testdata.setQuestion(rs.getString("QUESTION") + "<br/>");
I have code, where I have single quote or APOSTROPHE in my search
I have database which is having test table and in name column of value is "my'test"
When running
SELECT * from test WHERE name = 'my''test';
this works fine
If I use the same in a Java program I am not getting any error or any result
But If I give the name with only single quote then it works
SELECT * from test WHERE name = 'my'test';
Could you please help me out to understand.
Java code is
Connection con = null;
PreparedStatement prSt = null;
try {
Class.forName("oracle.jdbc.driver.OracleDriver");
con = DriverManager.
getConnection("jdbc:oracle:thin:#localhost:1521:orcl"
,"user","pwd");
String query = "SELECT * from "
+ "WHERE name = ? ";
prSt = con.prepareStatement(query);
String value = "my'mobile";
char content[] = new char[value.length()];
value.getChars(0, value.length(), content, 0);
StringBuffer result = new StringBuffer(content.length + 50);
for (int i = 0; i < content.length; i++) {
if (content[i] == '\'')
{
result.append("\'");
result.append("\'");
}
else
{
result.append(content[i]);
}
}
prSt.setObject(1, result.toString());
int count = prSt.executeUpdate();
System.out.println("===============> "+count);
} catch (ClassNotFoundException e) {
e.printStackTrace();
} catch (SQLException e) {
e.printStackTrace();
} finally{
try{
if(prSt != null) prSt.close();
if(con != null) con.close();
} catch(Exception ex){}
}
You don't have to escape anything for the parameter of a PreparedStatement
Just use:
prSt = con.prepareStatement(query);
prSt.setString("my'mobile");
Additionally: if you are using a SELECT statement to retrieve data, you need to use executeQuery() not executeUpdate()
ResultSet rs = prst.executeQuery();
while (rs.next())
{
// process the result here
}
You might want to go through the JDBC tutorial before you continue with your project: http://docs.oracle.com/javase/tutorial/jdbc/index.html
I am trying to map a particular character to show the full word so in this code below I have a translate method, a select method that is selecting from the remote database and update method that updates those column to my local database.
My remote database has a column called profile_ind this columns only has single chars B,C,F,P,S,W in the data, and once I do the update I want to be able to translate it to the full word only without the chars. B = BENIGN, C = CUSTOMER, F = FRAME, P = PPCOS, S = STANDARD, W = W-RED
And in my update method I want to call that translate method. cos_profile_type is the column on my local database that I want to update to either BENIGN, CUSTOMER, FRAME,PPCOS,STANDARD,W-RED depending on profile_ind column in the remote db. How can I implement it ? how can i call that translate method in the update method when I set the ? in that statement.
public static String translate(String indicator) throws Exception {
if (indicator == null || indicator.length() == 0) return "";
if (indicator.equals("B")) return "BENIGN";
if (indicator.equals("C")) return "CUSTOMER";
if (indicator.equals("F")) return "FRAME";
if (indicator.equals("P")) return "PPCOS";
if (indicator.equals("S")) return "STANDARD";
if (indicator.equals("W")) return "W-RED";
System.out.println(indicator);
return "";
}
public static void selectRecordsIcore() throws SQLException {
Connection dbConnection = null;
PreparedStatement preparedStatement = null;
Statement statement = null;
java.util.Date date= new java.util.Date();
String selectTableSQL = "SELECT profile_id, ingress_flag, egress_flag, ce_ingress_flag, ce_egress_flag, profile_ind from COS_PROFILE"
+ " WHERE profile_id >= ? AND profile_id <= ? ;";
try {
dbConnection = getInformixConnection(); //connects to ICORE database
System.out.println("ICORE Select Statement: " + selectTableSQL);
//Gets the max profile_id record
statement = dbConnection.createStatement();
ResultSet r = statement.executeQuery("SELECT max(profile_id) AS rowcount FROM COS_PROFILE");
r.next();
int maxCount = r.getInt("rowcount");
System.out.println("COS_PROFILE table has " + maxCount + " row(s).");
preparedStatement = dbConnection.prepareStatement(selectTableSQL);
preparedStatement.setInt(1, 1);
preparedStatement.setInt(2, maxCount);
// execute select SQL statement
rs = preparedStatement.executeQuery();
updateRecordIntoBids();
} catch (SQLException e) {
System.out.println(e.getMessage());
} finally {
System.out.println("inside the finally block ICORE " + new Timestamp(date.getTime()));
if (rs != null) {
rs.close();
}
if (statement != null) {
statement.close();
System.out.println("ICORE Statement Connection is closed");
}
if (preparedStatement != null) {
preparedStatement.close();
System.out.println("ICORE PreparedStatement Connection is closed");
}
if (dbConnection != null) {
dbConnection.close();
System.out.println("Database ICORE Connection is closed");
}
}
}
private static void updateRecordIntoBids() throws SQLException {
Connection dbConnection = null;
PreparedStatement preparedStatement = null;
Statement statement = null;
dbConnection = getOracleConnection(); //connects to BIDS database
java.util.Date date= new java.util.Date();
//Gets the max profile_id record
statement = dbConnection.createStatement();
dbConnection.setAutoCommit(false);
ResultSet r = statement.executeQuery("SELECT count (*) AS rowcount FROM traffic_profile_temp");
r.next();
int maxCount = r.getInt("rowcount");
System.out.println("Traffic_profile_temp table before update has " + maxCount + " row(s).");
String updateTableSQL =
"UPDATE traffic_profile_temp SET pe_ingress_flag = ?, "
+ " pe_egress_flag = ?,"
+ " ce_ingress_flag = ?,"
+ " ce_egress_flag = ?, "
+ " cos_profile_type = ? "
+ " WHERE traffic_profile_id = ? ";
// execute update SQL statement
System.out.println("BIDS Update Statement: " + updateTableSQL);
preparedStatement = dbConnection.prepareStatement(updateTableSQL);
System.out.println("Updating Bids Database in process ...");
try {
int rowCount = 0;
int expectedId = 1;
int first = 0;
while (rs.next()) {
String ingressflag = rs.getString("ingress_flag"); //BIDS column is pe_ingress_flag
String egressflag = rs.getString("egress_flag"); //BIDS column is pe_egress_flag
String ceingressflag = rs.getString("ce_ingress_flag"); //BIDS column is ce_ingress_flag
String ceegressflag = rs.getString("ce_egress_flag"); //BIDS column is ce_egress_flag
String profileind = rs.getString("profile_ind"); //BIDS column is cos_profile_type
int profileid = rs.getInt("profile_id"); //BIDS column is traffic_profile_id
preparedStatement.setString(1, ingressflag);
preparedStatement.setString(2, egressflag);
preparedStatement.setString(3, ceingressflag);
preparedStatement.setString(4, ceegressflag);
preparedStatement.setString(5, profileind);
preparedStatement.setInt(6, profileid);
preparedStatement.addBatch();
rowCount++;
if(expectedId == profileid){
if(first == 0){
first = expectedId-1;
}
}
if(expectedId != profileid){
if(first > 0){
System.out.println ("Profile id "+first+" to "+(expectedId-1)+" update.");
first = 0;
}
System.out.println ("Profile id "+expectedId+" to "+(profileid-1)+" missing.");
expectedId = profileid;
}
expectedId++;
}
if(first > 0){
System.out.println("Profile id "+first+" to "+(expectedId-1)+" update.");
first = 0;
}
preparedStatement.executeBatch();
dbConnection.commit();
System.out.println("Record(s) Updated : " + rowCount + new Timestamp(date.getTime()));
} catch (SQLException e) {
System.out.println("Update records Failed!! " + e.getMessage());
} finally {
System.out.println("inside the finally block BIDS" + " timestamp " + new Timestamp(date.getTime()));
if (statement != null) {
statement.close();
System.out.println("BIDS Statement Connection is closed");
}
if (preparedStatement != null) {
preparedStatement.close();
System.out.println("BIDS PreparedStatement Connection is closed");
}
if (dbConnection != null) {
dbConnection.close();
System.out.println("Database BIDS Connection is closed");
}
}
}
I am creating a resultset through an SQL query ('SELECT_PROCESS_INITIAL_REQUEST') which basically just grabs everything out of an SQL database table.
Statement stmt = conn.createStatement();
String sql = SQLDataAdaptor.SELECT_PROCESS_INITIAL_REQUEST;
ResultSet rsRequest = stmt.executeQuery(sql);
I then want to dump that data into an identical table to the original but in a different database. How would I do that?
Dumping it in another database means that Java will have to handle your resultset. So i'm afraid you have to do this programmatically.
Think of "prepared statements", "add batch method" and don't forget to execute your inserts every n records.
Using the example on Java Tutorial Website-
public static void viewTable(Connection con, String dbName)
throws SQLException {
//Modify this code according to 2nd database vendor and credentials-
String url = "jdbc:sqlserver://MYPC\\SQLEXPRESS;databaseName=MYDB;integratedSecurity=true";
Class.forName("com.microsoft.sqlserver.jdbc.SQLServerDriver");
Connection conn2 = DriverManager.getConnection(url);
Statement stmt = null;
Statement insertStmt = null;
String query = "select COF_NAME, SUP_ID, PRICE, " +
"SALES, TOTAL " +
"from " + dbName + ".COFFEES";
try {
stmt = con.createStatement();
insertStmt = conn2.createStatement();
ResultSet rs = stmt.executeQuery(query);
while (rs.next()) {
String coffeeName = rs.getString("COF_NAME");
int supplierID = rs.getInt("SUP_ID");
float price = rs.getFloat("PRICE");
int sales = rs.getInt("SALES");
int total = rs.getInt("TOTAL");
String insertQuery = "INSERT INTO COFFEES (COF_NAME, SUP_ID, PRICE, sales, total) VALUES (coffeeName,supplierID, proce, sales, total);
try{
insertStmt.execute(insertQuery);
}catch(SQLExeption e){
e.printStackTrace();
}
}
} catch (SQLException e ) {
e.printStackTrace();
} finally {
if (stmt != null) { stmt.close();}
if(insertStmt != null) {insertStmt.close();}
}
}
gurus,
I am new to Java SQL, and need some help.
I'm trying to get a parameter from MS SQL Server 2008. The data is definitely there - it is a current and valid DB, and I'm trying to use the users records to get cridentials for another application.
I asserted the following query:
String query = "SELECT [USER].qc_number FROM [USER] WHERE "[USER].login_name = '"
+ userNameInput + "' AND [USER].password = '" + passWordInput + "';";
Where userNameInput and passWordInput are received from the user. The URL, query and driver class are definitely correct: I checked the DB schema both from the application and from the server views. Furthermore, I verified all the Exceptions systems by changing parameters one by one, resulting in correct Exceptions messages. However, I get a resultSet with 1 column and 0 rows.
The code is below:
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
public class trOdbc
{// database URL
final String DB_URL = "***";
final String Class_URL = "com.microsoft.sqlserver.jdbc.SQLServerDriver";
private Connection connection = null; // manages connection
private Statement statement = null; // query statement
private ResultSet resultSet = null; // manages results
private Boolean connectedToDatabase = false;
// ----------------------------------------------------------
public void createJdbcConnection()
{ // connect to database books and query database
if (connectedToDatabase)
{ return; }
try
{ // connectedToDatabase is false - establish the connection
Class.forName(Class_URL);
connection = DriverManager.getConnection
(DB_URL, "***", "***" );
statement = connection.createStatement
(ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_READ_ONLY);
connectedToDatabase = true;
}
catch (SQLException ex)
{ System.out.println ("SQL Exception in connection establishment: " + ex); }
catch (ClassNotFoundException ex)
{ System.out.println ("Class not found exception in query process: " + ex); }
}
// ----------------------------------------------------------
public String [][] processJdbcQuery (String query)
{
createJdbcConnection ();
if (!connectedToDatabase)
{ return null; }// the connection wasn't established
try
{// query database
resultSet = statement.executeQuery(query);
int columns = resultSet.getMetaData().getColumnCount();
int rows = 0;
if (resultSet != null)
{
resultSet.beforeFirst();
resultSet.last();
rows = resultSet.getRow();
}
String [][] tempData = new String[rows][columns];
resultSet.beforeFirst();
rows = 0;
while (resultSet.next())
{
for (int x = 1; x <= columns; x++)
{
tempData [rows][x - 1] = resultSet.getString (x);
}
rows++;
}
CloseJdbcConnection ();
return tempData;
}
catch (SQLException ex)
{
System.out.println ("SQL Exception in query process: " + ex);
CloseJdbcConnection ();
return null;
}
} // end processJdbcQuery
// ----------------------------------------------------------
public void CloseJdbcConnection()
{
if ( connectedToDatabase )
{// close Statement and Connection. resultSet is closed automatically.
try
{
statement.close();
connection.close();
connectedToDatabase = false;
}
catch (SQLException ex)
{ System.out.println ("SQL Exception in connection closure: " + ex); }
} // end if
} // end method CloseJdbcConnection
} // end class trOdbc
Why don't you use Prepared Statement instead ?
Here is a good tutorial for using prepared statement in java
In your case it would be :
String query = "SELECT [USER].qc_number FROM [USER] " +
"WHERE [USER].login_name = ? AND [USER].password = ?;";
And then set it with different values each time you execute it like :
PreparedStatement ps = connection.prepareStatement(query);
ps.setString(1, userNameInput);
ps.setString(2, passWordInput);
resultSet = ps.executeQuery();