Related
Currently I am working on a solution to read a 454 character/line based huge file (minimum 50000 rows) via Java.As per the requirement it is a positioned based file, we first need to read the file , then parse the position based values and need to insert into a table. (minimum 96 positions will be inserted into 96 columns of the table).
I took this concept after the parsing.
[ INSERT ALL INTO<TABLE NAME> [COL1,COL2,COL3] Values [VAL1,VAL2,VAL3]
INTO<TABLE NAME> [COL1,COL2,COL3] Values [VAL1,VAL2,VAL3]
SELECT * FROM DUAL;]
Here is my code:
try{
char[] line = new char[456];
while(br.read(line) > 0){
StringBuilder input = new StringBuilder(new String(line));
if(batchCounter>0){
int detailFileId = interfaceFileSequence();
sql.append(initSql+"(" +
detailFileId + "," + interfaceHeaderId + ", SYSDATE," +
interfaceRunId + "," + isSpace(input.substring(0, 2).trim()) + "," + "TO_DATE("+isSpace(input.substring(2, 12).trim())+",'YYYY-MM-DD')" +","+isSpace(input.substring(12, 22).trim()) + "," +
Double.parseDouble(input.substring(22, 35)+ "." + input.substring(35, 37)) + ", " +
Double.parseDouble(input.substring(22, 35)+ "." + input.substring(35, 37)) + ", " +
isSpace(input.substring(38, 44).trim()) + ","+isSpace(input.substring(38, 44).trim())+"," +isSpace(input.substring(38, 44).trim())+"," + isSpace(input.substring(44, 54).trim())+","+
isSpace(input.substring(54, 60).trim()) + "," + isSpace(input.substring(60, 68).trim()) + "," + isSpace(input.substring(68, 83).trim()) + "," +
isSpace(input.substring(83, 89).trim()) + "," + isSpace(input.substring(89, 94).trim()) + "," +
isSpace(input.substring(94, 102).trim()) + "," +
isSpace(input.substring(102, 103).trim()) + ","+"TO_DATE("+isSpace(input.substring(103,113).trim())+",'YYYY-MM-DD')"+"," +isSpace(input.substring(113, 125).trim()) + "," + isSpace(input.substring(125, 128).trim()) + "," +
isSpace(input.substring(131, 133).trim()) + "," + isSpace(input.substring(133, 135).trim()) + "," + isSpace(input.substring(135, 136).trim()) + "," +
isSpace(input.substring(136, 137).trim()) + "," + isSpace(input.substring(137, 142).trim()) + "," + isSpace(input.substring(142, 147).trim()) + "," +
isSpace(input.substring(147, 148).trim()) + "," + isSpace(input.substring(149, 159).trim()) + "," +
isSpace(input.substring(159, 160).trim()) + "," + isSpace(input.substring(160, 175).trim()) + "," + isSpace(input.substring(160, 175).trim()) + "," +
isSpace(input.substring(190, 220).trim()) +"," +"TO_DATE("+isSpace(input.substring(216, 220)+"-"+input.substring(220, 222)+"-"+input.substring(222, 224))+",'YYYY-MM-DD')"+","+
"TO_DATE("+isSpace(input.substring(216, 220)+"-"+input.substring(220, 222)+"-"+input.substring(222, 224))+",'YYYY-MM-DD')"+","+ isSpace(input.substring(226,227).trim()) + "," + isSpace(input.substring(231,236).trim()) + "," +
isSpace(input.substring(242, 245).trim()) + "," + isSpace(input.substring(245,275).trim()) + "," + isSpace(input.substring(275,280).trim()) + "," +
isSpace(input.substring(280, 290).trim()) + "," + isSpace(input.substring(290,293).trim()) + "," + isSpace(input.substring(293,303).trim()) + "," +
isSpace(input.substring(303, 314).trim()) + "," +
isSpace(input.substring(313,316).trim()) + //need check
"," + isSpace(input.substring(317,337).trim()) + "," +
isSpace(input.substring(337, 422).trim()) +
"," + isSpace(input.substring(422,433).trim()) + "," + isSpace(input.substring(433,443).trim())+","+isSpace(input.substring(22, 39).trim())+
")");
sql.append('\n');
}
/*if (batchCounter % 500 == 0) {
System.out.println("sql:::::::::::"+sql);
int executeUpdate = em.createNativeQuery(sql.toString()).executeUpdate();
System.out.println("executeUpdateexecu:::"+executeUpdate);
em.flush();
insertionCounter += executeUpdate;
System.out.println("insertionCounter::::"+insertionCounter);
sql.setLength(0);
System.out.println("SQL");
System.out.println("initSql"+initSql);
sql.append(initSql);
}*/
batchCounter++;
}
sql.append(" SELECT * FROM dual");
int executeUpdate = em.createNativeQuery(sql.toString()).executeUpdate();
em.flush();
insertionCounter += executeUpdate;
System.out.println("Check Rows in file::"+(batchCounter-1)+" Insertion counter::"+insertionCounter);
if((batchCounter-1)==insertionCounter){
detailFileObj = new DetailFileObject(FileName, "DETAIL", (batchCounter-1), "SUCCESS");
}
else {
detailFileObj = new DetailFileObject(FileName, "DETAIL", (batchCounter-1), "FAILED");
}
} catch (IOException e) {
e.printStackTrace();
}
But once I am running the code, if there are 40-50 lines, it is working fine.But it is more than that I am getting exception. Can anyone share me proper approach along with running code, so that I can use it.
Thanks
First, DO NOT use string concatenation to build a SQL statement with text values from outside. It leaves you susceptible to SQL Injection attacks, and may cause SQL syntax errors. Use a PreparedStatement with ? parameter markers instead.
Second, if you're inserting a lot of records, use JDBC batching.
Here is an example of how you would use it:
String sql = "INSERT INTO MyTable" +
" ( Col1, Col2, Col3, Col4, Col5, Col6, Col7 )" +
" VALUES (?,?,SYSDATE,?,?,?,?)";
try (PreparedStatement stmt = conn.prepareStatement(sql)) {
char[] line = new char[456];
int batchSize = 0;
while (br.read(line) > 0) {
String input = new String(line);
int detailFileId = interfaceFileSequence();
stmt.setInt (1, detailFileId);
stmt.setInt (2, interfaceHeaderId);
stmt.setInt (3, interfaceRunId);
stmt.setString(4, toString(input.substring(0, 2)));
stmt.setDate (5, toDate(input.substring(2, 12)));
stmt.setString(6, toString(input.substring(12, 22)));
// ...
stmt.addBatch();
if (++batchSize == 1000) {
stmt.executeBatch();
batchSize = 0;
}
}
if (batchSize != 0) {
stmt.executeBatch();
}
} catch (IOException e) {
e.printStackTrace();
}
The above code uses the following helper methods to keep code DRY:
private static String toString(String text) {
String trimmed = text.trim();
return (trimmed.isEmpty() ? null : trimmed);
}
private static java.sql.Date toDate(String text) {
String trimmed = text.trim();
return (trimmed.isEmpty() ? null : java.sql.Date.valueOf(LocalDate.parse(trimmed)));
}
I have the following code and when it executes on the record being UPDATED, the record actually ends up being DELETED from the database table. This is an Oracle Database. Very odd. So far, I have been unable to pull any logging information from the database server that might explain what is happening on a lower level. The System.out.println statements that I included show the output that I am expecting at the moment they execute, yet, when I search for the record both via the java application itself, and then later via the SQL Developer tool - looking at the table itself, the actual data is gone. This is the java code that I have. Any thoughts, ideas, or answers as to how and why an update would perform a delete, instead of just an update is appreciated, thanks. Please let me know if there is anything else you might need that I have not properly provided, thanks.
Class.forName("oracle.jdbc.driver.OracleDriver");
.
.
.
if(PBupdate.equals("updaterecords")) {
try {
ArrayList resultsArray = (ArrayList) session.getAttribute("searchresults");
int pos = 0;
String ID = "";
System.out.println("updatestaff : pos = " + pos + " resultsArray.size() = " + resultsArray.size());
while (pos < resultsArray.size()) {
System.out.println("updatestaff : pos = " + pos);
System.out.println("updatestaff : resultsArray.get(pos + 1) = " + resultsArray.get(pos + 1));
ID = "" + resultsArray.get(pos + 0) + "";
Last_Name = "" + resultsArray.get(pos + 1) + "";
First_Name = "" + resultsArray.get(pos + 2) + "";
Middle = "" + resultsArray.get(pos + 3) + "";
Phone = "" + resultsArray.get(pos + 4) + "";
Dept_Code = "" + resultsArray.get(pos + 5) + "";
Email = "" + resultsArray.get(pos + 6) + "";
Title = "" + resultsArray.get(pos + 7) + "";
Fax_Code = "" + resultsArray.get(pos + 8) + "";
Loc_Code = "" + resultsArray.get(pos + 9) + "";
System.out.println("updatestaff : ID = " + ID);
System.out.println("updatestaff : Last_Name = " + Last_Name);
System.out.println("updatestaff : First_Name = " + First_Name);
System.out.println("updatestaff : Middle = " + Middle);
System.out.println("updatestaff : Phone = " + Phone);
System.out.println("updatestaff : Dept_Code = " + Dept_Code);
System.out.println("updatestaff : Email = " + Email);
System.out.println("updatestaff : Title = " + Title);
System.out.println("updatestaff : Fax_Code = " + Fax_Code);
System.out.println("updatestaff : Loc_Code = " + Loc_Code);
stmt = conn.prepareStatement("UPDATE STAFFDIR SET Last_Name = ?, First_Name = ?, Middle = ?, Phone = ?, Dept_Code = ?, Email = ?, Title = ?, Fax_Code = ?, Loc_Code = ? WHERE ID = ?");
stmt.setString(1, Last_Name);
stmt.setString(2, First_Name);
stmt.setString(3, Middle);
stmt.setString(4, Phone);
Integer DC = Integer.valueOf(Dept_Code);
System.out.println("updatestaff: Dept_Code = " + Dept_Code);
stmt.setInt(5, DC);
stmt.setString(6, Email);
stmt.setString(7, Title);
if ((Fax_Code == null) || (Fax_Code.equals("")) || (Fax_Code.equals("null"))) {
Fax_Code = "999";
}
System.out.println("updatestaff: Fax_Code = " + Fax_Code);
Integer FC = Integer.valueOf(Fax_Code);
stmt.setInt(8, FC);
Integer LC = Integer.valueOf(Loc_Code);
System.out.println("updatestaff: Loc_Code = " + Loc_Code);
stmt.setInt(9, LC);
stmt.setString(10, ID);
int rc = stmt.executeUpdate();
System.out.println("update : rc = " + rc);
SQLWarning warning = conn.getWarnings();
while (warning != null)
{
System.out.println(warning.getMessage());
warning = warning.getNextWarning();
}
pos = pos + 14;
}
response.sendRedirect(response.encodeRedirectURL("/phonebk/updatestaff.jsp"));
} // End of try
I've got this error from my jdbc driver. I don't know why, tho.
Here's the corresponding code:
try {
String colNames = " ";
for (int i=0; i<cols.size(); i++) {
if (i == cols.size()-1) {
colNames += cols.get(i);
} else if (i<cols.size()) {
colNames += cols.get(i)+", ";
}
}
String colValues = " ";
for (int i=0; i<values.size(); i++) {
if (i == values.size()-1) {
colValues += values.get(i);
} else if (i<values.size()) {
colValues += values.get(i) + ", ";
}
}
System.out.println(
"INSERT INTO `" + tableName + "` (" + colNames + ") VALUES (" + colValues + ") "
);
//System.out.println(kerdojel);
PreparedStatement pst = connHandler.conn.prepareStatement
("INSERT INTO `" + tableName + "` (" + colNames + ") VALUES (" + colValues + ") ");
pst.executeUpdate();
pst.close();
}
"values" and "cols" are ArrayLists that contains the data from the JTable.
cols are the Column names and values are the cell values.
The output for the Sysout:
INSERT INTO `TableOne` ( nev, kor, lakhely) VALUES ( asd, 1, asd)
The error:
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown column 'asd' in 'field list'
That is not how the PreaparedStatement was intended to be used. When you use a PreparedStatement, you can specify the values by using one of the "set" methods.
Here is an example:
String colNames = " ";
String colValues = " ";
for (int i=0; i<cols.size(); i++) {
if(i!=0){
colNames += ", ";
colValues += ", ";
}
colNames += cols.get(i);
colValues += "?";
}
try (PreparedStatement pst = connHandler.conn.prepareStatement("INSERT INTO `" + tableName + "` (" + colNames + ") VALUES (" + colValues + ") ");){
for (int i = 0; i < values.size(); i++) {
pst.setString(i+1,values.get(i));
}
pst.executeUpdate();
}
You should use the appropriate "set" method based on the column's data type (setInt(...), setDate(...), etc.). You can find more details here
Try to use command like these one to check problem existed.
PREPARE mycmd FROM 'INSERT INTO TableOne(nev,kor) VALUES (?, ?)'
I have a method getstaffinfo, which has 3 parameter (var_1, connection, filewriter fw), the var_1 value is read from a text file. So the method will be called as many times based on all the var_1 value passed from text file . approx ( 15000)
public static String getstaffid(String var_1, Connection connection,
FileWriter fw) throws SQLException, Exception
// Create a statement
{
String record = null;
ResultSet rs = null;
Statement stmt = connection.createStatement();
boolean empty = true;
try {
rs = stmt
.executeQuery("select username, firstname, lastname, middlename, street, city, stateorprovince, ziporpostalcode, countryorregion, fax, phone, extension, mobile, pager, title, primaryemail, secondaryemail, officename, description, comments, suspendeddate, userdata, employeeid, createuser, updateuser, createdate, updatedate, employeetype, servicedeskticketnumber, startdate, enddate, manager, businessapprover, technicalapprover, delegate, location, jobcodes, customproperty1, customproperty2, customproperty3, customproperty4, customproperty5, customproperty6, customproperty7, customproperty8, customproperty9, customproperty10 from globalusers where username = '"+ var_1 + "'");
ResultSetMetaData metaData = rs.getMetaData();
int columns = metaData.getColumnCount();
ArrayList<String> records = new ArrayList<String>();
while (rs.next()) {
empty = false;
//record = rs.getString(1) + " " + rs.getString(2) + " " + rs.getString(3) + " " + rs.getString(4) + " " + rs.getString(5) + " " + rs.getString(6) + " " + rs.getString(7) + " " + rs.getString(8) + " " + rs.getString(9) + " " + rs.getString(10) + " " + rs.getString(11) + " " + rs.getString(12) + " " + rs.getString(13) + " " + rs.getString(14) + " " + rs.getString(15) + " " + rs.getString(16) + " " + rs.getString(17) + " " + rs.getString(18) + " " + rs.getString(19) + " " + rs.getString(20) + " " + rs.getString(21) + " " + rs.getString(22) + " " + rs.getString(23) + " " + rs.getString(24) + " " + rs.getString(25) + " " + rs.getString(26) + " " + rs.getString(27) + " " + rs.getString(28) + " " + rs.getString(29) + " " + rs.getString(30) + " " + rs.getString(31) + " " + rs.getString(32) + " " + rs.getString(33) + " " + rs.getString(34) + " " + rs.getString(35) + " " + rs.getString(36) + " " + rs.getString(37) + " " + rs.getString(38) + " " + rs.getString(39) + " " + rs.getString(40) + " " + rs.getString(41) + " " + rs.getString(42) + " " + rs.getString(43) + " " + rs.getString(44) + " " + rs.getString(45) + " " + rs.getString(46) + " " + rs.getString(47);
for (int i = 1; i <= columns; i++) {
String value = rs.getString(i);
records.add(value);
}
for (int j = 0; j < records.size(); j++) {
record = records.get(j) + ",";
}
fw.append(record);
}
/*fw.append(rs.getString(1));
fw.append(',');
fw.append(rs.getString(2));
fw.append(',');
fw.append(rs.getString(3));
fw.append('\n'); */
} finally {
fw.flush();
rs.close();
stmt.close();
}
return record;
}
As you can see, am executing a query for 47 values, which could be null or it can have some value.
Then i iterate through this 47 column, take the value and store it to an array list. Then i iterate the array list and write all the values to the string record with comma seperated value. Which is written to a csv file.
But it does not work fine. Any inputs would be appreciated...
You may have already solved the problem. Just let you know that I tried to use your code just now and found the issue was here:
record = records.get(j) + ",";
You should use something like this:
record = record + records.get(j) + ",";
Also change String to StringBuffer will improve the performance.
You didn't write the exact problem you face, but there is one for sure: you never write a line break into the file, so all data gets in one line.
while (rs.next()) {
... // your code, with the for loops
fw.append(record); //writing out the line, from your code
fw.append("\r\n"); //line break -- add this line
} //this is the end of the "while(rs.next())" loop
...
I am having a small problem while I try to insert some data in my database (MS - SQL).
I am using this code (I just copy-paste all of it, so someone might understand smth that I don't)
try
{
Connection connection = null;
Statement Statement = null;
ResultSet ResultSet = null;
String query = "";
// jdbc:sqlserver://[serverName[\instanceName][:portNumber]][;property=value[;property=value]]
String host = "jdbc:sqlserver://server;databaseName=db";
String username = "user";
String password = "pass";
Statement = connection.createStatement();
DateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
Calendar cal = Calendar.getInstance();
DateFormat dateFormat1 = new SimpleDateFormat("dd.MM.yyyy");
Calendar cal1 = Calendar.getInstance();
for (int k = 0; k < idArray.length ; k ++ )
{
query = "INSERT INTO Companies (ID,Company,Company2,Company3,Country,ZIP,City,Street," +
"Phone,Fax,Email,Internet,CustomerNo,AccountMngr,Icon,AddressSource," +
"UserDefined1,UserDefined2,CreatedOn,CreatedBy,ColectedInformation) " +
" VALUES ('" + UUID.randomUUID() + "','" + company_nameArray[k]
+ "','" + company_name2Array[k] + "','" + company_name3Array[k] + "','DE','"
+ zipArray[k] + "','" + cityArray[k] + "','" + streetArray[k] + "','"
+ phone_Array[k] + "','" + faxArray[k] + "','" + emailArray[k] + "','"
+ internetArray[k] + "','" + customer_noArray[k]
+ "','d','131','60','Baufinder Import','Import Datum "
+ dateFormat1.format(cal1.getTime()) + "','"
+ dateFormat1.format(cal1.getTime()) + "','d','"
+ "cxcxvcx" +
"')";
ResultSet = Statement.executeQuery(query);
}
connection.close();
}
catch(NullPointerException e)
{
System.out.println("NullPointerException");
}
catch ( SQLException err )
{
System.out.println( "SQL Exception:" ) ;
// Loop through the SQL Exceptions
while( err != null )
{
System.out.println( "State : " + err.getSQLState() ) ;
System.out.println( "Message: " + err.getMessage() ) ;
System.out.println( "Error : " + err.getErrorCode() ) ;
err = err.getNextException() ;
}
}
catch( Exception e )
{
System.out.println( "There is a problem with " + e ) ;
}
}
and I am getting this error:
SQL Exception: State : null
Message: There was no result set returned by the statement.
Error : 0
I don't understand the problem with the result test. I have been using it a lot, since I try to read the data from another database and now I try to insert them to another.
Furthermore, all of these tables do not include null values.
Thank you in advance:)
Use Statement#executeUpdate(String) instead of Statement#execute(query);
for (int k = 0; k < idArray.length ; k ++ )
{
query = "INSERT INTO Companies (ID,Company,Company2,Company3,Country,ZIP,City,Street," +
"Phone,Fax,Email,Internet,CustomerNo,AccountMngr,Icon,AddressSource," +
"UserDefined1,UserDefined2,CreatedOn,CreatedBy,ColectedInformation) " +
" VALUES ('" + UUID.randomUUID() + "','" + company_nameArray[k]
+ "','" + company_name2Array[k] + "','" + company_name3Array[k] + "','DE','"
+ zipArray[k] + "','" + cityArray[k] + "','" + streetArray[k] + "','"
+ phone_Array[k] + "','" + faxArray[k] + "','" + emailArray[k] + "','"
+ internetArray[k] + "','" + customer_noArray[k]
+ "','d','131','60','Baufinder Import','Import Datum "
+ dateFormat1.format(cal1.getTime()) + "','"
+ dateFormat1.format(cal1.getTime()) + "','d','"
+ "cxcxvcx" +
"')";
int updatedRowCount = Statement.executeUpdate(query);
}
Notes:
Please consider using PreparedStatement to prevent possibility of SQL Injection if some data directly input by User.
Try to use batch insert if you need to improve performance of inserting.