Basically i have to read a csv file and perform some validation.
If duplicate record is found i've to delete the previous record and insert the latest 1.
The file contains about 100k records. I'm not sure what I'm doing wrongly but it's taking way too long to load the data.
public static ArrayList<BootstrapMessage> loadLocation(File file) {
ArrayList<BootstrapMessage> errors = new ArrayList<BootstrapMessage>();
CSVReader reader = null;
Connection conn = null;
Connection conn2 = null;
PreparedStatement pstmt = null;
PreparedStatement ps = null;
try {
conn = ConnectionManager.getConnection();
conn2 = ConnectionManager.getConnection();
conn.setAutoCommit(false);
pstmt = conn.prepareStatement(INSERT_LOCATION);
ps = conn2.prepareStatement("delete from location where `timestamp` = ? AND mac_address = ?");
reader = new CSVReader(new InputStreamReader(new FileInputStream(file), "UTF-8"));
reader.readNext();//header
String[] record = reader.readNext();
int counter = 2;//starting from line 2. Line 1 is header
int validRecords = 0;
while (record != null) {
ArrayList<String> message = null;
//try {
message = ValidatorUtil.validateLocation(record, file.getName(), counter);
if (message != null) {//contains error
errors.add(new BootstrapMessage(file.getName(), counter, message));
} else {//valid record
String key = record[0] + record[1];
if (locations.containsKey(key)) {//duplicate found.
pstmt.executeBatch();
message = new ArrayList<String>();
message.add("duplicate row");
errors.add(new BootstrapMessage(file.getName(), locations.get(key), message));
//delete record from database
ps.setTimestamp(1, Timestamp.valueOf(record[0]));
ps.setString(2, record[1]);
ps.executeUpdate();
//inserting the latest record
pstmt.setTimestamp(1, Timestamp.valueOf(record[0]));
pstmt.setString(2, record[1]);
pstmt.setInt(3, Integer.parseInt(record[2]));
pstmt.addBatch();
if (validRecords % 2000 == 0) {
pstmt.executeBatch();
}
} else {
pstmt.setTimestamp(1, Timestamp.valueOf(record[0]));
pstmt.setString(2, record[1]);
pstmt.setInt(3, Integer.parseInt(record[2]));
pstmt.addBatch();
validRecords++;
if (validRecords % 2000 == 0) {
pstmt.executeBatch();
}
}
}
if (validRecords > 0) {
pstmt.executeBatch();
conn.commit();
}
record = reader.readNext();
counter++;
}
System.out.println("valid location records = " + validRecords);
//numOfValidRecords.put(fileName, validRecords);
if (!errors.isEmpty()) {
return errors;
}
} catch (FileNotFoundException ex) {
Logger.getLogger(LocationDAO.class.getName()).log(Level.SEVERE, null, ex);
} catch (IOException ex) {
Logger.getLogger(LocationDAO.class.getName()).log(Level.SEVERE, null, ex);
} catch (SQLException ex) {
Logger.getLogger(LocationDAO.class.getName()).log(Level.SEVERE, null, ex);
} finally {
if (reader != null) {
try {
reader.close();
} catch (IOException ex) {
Logger.getLogger(LocationDAO.class.getName()).log(Level.SEVERE, null, ex);
}
}
ConnectionManager.close(conn2, ps);
ConnectionManager.close(conn, pstmt);
}
return null;
}
Why don't you use native database loaders to do the job?
Or I would first insert all the records into staging and then do the duplicate removals by using the database tools, either SQL or some database procedure. This way it has to be faster.
Related
I'm trying to get the ID of an insert row which is auto_increment in MySQL. I'm using the parameter Statement.RETURN_GENERATED_KEYS in the prepareStatement method and trying to obtain it via the getGeneratedKeys method. Somehow, it is not retrieving the ID.
It should be noted that the insertion is done, but since the ID is not obtained, I cannot perform the other inserts and that is why it gives me the error "MySQL ERROR 1452", because an insert is being made later with the ID 0 that does not exist.
public int insertar(Compra objC, ArrayList<DetalleCompra> listaDetalle) {
try {
conexion = MySQLConexion.conexion();
conexion.setAutoCommit(false);
procedimientoAlmacenado = conexion.prepareStatement("{call insertar_compra (?, ?)}", Statement.RETURN_GENERATED_KEYS);
fecha = FORMATO_FECHA.parse(objC.getFecha());
fechaSQL = new java.sql.Date(fecha.getTime());
procedimientoAlmacenado.setDate(1, fechaSQL);
procedimientoAlmacenado.setInt(2, objC.getIdCliente());
resultado = procedimientoAlmacenado.executeUpdate();
int idCompra = 0;
ResultSet idGenerada = procedimientoAlmacenado.getGeneratedKeys();
if (idGenerada.next()) {
idCompra = idGenerada.getInt(1);
}
if (resultado != 0) {
for (DetalleCompra x : listaDetalle) {
procedimientoAlmacenado = conexion.prepareCall("{call insertar_detalle_compra (?, ?, ?)}");
procedimientoAlmacenado.setInt(1, idCompra);
procedimientoAlmacenado.setInt(2, x.getIdProducto());
procedimientoAlmacenado.setInt(3, x.getCantidad());
resultado = procedimientoAlmacenado.executeUpdate();
procedimientoAlmacenado = conexion.prepareCall("{disminuir_stock (?, ?)}");
procedimientoAlmacenado.setInt(1, x.getIdProducto());
procedimientoAlmacenado.setInt(2, x.getCantidad());
resultado = procedimientoAlmacenado.executeUpdate();
}
}
conexion.commit();
} catch (Exception e) {
System.out.println(e.getMessage());
try {
conexion.rollback();
} catch (SQLException e1) {
e1.printStackTrace();
}
} finally {
try {
if (procedimientoAlmacenado != null)
procedimientoAlmacenado.close();
if (conexion != null)
conexion.close();
} catch (SQLException e) {
System.out.println(e.getMessage());
}
}
return resultado;
}
I also tried to create a variable:
String[] generatedId = { "ID" };
and place it in the method prepareStatement:
procedimientoAlmacenado = conexion.prepareStatement("{call insertar_compra (?, ?)}", generatedId);
However, it doesn't work either. I'm suspecting that it has to do with the fact that I'm doing the statement inside a stored procedure, or with setAutoCommit(false), but I can't think of anything else.
Ok so what I did was make the stored procedure return me the last inserted ID:
delimiter $$
create procedure insertar_compra (in fec datetime, in client_id int, out id int)
begin
insert into compra values (null, fec, client_id, 'P');
set id = LAST_INSERT_ID();
end; $$
delimiter ;
And then I got it with the registerOutParameter() method:
public int insertar(Compra objC, ArrayList<DetalleCompra> listaDetalle) {
try {
conexion = MySQLConexion.conexion();
conexion.setAutoCommit(false);
procedimientoAlmacenado = conexion.prepareCall("{call insertar_compra (?, ?, ?)}");
fecha = FORMATO_FECHA.parse(objC.getFecha());
fechaSQL = new Timestamp(fecha.getTime());
procedimientoAlmacenado.setTimestamp(1, fechaSQL);
procedimientoAlmacenado.setInt(2, objC.getIdCliente());
procedimientoAlmacenado.registerOutParameter(3, Types.INTEGER);
resultado = procedimientoAlmacenado.executeUpdate();
int idCompra = 0;
idCompra = procedimientoAlmacenado.getInt(3);
if (resultado != 0) {
for (DetalleCompra x : listaDetalle) {
procedimientoAlmacenado = conexion.prepareCall("{call insertar_detalle_compra (?, ?, ?)}");
procedimientoAlmacenado.setInt(1, idCompra);
procedimientoAlmacenado.setInt(2, x.getIdProducto());
procedimientoAlmacenado.setInt(3, x.getCantidad());
resultado = procedimientoAlmacenado.executeUpdate();
procedimientoAlmacenado = conexion.prepareCall("{call disminuir_stock (?, ?)}");
procedimientoAlmacenado.setInt(1, x.getIdProducto());
procedimientoAlmacenado.setInt(2, x.getCantidad());
resultado = procedimientoAlmacenado.executeUpdate();
}
}
conexion.commit();
} catch (Exception e) {
System.out.println(e.getMessage());
try {
conexion.rollback();
} catch (SQLException e1) {
e1.printStackTrace();
}
} finally {
try {
if (procedimientoAlmacenado != null)
procedimientoAlmacenado.close();
if (conexion != null)
conexion.close();
} catch (SQLException e) {
System.out.println(e.getMessage());
}
}
return resultado;
}
Thanks to Mark Rotteveel for the suggestion!
I have code, where I have single quote or APOSTROPHE in my search
I have database which is having test table and in name column of value is "my'test"
When running
SELECT * from test WHERE name = 'my''test';
this works fine
If I use the same in a Java program I am not getting any error or any result
But If I give the name with only single quote then it works
SELECT * from test WHERE name = 'my'test';
Could you please help me out to understand.
Java code is
Connection con = null;
PreparedStatement prSt = null;
try {
Class.forName("oracle.jdbc.driver.OracleDriver");
con = DriverManager.
getConnection("jdbc:oracle:thin:#localhost:1521:orcl"
,"user","pwd");
String query = "SELECT * from "
+ "WHERE name = ? ";
prSt = con.prepareStatement(query);
String value = "my'mobile";
char content[] = new char[value.length()];
value.getChars(0, value.length(), content, 0);
StringBuffer result = new StringBuffer(content.length + 50);
for (int i = 0; i < content.length; i++) {
if (content[i] == '\'')
{
result.append("\'");
result.append("\'");
}
else
{
result.append(content[i]);
}
}
prSt.setObject(1, result.toString());
int count = prSt.executeUpdate();
System.out.println("===============> "+count);
} catch (ClassNotFoundException e) {
e.printStackTrace();
} catch (SQLException e) {
e.printStackTrace();
} finally{
try{
if(prSt != null) prSt.close();
if(con != null) con.close();
} catch(Exception ex){}
}
You don't have to escape anything for the parameter of a PreparedStatement
Just use:
prSt = con.prepareStatement(query);
prSt.setString("my'mobile");
Additionally: if you are using a SELECT statement to retrieve data, you need to use executeQuery() not executeUpdate()
ResultSet rs = prst.executeQuery();
while (rs.next())
{
// process the result here
}
You might want to go through the JDBC tutorial before you continue with your project: http://docs.oracle.com/javase/tutorial/jdbc/index.html
I am getting null pointer exception again and again while getting value from database table.....why its giving me null pointer exception?
Here is my code :
private HashSet getPlayerList() {
HashSet hs = new HashSet();
String soccername = "";
try {
conn = ConnectionProvider.getConnection();
rs = null;
pstmt = null;
String sql = "Select * from Players where deleted = false";
if (conn != null) {
pstmt = conn.prepareStatement(sql);
rs = pstmt.executeQuery();
while (rs.next()) {
soccername = rs.getString("soccername").trim();// this line giving exception
if (soccername == null || soccername.isEmpty()) {
soccername = rs.getString("name").trim();
}
hs.add(soccername);
}
}
} catch (NamingException ex) {
System.out.println(ex);
} catch (SQLException ex) {
System.out.println(ex);
} finally {
try {
pstmt.close();
rs.close();
conn.close();
} catch (SQLException ex) {
System.out.println(ex);
}
}
return hs;
}
that is wrong:
soccername = rs.getString("soccername").trim();// this line giving exception
if (soccername == null || soccername.isEmpty()) {
if rs.getString("soccername") returns null it must leads to a NPE because it is followed by the function trim().
better is:
soccername = rs.getString("soccername");
if (soccername == null || soccername.trim().isEmpty()) {
It should be:
soccername = rs.getString("soccername");
if (soccername == null || soccername.isEmpty()) {
soccername = rs.getString("name").trim();
} else {
//now you are sure that soccername is not null
soccername = soccername.trim();
}
I'm importing database values from a .csv file. However, the last row in the .csv is actually the totals of some columns, and my importer updates the database with that being the last row and that is undesirable. And that messes up the queries that I'll do later with it. Please help how can I prevent this?
My code of my .csv importer:
try
{
BufferedReader br=new BufferedReader(new FileReader("v.csv"));
String line;
while((line=br.readLine())!=null)
{
String[]value = line.split(",");
String sql = "INSERT into main ([Ticket #], Status, Priority, Department, [Account Name]) "
+ "values ('"+value[0]+"','"+value[1]+"','"+value[2]+"','"+value[3]+"','"+value[4]+"')";
System.out.println(sql);
PreparedStatement pst = null;
try
{
pst = db.prepareStatement(sql);
pst.executeUpdate();
}finally{
if(pst != null){
pst.close();
}
}
}
br.close();
}
catch(Exception e)
{
JOptionPane.showMessageDialog(null, e);
}
Try this:
try
{
BufferedReader br=new BufferedReader(new FileReader("v.csv"));
ArrayList<String> lines = new ArrayList<String>();
String line;
while((line=br.readLine())!=null)
{
lines.Add(line);
}
br.close();
if (lines.size() > 1) { // be sure that the last row exists (prevents indexoutexception)
for (int i = 0; i < lines.size() - 1; i++) { // the - 1 excludes the last row
String[]value = lines.get(i).split(",");
String sql = "INSERT into main ([Ticket #], Status, Priority, Department, [Account Name]) "
+ "values ('"+value[0]+"','"+value[1]+"','"+value[2]+"','"+value[3]+"','"+value[4]+"')";
System.out.println(sql);
PreparedStatement pst = null;
try
{
pst = db.prepareStatement(sql);
pst.executeUpdate();
} finally {
if(pst != null) {
pst.close();
}
}
}
}
}
catch(Exception e)
{
JOptionPane.showMessageDialog(null, e);
}
If possible you can use the Apache commons IO ReversedLinesFileReader and read the CSV from the end. Then you can skip the first line.
Try this:
try
{
BufferedReader br=new BufferedReader(new FileReader("v.csv"));
String line;
String firstFlag =true;
while((line=br.readLine())!=null)
{
if (!firstFlag){
String sql = "INSERT into main ([Ticket #], Status, Priority, Department, [Account Name]) "
+ "values ('"+value[0]+"','"+value[1]+"','"+value[2]+"','"+value[3]+"','"+value[4]+"')";
System.out.println(sql);
PreparedStatement pst = null;
try
{
pst = db.prepareStatement(sql);
pst.executeUpdate();
}finally{
if(pst != null){
pst.close();
}
}
}
String[]value = line.split(",");
firstFlag=false;
}
br.close();
}
catch(Exception e)
{
JOptionPane.showMessageDialog(null, e);
}
I have a oracle(10.2) PLSQL procedure which fetches 15 records from a table in a sysrefcursor.
I then pass this cursor to a java class as a resultset. This java class is loaded to oracle.
Driver name : Oracle JDBC driver
Driver Version : 10.2.0.1.0
Driver Major Version : 10
Driver Minor Version : 2
Observations:
1 Inside the java class, when I iterate through the resultset I get only the first 10 records.
2 If the cursor fetched (20 or more records) or (10 or less) I could get all the records while iterating the resultset.
3 I found that the default fetchsize for the resultset is 10. If I change the fetchSize to 5, and the cursor fetches 8 records, I could get the first 5 records while iterating the resultset.
4 If the cursor fetched (10 or more records) or (5 or less) I could get all the records while iterating the resultset.
5 If I change the resultset fetchSize to 1, I could get all the records in the resultset no matter how many records are fetched by the cursor.
Why is the resultset behaving weirdly?
public static BLOB createZip(BLOB prevBlob, String outPrefix, ResultSet entries, ResultSet rs, Long[] resultRows) throws Exception
{
OracleConnection conn = null;
BLOB retBLOB = null;
int page = 1;
int curRow = 0;
long totalRows = 0;
try
{
conn = (OracleConnection) new OracleDriver().defaultConnection();
ArrayList entryList = loadEntries(entries);
retBLOB = BLOB.createTemporary(conn, true, BLOB.DURATION_SESSION);
retBLOB.open(BLOB.MODE_READWRITE);
OutputStream bOut = retBLOB.setBinaryStream(0L);
ZipOutputStream zipOut = new ZipOutputStream(bOut);
PrintStream out = new PrintStream(zipOut);
zipOut.putNextEntry(new ZipEntry(outPrefix + "-p" + page + ".csv"));
writeHeader(out, entryList);
while (rs.next())
{
curRow++;
totalRows++;
if (curRow >= maxPageSize)
{
zipOut.closeEntry();
page++;
zipOut.putNextEntry(new ZipEntry(outPrefix + "-p" + page + ".csv"));
writeHeader(out, entryList);
curRow = 0;
}
for (int i = 0; i < entryList.size(); i++)
{
Entry e = (Entry) entryList.get(i);
if (i != 0)
{
out.print(",");
}
if (e.isEscape())
out.print("\"" + escapeExcel(rs.getString(e.getColumn())) + "\"");
else
out.print("\"" + emptyExcel(rs.getString(e.getColumn())) + "\"");
}
out.println();
}
if (totalRows == 0)
{
out.println("\"No Entries Found\"");
}
resultRows[0] = new Long(totalRows);
out.flush();
zipOut.closeEntry();
if (prevBlob != null)
{
byte[] buf = new byte[1024];
InputStream bIn = prevBlob.binaryStreamValue();
ZipInputStream zipIn = new ZipInputStream(bIn);
ZipEntry inEntry = zipIn.getNextEntry();
while (inEntry != null)
{
zipOut.putNextEntry(new ZipEntry(inEntry.getName()));
int len;
while ((len = zipIn.read(buf)) > 0) {
out.write(buf, 0, len);
}
inEntry = zipIn.getNextEntry();
}
zipIn.close();
try
{
prevBlob.freeTemporary();
}
catch (SQLException e) { }
}
zipOut.close();
retBLOB.close();
return retBLOB;
}
catch (Exception sex)
{
if (retBLOB != null)
{
try
{
retBLOB.freeTemporary();
}
catch (SQLException e) { }
}
throw sex;
}
finally
{
try { entries.close(); } catch (SQLException sex) { }
try { rs.close(); } catch (SQLException sex) { }
try
{
if (conn != null || !conn.isClosed())
{
conn.close();
}
}
catch (SQLException ex)
{
ex.printStackTrace();
}
}
}
There is a workaround. I fetched the column index before while(rs.next()).
The below snippet works absolutely fine for me. But I still fail to understand why resultSet.getString(columnName); failed inside while(rs.next()).
ArrayList entryList = loadEntries(entries);
int[] colIndx = new int[entryList.size()];
ResultSetMetaData rsmd = rs.getMetaData();
int numberOfColumns = rsmd.getColumnCount();
for (int i = 0; i < entryList.size(); i++){
Entry e = (Entry) entryList.get(i);
for (int j = 1; j <= numberOfColumns; j++){
if(rsmd.getColumnName(j).equalsIgnoreCase(e.getColumn()))
colIndx[i] = j;
}
}
try{
while (rs.next()){
for (int i = 0; i < colIndx.length ; i++){
System.out.println("Column Values["+colIndx[i]+"] : "+rs.getString(colIndx[i]));
}
}
}