My application reads a html table and then a script (TableToCSV) converts it into a .csv format. After that I convert that .csv into a sqlite database. After that I run queries on the database. Problem is that upon executing, it shows that SQLITE_BUSY; database file is locked.
Whats the reason of this and how can I fix this?
Here is my code -
final JFileChooser fileDialog = new JFileChooser();
JButton btnInputFile = new JButton("Input File");
btnInputFile.addActionListener(new ActionListener() {
public void actionPerformed(ActionEvent arg0) {
int returnVal = fileDialog.showOpenDialog(rootPane);
if (returnVal == JFileChooser.APPROVE_OPTION) {
java.io.File file = fileDialog.getSelectedFile();
String name = file.getName();
name = name.substring(0, name.lastIndexOf("."));
name += ".html";
File newFile = new File(file.getParentFile(), name);
if (file.renameTo(newFile)) {
try {
TableToCSV tableToCSV = new TableToCSV(newFile, ',', '\"', '#', CSV.UTF8Charset );
System.out.println("action");
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
try
{
BufferedReader br=new BufferedReader(new FileReader("v.csv"));
String line;
while((line=br.readLine())!=null)
{
System.out.println(line);
String[]value = line.split(",");
System.out.println(value.length);
String sql = "INSERT into main ([Ticket #], Status, Priority, Department, [Account Name]) "
+ "values ('"+value[0]+"','"+value[1]+"','"+value[2]+"','"+value[3]+"','"+value[4]+"')";
PreparedStatement pst = DatabaseConnection.ConnectDB().prepareStatement(sql);
pst.executeUpdate();
}
br.close();
}
catch(Exception e)
{
JOptionPane.showMessageDialog(null, e);
}
}
}
});
UPDATE - DatabaseConnection.java
public class DatabaseConnection {
Connection conn = null;
Statement stmt = null;
public static Connection ConnectDB() {
try {
Class.forName("org.sqlite.JDBC");
Connection conn = DriverManager.getConnection("jdbc:sqlite:database.db");
conn.setAutoCommit(true);
return conn;
} catch (Exception e) {
JOptionPane.showMessageDialog(null, e);
return null;
}
}
}
It's probably due to you having multiple open references to the sqlite database.
I'd start by closing your PreparedStatement in a finally block inside your while loop.
PreparedStatement pst = null;
try{
pst = DatabaseConnection.ConnectDB().prepareStatement(sql);
pst.executeUpdate();
}finally{
if(pst != null) {
pst.close();
}
}
You should also close the database connection at the end of everything.
If that doesn't fix it, then please explain more about how you "convert that .csv into a sqlite database"
Related
I am creating a program in Java and I need to compare if data in a csv file is exactly the same that exists on a mysql table?
How can i do that?
For example, i have a table "Suplyers" with columns "Id, Name and Adress".
Thanks
Below is the code that i have that read csv file and that connect to database and shows the data in the table.
public static void le_csv() {
String row;
BufferedReader csvReader = null;
try {
csvReader = new BufferedReader(new FileReader("C:\\Users\\User\\Desktop\\ficheiros\\fornecedores.csv"));
} catch (FileNotFoundException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
try {
while ((row = csvReader.readLine()) != null) {
String[] data = row.split(",");
System.out.println(data[0] + "\t" + data[1] + "\t" + data[2]);
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
try {
csvReader.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
>
>
>
public static void query(){
try {
String url = "jdbc:mysql://127.0.0.1:3306/database";
String user = "user";
String password = "password";
Class.forName("com.mysql.jdbc.Driver");
Connection conn = DriverManager.getConnection(url, user, password);
String sql = "SELECT * FROM SUPLYERS";
Statement st = conn.createStatement();
ResultSet rs = st.executeQuery(sql);
while (rs.next()) {
System.out.println(rs.getString(1) + "\t" + rs.getString(2) + "\t" + rs.getString(3);
}
rs.close();
st.close();
conn.close();
} catch (Exception exc) {
exc.printStackTrace();
}
}
You may collect the CSV data and DB data into lists of String and then compare the lists using equals:
public static List<String> readCsvData() {
List<String> csvData = new ArrayList<>();
// use try-with-resources to auto close reader
try (BufferedReader csvReader = new BufferedReader(new FileReader("C:\\Users\\User\\Desktop\\ficheiros\\fornecedores.csv"))){
String row;
while ((row = csvReader.readLine()) != null) {
String[] data = row.split(",");
row = String.join("\t", data[0], data[1], data[2]);
System.out.println(row);
csvData.add(row);
}
} catch (IOException e) {
e.printStackTrace();
throw new RuntimeException(e);
}
return csvData;
}
public static void initDb() {
try {
Class.forName("com.mysql.jdbc.Driver");
} catch (ClassNotFoundException e) {
e.printStackTrace();
throw new RuntimeException(e);
}
}
public static List<String> readDbData(){
initDb();
String url = "jdbc:mysql://127.0.0.1:3306/database";
String user = "user";
String password = "password";
String sql = "SELECT * FROM SUPLYERS";
List<String> dbData = new ArrayList<>();
// use try-with-resources to auto close SQL connection, etc.
try (Connection conn = DriverManager.getConnection(url, user, password);
Statement st = conn.createStatement();
ResultSet rs = st.executeQuery(sql);
) {
while (rs.next()) {
String row = String.join("\t", rs.getString(1), rs.getString(2), rs.getString(3));
System.out.println(row);
dbData.add(row);
}
} catch (Exception exc) {
exc.printStackTrace();
throw new RuntimeException(exc);
}
return dbData;
}
public static boolean areCsvDataSameAsDb() {
List<String> csvData = readCsvData();
List<String> dbData = readDbData();
return csvData.equals(dbData);
}
Or you can read the data row by row to shortcut the check as soon as any discrepancy is detected.
I'm new to this. I need to make a jsp to upload a CSV file and send the data in it to a mysql database. I made a code in Java to get the latter part done, but not sure how to connect it to the jsp, so when the user press uploads the java code runs.
My java code:
import java.io.*;
import java.sql.*;
public class dbInserter {
public static void main(String[] args) {
String jdbcURL = "jdbc:mysql://localhost:3306/testDb";
String username = "root";
String password = "1234";
String csvFilePath = "C:/Users/Name/Desktop/Attendance.csv";
int batchSize = 20;
Connection connection = null;
try {
connection = DriverManager.getConnection(jdbcURL, username, password);
connection.setAutoCommit(false);
String sql = "INSERT INTO csv (name, `in-time`, `out-time`) VALUES (?, ?, ?)";
PreparedStatement statement = connection.prepareStatement(sql);
BufferedReader lineReader = new BufferedReader(new FileReader(csvFilePath));
String lineText = null;
int count = 0;
lineReader.readLine(); // skip header line
while ((lineText = lineReader.readLine()) != null) {
String[] data = lineText.split(",");
String name = data[0];
String inTime = data[1];
String outTime = data[2];
statement.setString(1, name);
statement.setString(2, inTime);
statement.setString(3, outTime);
statement.addBatch();
if (count % batchSize == 0) {
statement.executeBatch();
}
}
lineReader.close();
// execute the remaining queries
statement.executeBatch();
connection.commit();
connection.close();
} catch (IOException ex) {
System.err.println(ex);
} catch (SQLException ex) {
ex.printStackTrace();
try {
connection.rollback();
} catch (SQLException e) {
e.printStackTrace();
}
}
}
}
I tried using a servlet, but since I have very little experience using it, I ran into many errors. Would really appreciate it if someone can help me with this.
Hello fellow Stackoverflowers!
I am trying to INSERT INTO a database table 'dotcom' 4 columns of 5000 rows of data from a text file in the form [x, w, x, y, z] e.g.
1 google com null null
2 google co uk null
...
and the rows returned remains '1' instead of '5000' for some reason.
The problem is that the rows are not inserted into the database table even though the BufferedReader reads it. How may I solve this problem? E.g. inserting the data into the table with a loop?
Any help would be greatly appreciated!
import java.sql.*;
import java.util.Scanner;
import java.io.*;
public class Database {
public static Connection connectToDatabase(String user, String port,
String database) {
System.out.println("-------- PostgreSQL JDBC Connection Testing ------------");
try {
Class.forName("org.postgresql.Driver");
} catch (ClassNotFoundException e) {
System.out.println("Where is your PostgreSQL JDBC Driver? " + "Include in your library path!");
e.printStackTrace();
}
System.out.println("PostgreSQL JDBC Driver Registered!");
Connection connection = null;
try {
connection = DriverManager.getConnection("jdbc:postgresql://localhost:" + port + "/" + database, user, "doesn't matter!");
} catch (SQLException e) {
System.out.println("Connection Failed! Check output console");
e.printStackTrace();
}
return connection;
}
public static ResultSet executeSelect(Connection connection, String query) {
Statement st = null;
try {
st = connection.createStatement();
} catch (SQLException e) {
e.printStackTrace();
return null;
}
ResultSet rs = null;
try {
rs = st.executeQuery(query);
//st.close();
} catch (SQLException e) {
e.printStackTrace();
return null;
}
return rs;
}
public static void dropTable(Connection connection, String table) {
Statement st = null;
try {
st = connection.createStatement();
st.execute("DROP TABLE " + table);
st.close();
} catch (SQLException e) {
e.printStackTrace();
}
}
public static void createTable(Connection connection,
String tableDescription) {
Statement st = null;
try {
st = connection.createStatement();
st.execute("CREATE TABLE " + tableDescription);
st.close();
} catch (SQLException e) {
e.printStackTrace();
}
}
public static int insertIntoTableFromFile(Connection connection,
String table, String file) {
BufferedReader br = null;
int numRows = 0;
try {
Statement st = connection.createStatement();
String sCurrentLine, brokenLine[], composedLine = "";
br = new BufferedReader(new FileReader("src/TopURLs"));
while ((sCurrentLine = br.readLine()) != null) {
// Insert each line to the DB
brokenLine = sCurrentLine.split("\t");
composedLine = "INSERT INTO dotcom VALUES (";
int i;
for (i = 0; i < brokenLine.length - 1; i++) {
composedLine += "'" + brokenLine[i] + "',";
}
composedLine += "'" + brokenLine[i] + "')";
numRows = st.executeUpdate(composedLine);
//System.out.println(composedLine);
}
} catch (IOException e) {
e.printStackTrace();
} catch (SQLException e) {
e.printStackTrace();
} finally {
try {
if (br != null)
br.close();
} catch (IOException ex) {
ex.printStackTrace();
}
}
return numRows;
}
public static void main(String[] argv) throws SQLException {
/*
Scanner input = new Scanner(System.in);
System.out.println("Please enter your Username:");
String user = input.next();
System.out.println("Please enter your Port ID:");
String port = input.next();
*/
String user = "zbva777";
String port = "28046";
String database = "test";
Connection connection = connectToDatabase(user, port, database);
if (connection != null) {
System.out.println("You made it, take control your database now!");
} else {
System.out.println("Failed to make connection!");
return;
}
// Now we're ready to work on the DB
String query = "SELECT * FROM dotcom";
ResultSet rs = executeSelect(connection, query);
try {
while (rs.next()) {
System.out.print("Column 1 returned ");
System.out.println(rs.getString(1));
}
} catch (SQLException e) {
e.printStackTrace();
}
rs.close();
dropTable(connection, "dotcom");
createTable(connection, "dotcom (rank int primary key, name varchar(5000), type varchar(5000), subtype varchar(5000), subsubtype varchar(5000));");
int rows = insertIntoTableFromFile(connection, "dotcom", "src/TopURLs");
System.out.println(rows + " rows inserted.");
}
}
In each iteration of the loop you override numRows instead of incrementing it with the newly added row. Just replace the = with += and you should be OK:
numRows += st.executeUpdate(composedLine);
// Here ^
That being said, you should really look into PreparedStatements and executing batches.
How can you read & insert rows from a text file containing data in [x, y, z] form for example:
1 google com (separated by tabs)
into a database table (without having to insert every single line manually)?
I'm new at programming!
Thank you for your time and help kind sirs and madams!
import java.sql.*;
import java.util.Scanner;
import java.io.*;
public class Database {
public static Connection connectToDatabase(String user, String port, String database) {
System.out.println("-------- PostgreSQL " + "JDBC Connection Testing ------------");
try {
Class.forName("org.postgresql.Driver");
} catch (ClassNotFoundException e) {
System.out.println("Where is your PostgreSQL JDBC Driver? " + "Include in your library path!");
e.printStackTrace();
}
System.out.println("PostgreSQL JDBC Driver Registered!");
Connection connection = null;
try {
connection = DriverManager.getConnection("jdbc:postgresql://localhost:" + port + "/" + database, user,
"doesn't matter!");
} catch (SQLException e) {
System.out.println("Connection Failed! Check output console");
e.printStackTrace();
}
return connection;
}
public static ResultSet executeSelect(Connection connection, String query) {
Statement st = null;
try {
st = connection.createStatement();
} catch (SQLException e) {
e.printStackTrace();
return null;
}
ResultSet rs = null;
try {
rs = st.executeQuery(query);
st.close();
} catch (SQLException e) {
e.printStackTrace();
return null;
}
return rs;
}
public static void dropTable(Connection connection, String table) {
Statement st = null;
try {
st = connection.createStatement();
st.execute("DROP TABLE " + table);
st.close();
} catch (SQLException e) {
e.printStackTrace();
}
}
public static void createTable(Connection connection, String tableDescription) {
Statement st = null;
try {
st = connection.createStatement();
st.execute("CREATE TABLE " + tableDescription);
st.close();
} catch (SQLException e) {
e.printStackTrace();
}
}
public static int insertIntoTableFromFile(Connection connection, String table, String file) {
BufferedReader br = null;
int numRows = 0;
try {
Statement st = connection.createStatement();
String sCurrentLine, brokenLine[], composedLine = "";
br = new BufferedReader(new FileReader(file));
while ((sCurrentLine = br.readLine()) != null) {
// Insert each line to the DB
brokenLine = sCurrentLine.split("\t");
composedLine = "INSERT INTO dotcom VALUES (";
int i;
for (i = 0; i < brokenLine.length - 1; i++) {
composedLine += "'" + brokenLine[i] + "',";
}
composedLine += "'" + brokenLine[i] + "')";
numRows = st.executeUpdate(composedLine);
}
} catch (IOException e) {
e.printStackTrace();
} catch (SQLException e) {
e.printStackTrace();
} finally {
try {
if (br != null)
br.close();
} catch (IOException ex) {
ex.printStackTrace();
}
}
return numRows;
}
public static void main(String[] argv) throws SQLException, FileNotFoundException {
#SuppressWarnings("resource")
Scanner input = new Scanner(System.in);
System.out.println("Please enter your Username:");
String user = input.next();
System.out.println("Please enter your Port ID:");
String port = input.next();
String database = "test";
Connection connection = connectToDatabase(user, port, database);
Statement st = connection.createStatement();
if (connection != null) {
System.out.println("You made it, take control your database now!");
} else {
System.out.println("Failed to make connection!");
return;
}
// Now we're ready to work on the DB
// read TopURLs file
try {
BufferedReader fileReader = new BufferedReader(new FileReader("TopURLs"));
while (fileReader.readLine() != null) {
st.execute("DROP TABLE IF EXISTS dotcom;");
st.execute("CREATE TABLE dotcom (rank integer PRIMARY KEY, domainName varchar(128), domainType varchar(128));");
st.execute("INSERT INTO dotcom VALUES(1, 'google', 'com');");
//st.execute("INSERT INTO dotcom VALUES(2, 'facebook', 'com');");
}
} catch (IOException e) {
e.printStackTrace();
} finally {
}
/*
try {
BufferedReader fileReader = new BufferedReader(new FileReader("TopURLs"));
while (fileReader.readLine() != null) {
st.execute("DROP TABLE IF EXISTS dotcom;");
st.execute("CREATE TABLE dotcom (rank integer PRIMARY KEY, domainName varchar(128), domainType varchar(128));");
st.execute("INSERT INTO dotcom SELECT com, domainType FROM dotcom WHERE domainType = 'com'");
}
} catch (IOException e) {
e.printStackTrace();
} finally {
}
*/
/*
Statement createStatement = null;
PreparedStatement insertStatement = null;
try {
BufferedReader fileReader = new BufferedReader(new FileReader("TopURLs"));
String line = null;
createStatement = connection.createStatement();
createStatement.executeUpdate("DROP TABLE IF EXISTS dotcom");
createStatement.executeUpdate("CREATE TABLE dotcom (rank integer PRIMARY KEY, domainName varchar(128), domainType varchar(128))");
connection.setAutoCommit(false);//commit whole batch at the end
insertStatement = connection.prepareStatement("INSERT INTO dotcom VALUES (?, ?, ?)");
while ( (line = fileReader.readLine()) != null) {
line = fileReader.readLine();
String[] urls = line.split("\t");//space or any other delimiter that you're using
insertStatement.setInt(1, Integer.parseInt(urls[0]));
insertStatement.setString(2, urls[1]);
insertStatement.setString(3, urls[2]);
//insertStatement.setString(4, urls[3]);
insertStatement.addBatch();
}
insertStatement.executeBatch();
connection.commit();
} catch (IOException e) {
e.printStackTrace();
} finally {
if(connection != null) {
connection.setAutoCommit(true);
}
if(createStatement != null) {
createStatement.close();
}
if(insertStatement != null) {
insertStatement.close();
}
}
*/
// connection is of type Connection (in JDBC)
DatabaseMetaData dbm = connection.getMetaData();
// check if table is there
ResultSet tables = dbm.getTables(null, null, "table name", null);
if (tables.next()) {
System.out.println("Table exists");
} else {
System.out.println("Table does not exist");
}
// check if view is there?
//"create view foo as select * from table;"
//"select * from foo;"
ResultSet views = dbm.getTables("catalog name", null, null, null);
if (views.next()) {
System.out.println("View exists");
} else {
System.out.println("View does not exist");
}
String query = "SELECT * FROM internet";
ResultSet rs = executeSelect(connection, query);
try {
while (rs.next()) {
System.out.print("Column 1 returned ");
System.out.println(rs.getString(1));
}
} catch (SQLException e) {
e.printStackTrace();
}
rs.close();
dropTable(connection, "dotcom");
createTable(connection,
"dotcom (id int primary key, name varchar(15), type varchar(15));");
int rows = insertIntoTableFromFile(connection, "dotcom", "TopURLs");
System.out.println(rows + " rows inserted.");
}
}
This is too long for a comment.
When loading data from a text file, I generally load into a staging table that consists of character column. Then, I can do validate the data, convert to the appropriate data types, and even reparse some columns, if necessary.
Admittedly, the Postgres COPY command often works quite well just reading in the data.
You would seem to have two options:
Load the data into a staging table and do the data cleansing there.
Read the data line-by-line in java and do the data cleansing there.
Because my SQL skills >> my java skills, it is a no-brainer which I would choose (SQL). But these would seem to be your most reasonable options.
How can we insert an xml file directly into a mysql table in java?
You can adapt your xml to specific mysql format using xslt and after it you can use LOAD_XML statement.
http://dev.mysql.com/doc/refman/5.5/en/load-xml.html
"xml file directly into a mysql" --> I am not clear what does it mean.But, you can insert any files to database as Binary Steam or byte array in java.
Please check below example...
EXAMPLE
public class InsertXML {
private static final String INSERT_SQL = "INSERT INTO ATTACHMENT(FILENAME, MIMETYPE, CONTENT) values(?, ?, ?)";
public void insert(File xmlFile) {
PreparedStatement ps = null;
Connection con = null;
try {
ps = con.prepareStatement(INSERT_SQL);
ps.setString(1, xmlFile.getName());
ps.setString(2, getMimeType(xmlFile));
ps.setBinaryStream(3, new FileInputStream(xmlFile));
ps.executeUpdate();
} catch (SQLException e) {
// CLOSE ps and con;
} catch (FileNotFoundException e) {
// CLOSE ps and con;
} finally {
// CLOSE ps and con;
}
}
public String getMimeType(File xmlFile) {
String mimeType = null;
try {
InputStream is = new BufferedInputStream(new FileInputStream(xmlFile));
mimeType = URLConnection.guessContentTypeFromStream(is);
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return mimeType;
}
public static void main(String[] args) {
InsertXML insertXML = new InsertXML();
insertXML.insert(new File("D:\test.xml"));
}
}
SQL
CREATE TABLE ATTACHMENT (
FILENAME VARCHAR(45) NOT NULL,
MIMETYPE VARCHAR(45) NOT NULL,
CONTENT LONGBLOB NOT NULL
)