I have some problems in managing the storage of data via query (NetBeans-> Java-> Sqlite):
1) I have a folder with some txt file, containing several lines of text (the files do not exceed 2 Kb)
2) The program opens the files in sequence, and stores each word in a table
3)When the program comes to analyze too much data (some time more than 40 files ore more then 82) returns the following error
Exception in thread "main" java.sql.SQLException: unable to open database file
at org.sqlite.DB.throwex (DB.java: 288)
at org.sqlite.DB.executeBatch (DB.java: 236)
at org.sqlite.PrepStmt.executeBatch (PrepStmt.java: 83)
The error is in int [] upCountsb = prepb.executeBatch();
Here the code:
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.sql.Statement;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
public static void main(String[] args) throws ClassNotFoundException, SQLException, IOException, InterruptedException {
Class.forName("org.sqlite.JDBC");
Connection conn = DriverManager.getConnection("jdbc:sqlite:C:/Users/.../test.db");
Statement stmt;
stmt = conn.createStatement();
stmt.executeUpdate("DROP TABLE IF EXISTS words");
stmt.executeUpdate("CREATE TABLE words (words)");
String path_dir ="C:/Users/.../good";
File currentDIR = new File("C:/Users/.../good");
File files[]=currentDIR.listFiles();
String tmp="";
ArrayList app = new ArrayList();
//Search in DIR for Files
for( File f1 : files ){
String nameFile = f1.getName();
FileReader f = null;
BufferedReader fIN = null;
String s;
//Open the file xxx.txt
try{
f = new FileReader(path_dir+"/"+nameFile);
fIN = new BufferedReader(f);
s = fIN.readLine();
while(s != null) {
StringTokenizer st = new StringTokenizer(s);
while(st.hasMoreTokens()) {
String str = st.nextToken().toString().toLowerCase();
Pattern pattern =Pattern.compile("\\W", Pattern.MULTILINE);
String newAll = pattern.matcher(str).replaceAll("").trim();
tmp=newAll;
app.add(tmp); //Add all data in the ArrayList app
} // Close While 'hasMoreTokens'
s = fIN.readLine();
} //Close While on File
} //Close TRAY
catch (FileNotFoundException ex) {
ex.printStackTrace();
} catch (IOException ex) {
ex.printStackTrace();
}
f.close(); //Close FileReader
} //Close Scan DIR for FILE
//Add all data in the Tbl od Database
PreparedStatement prep = conn.prepareStatement("insert into words values (?);");
for (int z=0; z<app.size();z++){
prep.setString(1,app.get(z).toString().toLowerCase());
prep.addBatch();
conn.setAutoCommit(false);
prep.executeBatch(); ***//Here I get the error although i use int [] Count =prep.executeBatch();***
conn.setAutoCommit(true);
}
prep.close();
} //Close MAIN
You need to release the resources you are using once you are done with them. E.g. prepb.close() after executing the statement.
The same thing goes for your file handles.
Also, the point of batching is lost if you execute the statement for every insert.
Since your files are very small you might as well prepare all the data in memory before you persist it to a database.
package stackoverflow.wordanalysis;
import java.io.*;
import java.sql.*;
import java.util.*;
import java.util.regex.*;
public class WordFrequencyImporter {
public static void main(String[] args) throws Exception {
List<String> words = readWords("the-directory-from-which-to-read-the-files");
Class.forName("org.sqlite.JDBC");
Connection conn = DriverManager.getConnection("jdbc:sqlite:test.db");
try {
createWordsTable(conn);
persistWords(words, conn);
} finally {
conn.close();
}
}
private static void persistWords(List<String> words, Connection conn)
throws SQLException {
System.out.println("Persisting " + words.size() + " words");
PreparedStatement prep = conn
.prepareStatement("insert into words values (?);");
try {
for (String word : words) {
prep.setString(1, word);
prep.addBatch();
}
conn.setAutoCommit(false);
prep.executeBatch();
conn.setAutoCommit(true);
} finally {
prep.close();
}
}
private static void createWordsTable(Connection conn) throws SQLException {
Statement stmt = conn.createStatement();
try {
stmt.executeUpdate("DROP TABLE IF EXISTS words");
stmt.executeUpdate("CREATE TABLE words (words)");
} finally {
stmt.close();
}
}
private static List<String> readWords(String path_dir) throws IOException {
Pattern pattern = Pattern.compile("\\W", Pattern.MULTILINE);
List<String> words = new ArrayList<String>();
for (File file : new File(path_dir).listFiles()) {
BufferedReader reader = new BufferedReader(new FileReader(file));
System.out.println("Reading " + file);
try {
String s;
while ((s = reader.readLine()) != null) {
StringTokenizer st = new StringTokenizer(s);
while (st.hasMoreTokens()) {
String token = st.nextToken().toString().toLowerCase();
String word = pattern.matcher(token).replaceAll("")
.trim();
words.add(word);
}
}
} finally {
reader.close();
}
}
return words;
}
}
Related
I came through a link: https://github.com/hyee/OpenCSV which drastically improves the writing time of the JDBC ResultSet to CSV due to setAsyncMode, RESULT_FETCH_SIZE
//Extract ResultSet to CSV file, auto-compress if the fileName extension is ".zip" or ".gz"
//Returns number of records extracted
public int ResultSet2CSV(final ResultSet rs, final String fileName, final String header, final boolean aync) throws Exception {
try (CSVWriter writer = new CSVWriter(fileName)) {
//Define fetch size(default as 30000 rows), higher to be faster performance but takes more memory
ResultSetHelperService.RESULT_FETCH_SIZE=10000;
//Define MAX extract rows, -1 means unlimited.
ResultSetHelperService.MAX_FETCH_ROWS=20000;
writer.setAsyncMode(aync);
int result = writer.writeAll(rs, true);
return result - 1;
}
}
But the problem is I don't know how I can merge above into my requirement. As the link has many other classes involved which I am not sure what they do and if I even need it for my requirement. Still, I tried but it fails to compile whenever I enable 2 commented line code. Below is my code.
Any help on how I can achieve this will be greatly appreciated.
package test;
import java.io.BufferedWriter;
import java.io.FileWriter;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.Statement;
import java.util.Date;
import com.opencsv.CSVWriter;
import com.opencsv.ResultSetHelperService;
public class OpenCSVTest1
{
static Connection con =null;
static Statement stmt = null;
static ResultSet rs = null;
public static void main(String args[]) throws Exception
{
connection ();
retrieveData(con);
}
private static void connection() throws Exception
{
try
{
Class.forName("<jdbcdriver>");
con = DriverManager.getConnection("jdbc:","<username>","<pass>");
System.out.println("Connection successful");
}
catch (Exception e)
{
System.out.println("Exception while establishing sql connection");
throw e;
}
}
private static void retrieveData(Connection con) throws Exception
{
try
{
stmt=con.createStatement();
stmt = con.createStatement(ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY);
String query = "SELECT * FROM dbo.tablename";
rs=stmt.executeQuery(query);
CSVWriter writer = new CSVWriter(new BufferedWriter(new FileWriter("C:\\Data\\File1.csv")));
ResultSetHelperService service = new ResultSetHelperService();
/*** ResultSetHelperService.RESULT_FETCH_SIZE=10000; ***/ // to add
service.setDateTimeFormat("yyyy-MM-dd HH:mm:ss.SSS");
System.out.println("**** Started writing Data to CSV **** " + new Date());
writer.setResultService(service);
/*** writer.setAsyncMode(aync); ***/ // to add
int lines = writer.writeAll(rs, true, true, false);
writer.flush();
writer.close();
System.out.println("** OpenCSV -Completed writing the resultSet at " + new Date() + " Number of lines written to the file " + lines);
}
catch (Exception e)
{
System.out.println("Exception while retrieving data" );
e.printStackTrace();
throw e;
}
finally
{
rs.close();
stmt.close();
con.close();
}
}
}
UPDATE
I have updated my code. Right now code is writing complete resultset in CSV at once using writeAll method which is resulting in time consumption.
Now what I want to do is write resultset to CSV in batches as resultset's first column will always have dynamically generated via SELECT query Auto Increment column (Sqno) with values as (1,2,3..) So not sure how I can read result sets first column and split it accoridngly to write in CSV. may be HashMap might help, so I have also added resultset-tohashmap conversion code if required.
import com.opencsv.CSVWriter;
import com.opencsv.ResultSetHelperService;
import java.io.BufferedWriter;
import java.io.FileWriter;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.ArrayList;
import java.util.Date;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
public class OpenCSVTest1
{
static int fetchlimit_src = 100;
static Connection con =null;
static Statement stmt = null;
static ResultSet rs = null;
static String filename = "C:\\Data\\filename.csv";
static CSVWriter writer;
public static void main(String args[])
{
try
{
connection();
retrieveData(con);
}
catch(Exception e)
{
System.out.println(e);
}
}
private static void connection() throws Exception
{
try
{
Class.forName("<jdbcdriver>");
con = DriverManager.getConnection("jdbc:","<username>","<pass>");
System.out.println("Connection successful");
}
catch (Exception e)
{
System.out.println("Exception while establishing sql connection");
throw e;
}
}
private static void retrieveData(Connection con) throws Exception
{
try
{
stmt=con.createStatement();
String query = "SELECT ROWNUM AS Sqno, * FROM dbo.tablename "; // Oracle
// String query = "SELECT ROW_NUMBER() OVER(ORDER BY Id ASC) AS Sqno, * FROM dbo.tablename "; // SQLServer
System.out.println(query);
stmt = con.createStatement(ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY);
stmt.setFetchSize(fetchlimit_src);
System.out.println("**** Started querying src **** " + new Date());
rs=stmt.executeQuery(query);
System.out.println("**** Completing querying src **** " + new Date());
// resultset_List(rs); // If required store resultset(rs) to HashMap
writetoCSV(rs,filename);
/** How to write resultset to CSV in batches instead of writing all at once to speed up write performance ?
* Hint: resultset first column is Autoincrement [Sqno] (1,2,3...) which might help to split result in batches.
*
**/
}
catch (Exception e)
{
System.out.println("Exception while retrieving data" );
e.printStackTrace();
throw e;
}
finally
{
rs.close();
stmt.close();
con.close();
}
}
private static List<Map<String, Object>> resultset_List(ResultSet rs) throws SQLException
{
ResultSetMetaData md = rs.getMetaData();
int columns = md.getColumnCount();
List<Map<String, Object>> rows = new ArrayList<Map<String, Object>>();
while (rs.next())
{
Map<String, Object> row = new HashMap<String, Object>(columns);
for(int i = 1; i <= columns; ++i)
{
row.put(md.getColumnName(i), rs.getObject(i));
}
rows.add(row);
}
// System.out.println(rows.toString());
return rows;
}
private static void writetoCSV(ResultSet rs, String filename) throws Exception
{
try
{
writer = new CSVWriter(new BufferedWriter(new FileWriter(filename)));
ResultSetHelperService service = new ResultSetHelperService();
service.setDateTimeFormat("yyyy-MM-dd HH:mm:ss.SSS");
long batchlimit = 1000;
long Sqno = 1;
ResultSetMetaData rsmd = rs.getMetaData();
String columnname = rsmd.getColumnLabel(1); // To retrieve columns with labels (for example SELECT ROWNUM AS Sqno)
System.out.println("**** Started writing Data to CSV **** " + new Date());
writer.setResultService(service);
int lines = writer.writeAll(rs, true, true, false);
System.out.println("** OpenCSV -Completed writing the resultSet at " + new Date() + " Number of lines written to the file " + lines);
}
catch (Exception e)
{
System.out.println("Exception while writing data" );
e.printStackTrace();
throw e;
}
finally
{
writer.flush();
writer.close();
}
}
}
You should be able to use the OpenCSV sample, pretty much exactly as it is provided in the documentation. So, there should be no need for you to write any of your own batching logic.
I was able to write a 6 million record result set to a CSV file in about 10 seconds. To be clear -that was just the file-write time, not the DB data-fetch time - but I think that should be fast enough for your needs.
Here is your code, with adaptations for using OpenCSV based on its documented approach... But please see the warning at the end of my notes!
import com.opencsv.CSVWriter;
import com.opencsv.ResultSetHelperService;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.Statement;
import java.util.Date;
import java.text.SimpleDateFormat;
public class OpenCSVDemo {
static int fetchlimit_src = 100;
static Connection con = null;
static Statement stmt = null;
static ResultSet rs = null;
static String filename = "C:\\Data\\filename.csv";
public static void main(String args[]) {
try {
connection();
retrieveData(con);
} catch (Exception e) {
System.out.println(e);
}
}
private static void connection() throws Exception {
try {
final String jdbcDriver = "YOURS GOES HERE";
final String dbUrl = "YOURS GOES HERE";
final String user = "YOURS GOES HERE";
final String pass = "YOURS GOES HERE";
Class.forName(jdbcDriver);
con = DriverManager.getConnection(dbUrl, user, pass);
System.out.println("Connection successful");
} catch (Exception e) {
System.out.println("Exception while establishing sql connection");
throw e;
}
}
private static void retrieveData(Connection con) throws Exception {
try {
stmt = con.createStatement();
String query = "select title_id, primary_title from imdb.title";
System.out.println(query);
stmt = con.createStatement(ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY);
stmt.setFetchSize(fetchlimit_src);
System.out.println("**** Started querying src **** " + new Date());
rs = stmt.executeQuery(query);
System.out.println("**** Completing querying src **** " + new Date());
// resultset_List(rs); // If required store resultset(rs) to HashMap
System.out.println();
String timeStamp = new SimpleDateFormat("yyyy.MM.dd.HH.mm.ss").format(new Date());
System.out.println("Started writing CSV: " + timeStamp);
writeToCsv(rs, filename, null, Boolean.FALSE);
timeStamp = new SimpleDateFormat("yyyy.MM.dd.HH.mm.ss").format(new Date());
System.out.println("Finished writing CSV: " + timeStamp);
System.out.println();
} catch (Exception e) {
System.out.println("Exception while retrieving data");
e.printStackTrace();
throw e;
} finally {
rs.close();
stmt.close();
con.close();
}
}
public static int writeToCsv(final ResultSet rs, final String fileName,
final String header, final boolean aync) throws Exception {
try (CSVWriter writer = new CSVWriter(fileName)) {
//Define fetch size(default as 30000 rows), higher to be faster performance but takes more memory
ResultSetHelperService.RESULT_FETCH_SIZE = 1000;
//Define MAX extract rows, -1 means unlimited.
ResultSetHelperService.MAX_FETCH_ROWS = 2000;
writer.setAsyncMode(aync);
int result = writer.writeAll(rs, true);
return result - 1;
}
}
}
Points to note:
1) I used "async" set to false:
writeToCsv(rs, filename, null, Boolean.FALSE);
You may want to experiment with this and the other settings to see if they make any significant difference for you.
2) Regarding your comment "the link has many other classes involved": The OpenCSV library's entire JAR file needs to be included in your project, as does the related disruptor JAR:
opencsv.jar
disruptor-3.3.6.jar
To get the JAR files, go to the GitHub page, click on the green button, select the zip download, unzip the zip file, and look in the "OpenCSV-master\release" folder.
Add these two JARs to your project in the usual way (depends on how you build your project).
3) WARNING: This code runs OK when you use Oracle's Java 8 JDK/JRE. If you try to use OpenJDK (e.g. for Java 13 or similar) it will not run. This is because of some changes behind the scenes to hidden classes. If you are interested, there are more details here.
If you need to use an OpenJDK version of Java, you may therefore have better luck with the library on which this CSV library is based: see here.
This question already has answers here:
What is a NullPointerException, and how do I fix it?
(12 answers)
Closed 4 years ago.
I do not speak English well
An error occurred while doing JDBC.
I knew during coding that ' executequery ' should only be done by ' select ' in the database, and ' insert ' should be ' executeupdate '.
However, when I did executeupdate, I had a problem with the variable and changed it to " int "
Then an error occurred.
How can it be carried out as I intended?
I'll show you the code.
import java.io.BufferedReader;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.ArrayList;
public class EdgeCloud {
static ArrayList<String> deleteDB = new ArrayList<>();
static ArrayList<String> DBCarIP = new ArrayList<>();
static ArrayList<String> ContainerCarIP = new ArrayList<>();
static String carSSID = "192.0.1";
static String state;
static String upDateInputContainer;
static String upDateInputCar;
static int i = 0;
static String cIP;
//-------------------------------------------( main start )----------------------------------------------------//
public static void main(String[] args) throws Exception {
System.out.println("main start");
while(true) {
// JDBC(state="check");
if(carSSID!=null) {
makeContainer();
}else {
UPDATE();
}
}
}
//-------------------------------------------( main end )----------------------------------------------------//
//-------------------------------------------( make container start )----------------------------------------------------//
public static void makeContainer() throws Exception {
System.out.println("mc start");
String makeContainer = "/usr/local/bin/docker run --name "+carSSID+" ubuntu:16.04";
Runtime runtime = Runtime.getRuntime();
Process process = runtime.exec(makeContainer);
InputStream is = process.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line;
while((line = br.readLine()) != null) {
System.out.println(line);
}
String ipload = "/usr/local/bin/docker inspect -f {{.NetworkSettings.IPAddress}} "+carSSID;
IPLoad(ipload);
}
//-------------------------------------------( make container end )----------------------------------------------------//
//-------------------------------------------( get container ip start )----------------------------------------------------//
public static void IPLoad(String ipload) throws Exception {
System.out.println("ipload start");
upDateInputCar = null;
upDateInputContainer = null;
upDateInputCar = carSSID;
carSSID = null;
state = "insert";
Runtime runtime = Runtime.getRuntime();
Process process = runtime.exec(ipload);
InputStream is = process.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
while((cIP = br.readLine()) != null) {
upDateInputContainer= cIP;
System.out.println("cip : "+cIP);
}
System.out.println("upd"+upDateInputContainer);
JDBC(state);
}
//-------------------------------------------( get container ip end )----------------------------------------------------//
//-------------------------------------------( update start )----------------------------------------------------//
public static void UPDATE() throws Exception {
state = "check";
JDBC(state);
String getContainerName = "/usr/local/bin/docker ps --format {{.Names}}";
Runtime runtime = Runtime.getRuntime();
Process process = runtime.exec(getContainerName);
InputStream is = process.getInputStream();
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line;
while((line = br.readLine()) != null) {
ContainerCarIP.add(line);
}
deleteDB.addAll(DBCarIP);
deleteDB.removeAll(ContainerCarIP);
if(deleteDB.get(0)!=null) {
state="delete";
while(deleteDB.get(i++)!=null) {
//JDBC(state);
}
i = 0;
}
}
//-------------------------------------------( update end )----------------------------------------------------//
//-------------------------------------------( socket start )----------------------------------------------------//
/*socket place
-
-
-
-
-
-
*/
//-------------------------------------------( soket end )----------------------------------------------------//
//-------------------------------------------( JDBC start )----------------------------------------------------//
public static void JDBC(String state){
System.out.println("JDBC start");
Connection conn = null ;
Statement stmt = null;
String query = null;
if(state=="insert") {
query = "insert into inpo (ip, carid) values ('"+upDateInputContainer+"', '"+upDateInputCar+"')";
System.out.println(query);
}
if(state=="delete") {
query = "delete from inpo where carid = " +deleteDB.get(i);
}
if(state=="check") {
query = "select carid from inpo";
}
try{
Class.forName("com.mysql.cj.jdbc.Driver").newInstance();
}
catch(ClassNotFoundException e ){
System.out.println( "드라이버 연결 에러." ) ;
}
catch(Exception etc) {
System.out.println(etc.getMessage());
}
try{
// String url = "jdbc:mysql://192.168.44.122:3306/ssidIp?serverTimezone=UTC&autoReconnect=true&useSSL=false";
String url = "jdbc:mysql://localhost:3306/ssidIp?serverTimezone=UTC&autoReconnect=true&useSSL=false";
String userPass = "123456789" ;
conn = DriverManager.getConnection(url, userId, userPass);
stmt = conn.createStatement();
ResultSet rs = null;
int rss = 0;
if(state=="check") {
rs = stmt.executeQuery(query);
}else {
rss = stmt.executeUpdate(query);
}//////////////////////////////////////////////////////
if (state=="check") {
rs = stmt.getResultSet();
}
System.out.println("------------------------------------");
System.out.println("debug1 ");
if(state=="check") {
System.out.println("debug 2");
while (rs.next()) {
String str = rs.getNString(1);
DBCarIP.add(str);
}
}else {
System.out.println("debug 3");
while(rs.next()) { // here NullpointerException
System.out.println("debug 4");
String str = rs.getNString(1);
System.out.println(str);
}
}
System.out.println("연결");
stmt.close();
conn.close();
}
catch( SQLException e ){
System.out.println( "SQLException : " + e.getMessage() ) ;
}
}
}
//-------------------------------------------( JDBC end )----------------------------------------------------//
ERROR part
main start
mc start
ipload start
cip : 172.17.0.3
upd172.17.0.3
JDBC start
insert into inpo (ip, carid) values ('172.17.0.3', '192.0.1')
------------------------------------
debug1
debug 3
Exception in thread "main" java.lang.NullPointerException
at edgeCloud.EdgeCloud.JDBC(EdgeCloud.java:195)
at edgeCloud.EdgeCloud.IPLoad(EdgeCloud.java:86)
at edgeCloud.EdgeCloud.makeContainer(EdgeCloud.java:61)
at edgeCloud.EdgeCloud.main(EdgeCloud.java:33)
You’re reading from rs instead of rss but rs is null for an insert or delete query.
else {
System.out.println("debug 3");
while(rs.next()) {
System.out.println("debug 4");
String str = rs.getNString(1);
System.out.println(str);
}
}
You need to change this to something like this since rss will contain the number of rows inserted or deleted.
else {
System.out.println("debug 3");
if (rss > 0) {
System.out.println("debug 4");
//...
}
I have written this code to read from a table and write onto a file, but I am unable to write to a file. The file gets created, but its empty. I don't have a mysql problem tried to insert comments and debug. Code compiles fine, but has a problem in creating a file:
import java.io.File;
import java.io.FileNotFoundException;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.Statement;
import java.util.Formatter;
import java.util.Scanner;
import java.util.logging.Level;
import java.util.logging.Logger;
public class tabletocsv {
public static void main(String[] args) {
System.out.println("enter the table name");
Scanner input = new Scanner(System.in);
String table = input.next();
System.out.println("enter number of columns");
int columns = input.nextInt();
createcsv(table, columns);
System.out.print(" csv created");
displaycsv(table, columns);
}
static void displaycsv(String table, int count) {
Scanner file123;
try {
file123 = new Scanner(new File("/Users/Tannishk/Documents/csv/" + table + ".csv"));
//System.out.printf("reading");
while (file123.hasNext()) {
System.out.println("going inside");
for (int i = 1; i <= count; i++) {
String a = file123.next();
System.out.print(a + " ");
}
System.out.println();
}
} catch (FileNotFoundException ex) {
Logger.getLogger(tabletocsv.class.getName()).log(Level.SEVERE, null, ex);
}
}
static void createcsv(String table, int count) {
Formatter x;
Connection connection;
Statement st;
try {
x = new Formatter("/Users/Tannishk/Documents/csv/" + table + ".csv");
Class.forName("com.mysql.jdbc.Driver");
connection = DriverManager.getConnection("jdbc:mysql://localhost:3306/db", "root", "password");
st = connection.createStatement();
ResultSet rs = st.executeQuery("select * from " + table + ";");
while (rs.next()) {
for (int i = 1; i <= count; i++) {
String a = rs.getString(i);
x.format("%s ", a);
// System.out.print(a);
}
x.format("\n");
}
} catch (Exception e) {
} finally {
}
}
}
Try flushing and closing the formatter
I think you have missed to flush the file.
static void createcsv(String table,int count)
{
Formatter x = null;
Connection connection;
Statement st;
try{
x = new Formatter("/Users/Tannishk/Documents/csv/"+table+".csv");
Class.forName("com.mysql.jdbc.Driver");
connection = DriverManager.getConnection("jdbc:mysql://localhost:3306/db","root","password");
st = connection.createStatement();
ResultSet rs= st.executeQuery("select * from "+table+";");
while(rs.next())
{
for(int i=1;i<=count;i++)
{
String a = rs.getString(i);
x.format("%s ",a);
// System.out.print(a);
}
x.format("\n");
}
}
catch(Exception e)
{
}
finally
{
try {
x.flush();
x.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
Try this ...
You need to close the output stream that you're using to write to the file. For example if you're using a FileWriter object named fout to write to the file, you'll have to do fout.close() to actually have anything written to the file as otherwise it would just be in memory.
I have a database in mysql with field userId, userName etc,. from here I'm retrieving userid based on some criteria.
I have a folder with files and name of the file is some userId . I want to compare retrieved userId with those files and if they match further process those files.
I tried this code but its giving number format exception. not able to understand why?
package read;
import java.io.*;
import java.util.*;
import java.sql.*;
public class read1 {
static final String JDBC_DRIVER = "com.mysql.jdbc.Driver";
static final String DB_URL = "jdbc:mysql://localhost/test";
static final String USER = "root";
static final String PASS = "root";
public static void main(String args[])throws Exception{
Connection conn = null;
Statement stmt = null;
try{
Class.forName("com.mysql.jdbc.Driver");
System.out.println("Connecting to database...");
conn = DriverManager.getConnection(DB_URL,USER,PASS);
System.out.println("Creating statement...");
stmt = conn.createStatement();
String sql;
sql = "SELECT userId FROM profile1 where friendCount >200 AND statusCount > 200";
ResultSet rs = stmt.executeQuery(sql);
processing p = new processing();
File directory = new File("/home/abc/tweet");
//get all the files from a directory
File[] fList = directory.listFiles();
while(rs.next()){
int uid = rs.getInt("userId");
System.out.print("ID: " + uid +"\n");
for (File file : fList){
if(file.isFile())
System.out.println(file.getName());
if(Integer.parseInt(file.getName())== uid)
p.process(file.getName());
}
}
}
catch(SQLException e){
e.printStackTrace();
}
}
}
class processing{
void process(String filename)throws Exception{
FileReader fr = new FileReader("/home/abc/tweet/"+filename);
BufferedReader br = new BufferedReader(fr);
FileWriter wr = new FileWriter("/home/abc/newtweet/"+filename);
String s;
String str="Text:";
String a[];
BufferedWriter bw = new BufferedWriter(wr);
while ((s = br.readLine()) != null) {
if (s.startsWith("Text:")) { // worked
a= s.split(" ");
for(int k=1;k<a.length;k++)
bw.write(a[k]+"\n"); // worked
}
}
br.close();
bw.close();
}
}
One reason could be that there are also other files in the folder /home/abc/tweet/ with names that cant be parsed as an Integer.
I made a program that can parse .csv file to database,
I want to make semicolon as delimiter but I got some trouble here ~
This is my code CSVLoader.java
package id.co.lolo.coreservice;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.sql.Connection;
import java.sql.PreparedStatement;
import java.util.Date;
import org.apache.commons.lang.StringUtils;
import au.com.bytecode.opencsv.CSVReader;
public class CSVLoader {
private static final String SQL_INSERT = "INSERT INTO ${table}(${keys}) VALUES(${values})";
private static final String TABLE_REGEX = "\\$\\{table\\}";
private static final String KEYS_REGEX = "\\$\\{keys\\}";
private static final String VALUES_REGEX = "\\$\\{values\\}";
private Connection connection;
private char seprator;
/**
* Public constructor to build CSVLoader object with Connection details. The
* connection is closed on success or failure.
*
* #param connection
*/
public CSVLoader(Connection connection) {
this.connection = connection;
// Set default separator
this.seprator = ',';
}
/**
* Parse CSV file using OpenCSV library and load in given database table.
*
* #param csvFile
* Input CSV file
* #param tableName
* Database table name to import data
* #param truncateBeforeLoad
* Truncate the table before inserting new records.
* #throws Exception
*/
#SuppressWarnings("resource")
public void loadCSV(String csvFile, String tableName,
boolean truncateBeforeLoad) throws Exception {
CSVReader csvReader = null;
if (null == this.connection) {
throw new Exception("Not a valid connection.");
}
try {
csvReader = new CSVReader(new FileReader(csvFile), this.seprator);
} catch (Exception e) {
e.printStackTrace();
throw new Exception("Error occured while executing file. "
+ e.getMessage());
}
String[] headerRow = csvReader.readNext();
if (null == headerRow) {
throw new FileNotFoundException(
"No columns defined in given CSV file."
+ "Please check the CSV file format.");
}
String questionmarks = StringUtils.repeat("?,", headerRow.length);
questionmarks = (String) questionmarks.subSequence(0,
questionmarks.length() - 1);
String query = SQL_INSERT.replaceFirst(TABLE_REGEX, tableName);
query = query
.replaceFirst(KEYS_REGEX, StringUtils.join(headerRow, ","));
query = query.replaceFirst(VALUES_REGEX, questionmarks);
System.out.println("Query: " + query);
String[] nextLine;
Connection con = null;
PreparedStatement ps = null;
try {
con = this.connection;
con.setAutoCommit(false);
ps = con.prepareStatement(query);
if (truncateBeforeLoad) {
// delete data from table before loading csv
con.createStatement().execute("DELETE FROM " + tableName);
}
final int batchSize = 1000;
int count = 0;
Date date = null;
while ((nextLine = csvReader.readNext()) != null) {
if (null != nextLine) {
int index = 1;
for (String string : nextLine) {
date = DateUtil.convertToDate(string);
if (null != date) {
ps.setDate(index++,
new java.sql.Date(date.getTime()));
} else {
ps.setString(index++, string);
}
}
ps.addBatch();
}
if (++count % batchSize == 0) {
ps.executeBatch();
}
}
ps.executeBatch(); // insert remaining records
con.commit();
} catch (Exception e) {
con.rollback();
e.printStackTrace();
throw new Exception(
"Error occured while loading data from file to database."
+ e.getMessage());
} finally {
if (null != ps)
ps.close();
if (null != con)
con.close();
csvReader.close();
}
}
public char getSeprator() {
return seprator;
}
public void setSeprator(char seprator) {
this.seprator = seprator;
}
}
this is Main.java
package id.co.lolo.coreservice;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
public class Main {
#SuppressWarnings("unused")
private static String JDBC_CONNECTION_URL =
"jdbc:mysql://localhost:3306/bni";
public static void main(String[] args) {
try {
CSVLoader loader = new CSVLoader(getCon());
loader.setSeprator(';');
loader.loadCSV("C:\\Log\\Logtima.csv", "coreservice", true);
} catch (Exception e) {
e.printStackTrace();
}
}
private static Connection getCon() {
Connection connection = null;
try {
Class.forName("com.mysql.jdbc.Driver");
connection = DriverManager.getConnection("jdbc:mysql://localhost:3306/bni","root","shikamaru");
} catch (ClassNotFoundException e) {
e.printStackTrace();
} catch (SQLException e) {
e.printStackTrace();
}
return connection;
}
}
I got this Error,
Query: INSERT INTO coreservice(MODULE_NAME,SERVICE_NAME,COUNTER,LOG_DATE,UPDATE_DATE) VALUES(?)
java.sql.BatchUpdateException: Column count doesn't match value count at row 1
at com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1693)
at com.mysql.jdbc.PreparedStatement.executeBatch(PreparedStatement.java:1108)
at id.co.bni.coreservice.CSVLoader.loadCSV(CSVLoader.java:118)
at id.co.bni.coreservice.Main.main(Main.java:20)
java.lang.Exception: Error occured while loading data from file to database.Column count doesn't match value count at row 1
at id.co.bni.coreservice.CSVLoader.loadCSV(CSVLoader.java:123)
at id.co.bni.coreservice.Main.main(Main.java:20)