I am trying to write some query results in a csv file.
I thought it was working fine, until I saw the last line of the file after it's been written:
value;nettingNodeData;BLE57385;CVAEngine;BLE;;.4;;BDR;NA;ICE;;RDC;;CVAEngine;;4841263;RDC
value;ne
The part where I am writing the file :
public int getLeNodes(String runId, File file) {
Connection connect = null;
PreparedStatement ps = null;
ResultSet rs = null;
int lines = 0;
try {
connect = newConnection();
ps = connect.prepareStatement(HIER_NTT.replace("?", runId));
ps.setFetchSize(1500);
rs = ps.executeQuery();
lines += nnpw.writeCore(file, rs);
} catch (Exception e) {
e.printStackTrace();
} finally {
close(rs, ps, connect);
}
return lines;
}
public int writeCore(File file, ResultSet rs) throws FileNotFoundException, IOException {
int count = 0;
try {
BufferedWriter output = new BufferedWriter(new FileWriter(file, true));
ResultSetMetaData rsmd = rs.getMetaData();
int colCount = rsmd.getColumnCount();
while (rs.next()) {
for (int col = 1; col <= colCount; col++) {
try {
String val = rs.getString(col);
if (rs.wasNull()) {
val = "";
}
output.append(val);
} catch (ArrayIndexOutOfBoundsException e) {
String dec = rs.getBigDecimal(col).toPlainString();
System.err.println(String.format("%s %d %s %s %d %s %d", rs.getString(1), col,
rsmd.getColumnTypeName(col), file, count, dec, dec.length()));
output.append(dec);
}
if (col < colCount) {
output.append(";");
}
}
output.newLine();
count++;
}
} catch (Exception e) {
e.printStackTrace();
}
return count;
}
And the SQL (HIER_NTT):
select 'value', 'nettingNodeData', 'BLE' || d.deal_numadm,
'CVAEngine', 'BLE', '', (1-ntt.lgd_cp), '', 'BDR', 'NA', 'ICE', '', 'RDC', '',
'CVAEngine', '', d.ntt_id, 'RDC' from ntt ntt
join deals d on d.ntt_id = ntt.ntt_id and d.deal_scope='Y'
join dt_runs r on r.run_id = ntt.run_id
where r.run_id=? and d.deal_numadm!=0 group by d.deal_numadm, d.deal_nummas, d.ntt_id, ntt.lgd_cp
So, why does my file end abruptly like this?
You should call output.close() when you are done writing to the file. The missing output is probably still in the buffer when the java process exits
Related
I have written the below Java code, which is executing fine in order to export the results in xlsx file from Oracle DB. But can someone help me to improve the performance of this code by which this below code can extract the result very fast.
Below is the Java Code which I have tried as of now; But it is slow while exporting result as compared to excel export from SQL Developer
public class FilesFromFolder {
private Workbook writeWorkbook;
public void ExportService(DBConnection con) {
writeWorkbook = new XSSFWorkbook();
Sheet desSheet = writeWorkbook.createSheet("Data");
Statement stmt = null;
ResultSet rs = null;
int columnsNumber = 0;
ResultSetMetaData rsmd = null;
FileOutputStream fileOut = null;
Connection cntn = null;
String filePath = "C:\\Users\\Desktop\\OracleExport";
File files = new File(filePath);
File[] file = files.listFiles();
String fileNameWithOutExt = null;
if (file != null) {
for (int i = 0; i < file.length; i++) {
if (file[i].isFile()) {
String tempFilename = file[i].getName();
fileNameWithOutExt = tempFilename.replaceFirst("[.][^.]+$", "");
File fileTemp = file[i];
try {
String fileContent = FileUtils.readFileToString(fileTemp, "UTF8");
// System.out.println(fileContent);
cntn = con.getConnection();
stmt = cntn.createStatement();
rs = stmt.executeQuery(fileContent);
rsmd = rs.getMetaData();
columnsNumber = rsmd.getColumnCount();
Row desRow1 = desSheet.createRow(0);
for (int col = 0; col < columnsNumber; col++) {
Cell newpath = desRow1.createCell(col);
newpath.setCellValue(rsmd.getColumnLabel(col + 1));
}
while (rs.next()) {
System.out.println("Row number -->" + rs.getRow());
Row desRow = desSheet.createRow(rs.getRow());
for (int col = 0; col < columnsNumber; col++) {
Cell newpath = desRow.createCell(col);
newpath.setCellValue(rs.getString(col + 1));
}
String outputFile = "C:\\Users\\Desktop\\OracleExport\\" + fileNameWithOutExt
+ ".xlsx";
fileOut = new FileOutputStream(outputFile);
writeWorkbook.write(fileOut);
}
System.out.println(fileNameWithOutExt + " export complete");
} catch (IOException e) {
e.printStackTrace();
} catch (SQLException e) {
e.printStackTrace();
} finally {
if (fileOut!= null) {
try {
fileOut.close();
} catch (IOException e) {
e.printStackTrace();
}
}
if (cntn != null) {
con.closeConnection();
}
}
}
}
}
}
}
Quick questions...
I'm trying to make a Dynamic JTree but I can't get to put every database I have into one single node for each one. This is my code so far:
jTree2 = new javax.swing.JTree();
try {
String DSN = "jdbc:mysql://localhost";
String user = "root";
String password = "";
conexion = DriverManager.getConnection(DSN, user, password);
}
catch(Exception e) {
System.out.println("ERROR");
}
try {
sentencia = conexion.createStatement(
ResultSet.TYPE_SCROLL_INSENSITIVE,
ResultSet.CONCUR_READ_ONLY);
}
catch(Exception e) {
System.out.println("ERROR2");
}
try {
String hi = "";
ResultSet rs1 = conexion.getMetaData().getCatalogs();
ResultSetMetaData rsmd = rs1.getMetaData();
int columnCount = rsmd.getColumnCount();
while (rs1.next()) {
for (int i = 1; i <= columnCount; i++ ) {
hi = hi + rs1.getString(i) + ", ";
}
//for
String sb = hi.substring(0, hi.length()-2);
jTree2.setModel(new FileSystemModel(new File(sb)));
}
}
catch(Exception ae) {
System.out.println("ERROR3");
}
jScrollPane3.setViewportView(jTree2);
And the result I get is this:
Every database is splitted by a "," but I want them to be on a single node for each one. Any help?
This should do it for you:
DefaultMutableTreeNode parent = new DefaultMutableTreeNode("Databases", true);
while (rs1.next()) {
for (int i = 1; i <= columnCount; i++) {
DefaultMutableTreeNode node = new DefaultMutableTreeNode(rs1.getString(i), true);
parent.add(node);
}
jTree2.setModel(new DefaultTreeModel(parent));
}
so i found this code on the internet, basically what supposedly it can do is backup all the tables from a db, my question is on this line:
res = st.executeQuery("select * from xcms." + tableName);
i get the following excpetion exception: SQLException information
what does xcms. stands for? what else can i put here?
res = st.executeQuery("select * from " + tableName);
btw if i erase xcms. and put it like this ^, i can save only the first table not all the tables, thx
the sourcecode webpage:
https://gauravmutreja.wordpress.com/2011/10/13/exporting-your-database-to-csv-file-in-java/#comment-210
public static void main(String[] args) {
Connection con = null;
String url = "jdbc:mysql://localhost:3306/";
String db = "gg";
String driver = "com.mysql.jdbc.Driver";
String user = "root";
String pass = "";
FileWriter fw;
try {
Class.forName(driver);
con = DriverManager.getConnection(url + db, user, pass);
Statement st = con.createStatement();
ResultSet res = st.executeQuery("SELECT table_name FROM INFORMATION_SCHEMA.TABLES WHERE table_schema = 'gg'");
List<String> tableNameList = new ArrayList<String>();
while (res.next()) {
tableNameList.add(res.getString(1));
}
String filename = "C:\\Users\\Angel Silva\\Documents";
for (String tableName : tableNameList) {
int k = 0;
int j = 1;
System.out.println(tableName);
List<String> columnsNameList = new ArrayList<String>();
res = st.executeQuery("select * from " + tableName);
int columnCount = getColumnCount(res);
try {
fw = new FileWriter("C:\\Users\\Angel Silva\\Documents\\popo1121.csv");
for (int i = 1; i <= columnCount; i++) {
fw.append(res.getMetaData().getColumnName(i));
fw.append(",");
}
fw.append(System.getProperty("line.separator"));
while (res.next()) {
for (int i = 1; i <= columnCount; i++) {
if (res.getObject(i) != null) {
String data = res.getObject(i).toString();
fw.append(data);
fw.append(",");
} else {
String data = "null";
fw.append(data);
fw.append(",");
}
}
fw.append(System.getProperty("line.separator"));
}
fw.flush();
fw.close();
} catch (IOException ioe) {
ioe.printStackTrace();
}
}
con.close();
} catch (ClassNotFoundException cnfe) {
System.err.println("Could not load JDBC driver");
cnfe.printStackTrace();
}catch (SQLException sqle) {System.err.println("SQLException information");}
}
public static int getRowCount(ResultSet res) throws SQLException {
res.last();
int numberOfRows = res.getRow();
res.beforeFirst();
return numberOfRows;
}
public static int getColumnCount(ResultSet res) throws SQLException {
return res.getMetaData().getColumnCount();
}
}
This is what I used:
public void sqlToCSV (String query, String filename){
log.info("creating csv file: " + filename);
try {
FileWriter fw = new FileWriter(filename + ".csv");
if(conn.isClosed()) st = conn.createStatement();
ResultSet rs = st.executeQuery(query);
int cols = rs.getMetaData().getColumnCount();
for(int i = 1; i <= cols; i ++){
fw.append(rs.getMetaData().getColumnLabel(i));
if(i < cols) fw.append(',');
else fw.append('\n');
}
while (rs.next()) {
for(int i = 1; i <= cols; i ++){
fw.append(rs.getString(i));
if(i < cols) fw.append(',');
}
fw.append('\n');
}
fw.flush();
fw.close();
log.info("CSV File is created successfully.");
conn.close();
} catch (Exception e) {
log.fatal(e);
}
}
The xms stands for the Database name, in your Connection in the java program you already are establishing the connection to the Database:
(DriverManager.getConnection(url + db, user, pass);
The string db is the name of the DB to connect to.
So no need to have the xms. .just for example use this query:
"SELECT * FROM "+tableName+";"
This is executed in the database you are connected to, for example ggin your code.
For writting the CSV file chillyfacts.com/export-mysql-table-csv-file-using-java/ may help!
SELECT * FROM <MENTION_TABLE_NAME_HERE> INTO OUTFILE <FILE_NAME> FIELDS TERMINATED BY ',';
Example :
SELECT * FROM topic INTO OUTFILE 'D:\5.csv' FIELDS TERMINATED BY ',';
use opencsv dependency to export SQL data to CSV using minimal lines of code.
import com.opencsv.CSVWriter;
import java.io.FileWriter;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
public class CsvWriter {
public static void main(String[] args) throws Exception {
CSVWriter writer = new CSVWriter(new FileWriter("filename.csv"), '\t');
Boolean includeHeaders = true;
Statement statement = null;
ResultSet myResultSet = null;
Connection connection = null;
try {
connection = //make database connection here
if (connection != null) {
statement = connection.createStatement();
myResultSet = statement.executeQuery("your sql query goes here");
writer.writeAll(myResultSet, includeHeaders);
}
} catch (SQLException throwables) {
throwables.printStackTrace();
}
}
}
I'm trying to export oracle table into csv file and I have created a Class doing so but the output file format was as follow:
12345
1002988846
1
Salary is Here
67891
1009007305
0
Ma3ash is Here!
55555
1095003139
0
Ma3ash is Here!
77777
1023456789
1
Salary is Here
and here is the class:
import java.io.*;
import java.sql.*;
public class newway {
public void mymethod() throws Exception {
try
{
Class.forName("oracle.jdbc.OracleDriver");
Connection conn= DriverManager.getConnection("jdbc:oracle:thin:#172.17.60.225:1521/FRSTEST", "TRASSET", "TRASSET");
conn.setAutoCommit(false);
Statement st=conn.createStatement();
ResultSet rs=st.executeQuery("Select * from Table1");
ResultSetMetaData rsmd = rs.getMetaData();
FileWriter cname = new FileWriter("D:\\asd.csv");
BufferedWriter bwOutFile = new BufferedWriter(cname);
StringBuffer sbOutput = new StringBuffer();
sbOutput.append("S_DATE");
bwOutFile.append(sbOutput);
bwOutFile.append(System.getProperty("line.separator"));
System.out.println("No of columns in the table:"+ rsmd.getColumnCount());
for (int i = 1; i <= rsmd.getColumnCount(); i++)
{
String fname = rsmd.getColumnName(i);
}
System.out.println();
while(rs.next())
{
for(int i=1; i<5;i++){
System.out.print(rs.getString(i));
bwOutFile.append(rs.getString(i));
bwOutFile.append(System.getProperty("line.separator"));
}
bwOutFile.flush();
System.out.println();
}
conn.close();
}
catch(SQLException se)
{
se.printStackTrace();
}
catch(Exception e)
{
System.out.println("Unable to connect to database" +e);
}
}
}
I want the output to be separated by comma and each record in a line.
Any Help Please?!
Where are you using openCSV?
Here is one of the test from CSVWriter in openCSV that tests out writing records from ResultSet. Just modify it to meet your needs.
#Test
public void testResultSetWithHeaders() throws SQLException, IOException {
String[] header = {"Foo", "Bar", "baz"};
String[] value = {"v1", "v2", "v3"};
StringWriter sw = new StringWriter();
CSVWriter csvw = new CSVWriter(sw);
csvw.setResultService(new ResultSetHelperService());
ResultSet rs = MockResultSetBuilder.buildResultSet(header, value, 1);
int linesWritten = csvw.writeAll(rs, true); // don't need a result set since I am mocking the result.
assertFalse(csvw.checkError());
String result = sw.toString();
assertNotNull(result);
assertEquals("\"Foo\",\"Bar\",\"baz\"\n\"v1\",\"v2\",\"v3\"\n", result);
assertEquals(2, linesWritten);
}
I have a oracle(10.2) PLSQL procedure which fetches 15 records from a table in a sysrefcursor.
I then pass this cursor to a java class as a resultset. This java class is loaded to oracle.
Driver name : Oracle JDBC driver
Driver Version : 10.2.0.1.0
Driver Major Version : 10
Driver Minor Version : 2
Observations:
1 Inside the java class, when I iterate through the resultset I get only the first 10 records.
2 If the cursor fetched (20 or more records) or (10 or less) I could get all the records while iterating the resultset.
3 I found that the default fetchsize for the resultset is 10. If I change the fetchSize to 5, and the cursor fetches 8 records, I could get the first 5 records while iterating the resultset.
4 If the cursor fetched (10 or more records) or (5 or less) I could get all the records while iterating the resultset.
5 If I change the resultset fetchSize to 1, I could get all the records in the resultset no matter how many records are fetched by the cursor.
Why is the resultset behaving weirdly?
public static BLOB createZip(BLOB prevBlob, String outPrefix, ResultSet entries, ResultSet rs, Long[] resultRows) throws Exception
{
OracleConnection conn = null;
BLOB retBLOB = null;
int page = 1;
int curRow = 0;
long totalRows = 0;
try
{
conn = (OracleConnection) new OracleDriver().defaultConnection();
ArrayList entryList = loadEntries(entries);
retBLOB = BLOB.createTemporary(conn, true, BLOB.DURATION_SESSION);
retBLOB.open(BLOB.MODE_READWRITE);
OutputStream bOut = retBLOB.setBinaryStream(0L);
ZipOutputStream zipOut = new ZipOutputStream(bOut);
PrintStream out = new PrintStream(zipOut);
zipOut.putNextEntry(new ZipEntry(outPrefix + "-p" + page + ".csv"));
writeHeader(out, entryList);
while (rs.next())
{
curRow++;
totalRows++;
if (curRow >= maxPageSize)
{
zipOut.closeEntry();
page++;
zipOut.putNextEntry(new ZipEntry(outPrefix + "-p" + page + ".csv"));
writeHeader(out, entryList);
curRow = 0;
}
for (int i = 0; i < entryList.size(); i++)
{
Entry e = (Entry) entryList.get(i);
if (i != 0)
{
out.print(",");
}
if (e.isEscape())
out.print("\"" + escapeExcel(rs.getString(e.getColumn())) + "\"");
else
out.print("\"" + emptyExcel(rs.getString(e.getColumn())) + "\"");
}
out.println();
}
if (totalRows == 0)
{
out.println("\"No Entries Found\"");
}
resultRows[0] = new Long(totalRows);
out.flush();
zipOut.closeEntry();
if (prevBlob != null)
{
byte[] buf = new byte[1024];
InputStream bIn = prevBlob.binaryStreamValue();
ZipInputStream zipIn = new ZipInputStream(bIn);
ZipEntry inEntry = zipIn.getNextEntry();
while (inEntry != null)
{
zipOut.putNextEntry(new ZipEntry(inEntry.getName()));
int len;
while ((len = zipIn.read(buf)) > 0) {
out.write(buf, 0, len);
}
inEntry = zipIn.getNextEntry();
}
zipIn.close();
try
{
prevBlob.freeTemporary();
}
catch (SQLException e) { }
}
zipOut.close();
retBLOB.close();
return retBLOB;
}
catch (Exception sex)
{
if (retBLOB != null)
{
try
{
retBLOB.freeTemporary();
}
catch (SQLException e) { }
}
throw sex;
}
finally
{
try { entries.close(); } catch (SQLException sex) { }
try { rs.close(); } catch (SQLException sex) { }
try
{
if (conn != null || !conn.isClosed())
{
conn.close();
}
}
catch (SQLException ex)
{
ex.printStackTrace();
}
}
}
There is a workaround. I fetched the column index before while(rs.next()).
The below snippet works absolutely fine for me. But I still fail to understand why resultSet.getString(columnName); failed inside while(rs.next()).
ArrayList entryList = loadEntries(entries);
int[] colIndx = new int[entryList.size()];
ResultSetMetaData rsmd = rs.getMetaData();
int numberOfColumns = rsmd.getColumnCount();
for (int i = 0; i < entryList.size(); i++){
Entry e = (Entry) entryList.get(i);
for (int j = 1; j <= numberOfColumns; j++){
if(rsmd.getColumnName(j).equalsIgnoreCase(e.getColumn()))
colIndx[i] = j;
}
}
try{
while (rs.next()){
for (int i = 0; i < colIndx.length ; i++){
System.out.println("Column Values["+colIndx[i]+"] : "+rs.getString(colIndx[i]));
}
}
}