We Keep Coding

Java writing CSV files to be loaded into SQL Server, format incorrect

Attempting to develop a script which will output a CSV file. Initially I was using CSVWriter, but switched to BufferedWriter in an attempting to debug the problem.
I have attempting many iterations of this CSV output, however they all seem to break on a row which has a very large amount of text >16000 characters in some cases. Everything looks fine when I attempt to open in notepad++, but when I attempt to load into SQL server to bring into excel it breaks my last column.
String[] row = null;
//change all " to "" in order to escape properly
tmp = tmp.replace("\"","\"\"");
bw.write(row[0]+","+row[1]+","+row[2]+","+row[3]+","+row[4]+","+row[16]+","+"\""+tmp+"\""+ ";" );
Above is what I am performing at the moment. However I attempted using the CSVWriter which would wrap every field in a "", this did not load.
Does anyone have experience with this?

How to write file in Windows "Text Mode"? - And what is the difference?

I was trying to import a CSV file into SQL Server using bulk insert.
Now I was repeatedly getting an error that
The bulk load failed. The column is too long in the data file for row 1, column 6. Verify that the field terminator and row terminator are specified correctly.
However, first I ran some java code that used regex to remove all unprintable chars and put a "\n" explicitly at the end of each line (and inserted the whole data into a new file). But even after this, the import was giving the above error.
However I opened the file in Textpad and saw that it was saved as a "Unix" file-- I just changed this to "Windows" file format, and then saved it-- and after that the import worked!
What I wish to know is the difference between the Unix and Windows file formats, so that I can take care of this issue in my java code itself, rather than open the Unix file and save it again in Windows file format.
EDIT--
This is the code that is giving the error mentioned above---
BULK INSERT GooglePatentsIndividualDec2012.dbo.patent
FROM 'C:\Arvind_gpd\patents\4patents_corrected.csv'
WITH
(
DATAFILETYPE = 'char',
FIELDTERMINATOR = '^',
ROWTERMINATOR = '\n'
);

Save result of XSQL query to a file in Java

I want to Save result of an XSQL query to a file using Java.
Does any one know a way to do this?

The Oracle document on Using XSQL in Java Programs has instructions on how to call an XSQL from Java and get the result as an XMLDocument or send it to a PrintWriter or OutputStream. There's a short example program there that sends the result to System.out, but it could be easily modified to send it to a file.

Import CSV file to oracle DB

Is there a simple Java library or approach that will take a SQL query and load data in a CSV file to oracle database. Pls help

You don't have to use Java to load a data file into a table unless it is absolutely necessary. Instead, I'd recommend Oracle's command-line SQL*Loader utility which was designed specially for this purpose.

For similar tasks I usually use Groovy scripts as it's really easy and quick to write and runs on the JVM off course.
...an example:
import groovy.sql.Sql
def file1 = new File(/C:\Documents and Settings\USER\Desktop\Book1.csv/)
def reader = new FileReader(file1)
def sql = Sql.newInstance("jdbc:oracle:thin:#XXXXXX:XXXX:XXX", "SCHEMA",
"USER", "oracle.jdbc.driver.OracleDriver")
reader.each { line ->
fields = line.split(';')
sql.executeInsert("insert into YOUR_TABLE values(${fields[0]},${fields[1]},${fields[2]})")
}
It's a basic example, if you have double quotes and semi columns in your csv you will probably want to use something like OpenCSV to handle that.

You could transform each line in the CSV with regular expressions, to make an insert query, and then send to Oracle (with JDBC).

I think this tool will help you for any type of database import-export problem.
http://www.dmc-fr.com/home_en.php

Do you have that CSV in a file on the database server or can you store it there? Then you may try to have Oracle open it by declaring a DIRECTORY object for the path the file is in and then create an EXTERNAL TABLE which you can query in SQL afterwards. Oracle does the parsing of the file for you.

If you are open to Python you can do bulk load using SQL*Loader
loadConf=('sqlldr userid=%s DATA=%s control=%s LOG=%s.log BAD=%s.bad DISCARD=%s.dsc' % (userid,datafile, ctlfile,fn,fn,fn)).split(' ')
p = Popen(loadConf, stdin=PIPE, stdout=PIPE, stderr=PIPE, shell=False, env=os.environ)
output, err = p.communicate()
It's will be much faster that row insert.
I uploaded basic working example here.

How do I read Unicode characters from an MS Access 2007 database through Java?

In Java, I have written a program that reads a UTF8 text file. The text file contains a SQL query of the SELECT kind. The program then executes the query on the Microsoft Access 2007 database and writes all fields of the first row to a UTF8 text file.
The problem I have is when a row is returned that contains unicode characters, such as "♪". These characters show up as "?" in the text file.
I know that the text files are read and written correctly, because a dummy UTF8 character ("◎") is read from the text file containing the SQL query and written to the text file containing the resulting row. The UTF8 character looks correct when the written text file is opened in Notepad, so the reading and writing of the text files are not part of the problem.
This is how I connect to the database and how I execute the SQL query:
Connection c = DriverManager.getConnection("jdbc:odbc:Driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=C:/database.accdb;Pwd=temp");
ResultSet r = c.createStatement().executeQuery(sql);
I have tried making a charSet property to the Connection but it makes no difference:
Properties p = new Properties();
p.put("charSet", "utf-8");
p.put("lc_ctype", "utf-8");
p.put("encoding", "utf-8");
Connection c = DriverManager.getConnection("...", p);
Tried with "utf8"/"UTF8"/"UTF-8", no difference. If I enter "UTF-16" I get the following exception:
java.lang.IllegalArgumentException: Illegal replacement
Been searching around for hours with no results and now turn my hope to you. Please help!
I also accept workaround suggestions. =) What I want to be able to do is to make a Unicode query (for example one that searches for posts that contain the "あ" character) and to have results with Unicode characters receieved and saved correctly.
Thank you!
Update. Here is a self-contained example of the issue:
package test;
import java.io.BufferedReader;import java.io.File;import java.io.FileInputStream;import java.io.FileOutputStream;import java.io.InputStreamReader;import java.io.OutputStreamWriter;import java.nio.charset.Charset;import java.sql.Connection;import java.sql.DriverManager;import java.sql.ResultSet;import java.util.Properties;
public class Standalone {
public static void main(String[] args) {
try {
Properties p = new Properties();
p.put("charSet", "UTF8");
Connection c = DriverManager.getConnection("jdbc:odbc:Driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=./dummy.accdb;Pwd=pass", p);
ResultSet r = c.createStatement().executeQuery("SELECT TOP 1 * FROM main;");
r.next();
OutputStreamWriter osw = new OutputStreamWriter(new FileOutputStream(new File("results.txt")), Charset.forName("UTF-8"));
osw.write(new BufferedReader(new InputStreamReader(new FileInputStream("utf8.txt"), Charset.forName("UTF-8"))).readLine() +" : "+ r.getString("content"));
osw.close();
c.close();
System.out.println("Done.");
} catch (Exception e) {
e.printStackTrace();
}
}
}
What the example does is that it opens the database "dummy.accdb" encrypted with the password "pass" and pulls the first post out of the table "main". It then reads the text file "utf8.txt" and writes a text file "results.txt" which will contain the first row of "utf8.txt" plus the value of the field "content" it got from the database.
In the file "utf8.txt" I have stored "♜♞♝♛♚♝♞♜♟♖♘♗♕♔♗♘♖♙".
In the database's "main" table's "content" field I have stored "♫♪あｷﾀℳℴℯ♥∞۞♀♂".
After the application has finished running the "results.txt" has the following content: "♜♞♝♛♚♝♞♜♟♖♘♗♕♔♗♘♖♙ : ?????Moe?8???".
It successfully read and write the UTF8 characters of the "utf8.txt" text file, but failed to obtain the correct characters from the database. This is where the problem lies.
Update. Thought I should mention that the field in the database is of the type "memo", I have tried havig "Unicode Compression" set both to "No" and to "Yes" (recreating the post between tries to make sure no compression were there when "No" was selected). To my understanding Access uses UTF-16 when it saves Unicode characters, however with compression on it changes to UTF-8. In any case this did not make any difference.
Bonus question, anyone know how to connect to the database using a pure ODBC provider in Java? Or any other kind of method? This would provide me with a good workaround.
Update. I have been trying to feed these four to getConnection:
"Provider=Microsoft.Jet.OLEDB.4.0;Data Source=./dummy.accdb"
"jdbc:odbc:Provider=Microsoft.Jet.OLEDB.4.0;Data Source=./dummy.accdb"
"jdbc:odbc:Driver={Microsoft.Jet.OLEDB.4.0};Data Source=./dummy.accdb"
"jdbc:odbc:Provider=Microsoft.ACE.OLEDB.12.0;Data Source=./dummy.accdb"
The first give the error "java.sql.SQLException: No suitable driver found for Provider=Microsoft.Jet.OLEDB.4.0;Data Source=./dummy.accdb" and the two in the middle gets "java.sql.SQLException: [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified". The last one gets "java.sql.SQLException: [Microsoft][ODBC Driver Manager] Data source name too long".
I don't understand what getConnection wants. The parameter description is as follows: "url - a database url of the form jdbc:subprotocol:subname". Huh? I clearly don't get what that means.
Anyone know any alternative working ways of connecting to the Access 2007 database through Java? Maybe the providers I tried aren't supported but some other might be?

An alternative to using jdbc with Access is the Jackcess library. the pro of using this library is that there are no problematic translation layers (like odbc) or any external system dependencies. the con of using this library is that it does not support executing queries.

Since you mentioned switching to some other DB than Access is possible, I urge you to do so. Making your software on Microsoft Office products has always been a maintenance nightmare for me, so choose anything else from this list: http://java-source.net/open-source/database-engines.
I would go with Apache Derby for this, or just use the Java Database JavaDB that comes preinstalled with any current Sun Java Installation (and is in fact a repackaged Derby DB)

Now that the JDBC-ODBC Bridge has been dropped from Java SE 8 and Oracle has confirmed that this issue will never be fixed (ref: here) a good alternative would be to use UCanAccess. For more information, see
UCanAccess on SourceForge
Manipulating an Access database from Java without ODBC

utf-8 is not supported:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6345277