We Keep Coding

Charset issues in ojdbc7/ojdbc8 vs. correct behaviour in ojdbc6

We have an Oracle database with the following charset settings
SELECT parameter, value FROM nls_database_parameters WHERE parameter like 'NLS%CHARACTERSET'
NLS_NCHAR_CHARACTERSET: AL16UTF16
NLS_CHARACTERSET: WE8ISO8859P15
In this database we have a table with a CLOB field, which has a record that starts with the following string, stored obviously in ISO-8859-15: X²ARB (here correctly converted to unicode, in particular that 2-superscript is important and correct).
Then we have the following trivial piece of code to get the value out, which is supposed to automatically convert the charset to unicode via globalization support in Oracle:
private static final String STATEMENT = "SELECT data FROM datatable d WHERE d.id=2562456";
public static void main(String[] args) throws Exception {
Class.forName("oracle.jdbc.driver.OracleDriver");
try (Connection conn = DriverManager.getConnection(DB_URL);
ResultSet rs = conn.createStatement().executeQuery(STATEMENT))
{
if (rs.next()) {
System.out.println(rs.getString(1).substring(0, 5));
}
}
}
Running the code prints:
with ojdbc8.jar and orai18n.jar: X�ARB -- incorrect
with ojdbc7.jar and orai18n.jar: X�ARB -- incorrect
with ojdbc-6.jar: X²ARB -- correct
By using UNISTR and changing the statement to SELECT UNISTR(data) FROM datatable d WHERE d.id=2562456 I can bring ojdbc7.jar and ojdbc8.jar to return the correct value, but this would require an unknown number of changes to the code as this is probably not the only place where the problem occurs.
Is there anything I can do to the client or server configurations to make all queries return correctly encoded values without statement modifications?

It definitely looks like a bug in the JDBC thin driver (I assume you're using thin). It could be related to LOB prefetch where the CLOB's length, character set id and the first part of the LOB data is sent inband. This feature was introduced in 11.2. As a workaround, you can disable lob prefetch by setting the connection property
oracle.jdbc.defaultLobPrefetchSize
to "-1". Meanwhile I'll follow up on this bug to make sure that it gets fixed.

Please have a look at Database JDBC Developer's Guide - Globalization Support
The basic Java Archive (JAR) file ojdbc7.jar, contains all the
necessary classes to provide complete globalization support for:
CHAR or VARCHAR data members of object and collection for the character sets US7ASCII, WE8DEC, WE8ISO8859P1, WE8MSWIN1252, and UTF8.
To use any other character sets in CHAR or VARCHAR data members of
objects or collections, you must include orai18n.jar in the CLASSPATH
environment variable:
ORACLE_HOME/jlib/orai18n.jar

Fragmented rows in firebird embedded mode

I was doing this code, and something weird has happened.
I make a bulk insert from an external file. But the result it's just fragmented, or maybe corrupted.
cnx=factoryInstace.getConnection();
pstmt = cnx.prepareStatement("DELETE FROM TEMPCELULAR");
pstmt.executeUpdate();
pstmt = cnx.prepareStatement("EXECUTE BLOCK AS BEGIN if (exists(select 1 from rdb$relations where rdb$relation_name = 'EXT_TAB')) then execute statement 'DROP TABLE EXT_TAB;'; END");
pstmt.executeUpdate();
pstmt = cnx.prepareStatement("CREATE TABLE EXT_TAB EXTERNAL '"+txtarchivoProcesar.getText()+"'(CELULAR varchar(11))");
pstmt.executeUpdate();
pstmt = cnx.prepareStatement("INSERT INTO TEMPCELULAR (CELULAR)SELECT CELULAR FROM EXT_TAB");
pstmt.executeUpdate();
pstmt = cnx.prepareStatement("SELECT CELULAR FROM TEMPCELULAR");
ResultSet rs=pstmt.executeQuery();
while(rs.next()){
System.out.println("::"+rs.getString(1));
}
And now, all of a sudden the rows on my table look like this:
::c#gmail.com
::abc2#gmail.
::m
abc3#gma
::.com
abc4#
::ail.com
ab
::#gmail.com
::bc6#gmail.c
::abc7#gmai
::com
abc8#g
::il.com
abc
::gmail.com
::c10#gmail.c
::
The blank spaces between results were not made by me. This is the result as it is.
Source file for external table:
abc#gmail.com
abc2#gmail.com
abc3#gmail.com
abc4#gmail.com
abc5#gmail.com
abc6#gmail.com
abc7#gmail.com
abc8#gmail.com
abc9#gmail.com
abc10#gmail.com
sneciosup#hotmail.com
¿What's wrong with my code?
I've haven't seen this wack results in years.
The database is created of the users pc on the first run. Hence, while in production every time I run the program.
Any help, will be appreciated.

The external table file in Firebird is not just a plaintext file, it is a fixed width format with special requirements to the content and layout. See the Interbase 6.0 Data Definition Guide, page 107-111 (available for download from http://www.firebirdsql.org/en/reference-manuals/ ) or The Firebird Book by Helen Borrie page 281-287.
The problems I see right now are:
you declare the column in the external table to be VARCHAR(11),
while the shortest emailaddress in your file is 13 characters, the
longest is 21 characters, so Firebird will never be able to read a
full emailaddress from the file
you don't specify a separate column for your linebreaks, so your
linebreaks will simply be part of the data which is read
you have declared the column as VARCHAR, this requires the records
to have a very specific format, where the first two bytes declare
the actual data length followed by a string of that length (and even then it
only reads upto the declared length of the column). Either make sure
you follow the requirements for VARCHAR columns, or simply use the
CHAR datatype and make sure you pad the column with spaces upto the
declared length.
I am not 100% sure, but your default database characterset may also be involved in how the data is read.

How to compare utf8 text in mysql query using java

I have a problem while selecting from a table containg data in utf-8 format in MySQL using java, in the WHERE clause I need to compare if a column value equals to a java String but they don't match
"Select age from student where name = '"+stringVariable+"';"
the stringVariable can sometimes be in arabic so in this case they don't match
The database is utf8mb4 and the connection also between java and database is utf_8 and I dont have a problem while inserting or just selecting data but the problem occurs while comparing
I tried to convert the string like this and it also didnt match
byte[] b = stringVariable.getBytes("UTF-8");
String str = new String(b,"UTF-8");
So anyone has any solution for that ?!
Thank you in advance

Use parameters. The driver should then encode them correctly according to the connection properties.
PreparedStatement statement = connection.prepareStatement(
"select age from student where name = ?");
statement.setString(1, stringVariable);
In addition, this also correctly escapes special SQL characters (like single quotes).

You shouldn't have to transform anything. The driver takes care of everything. Try using a prepared statement rather than string concatenation. This will have the additional advantage of avoiding SQL injection attacks, and make sure your statement works even if the string variable contains a quote.

I solved this problem it was not only related with the database query but I get the string from ajax request so I had to decode it from utf8 first and then get the parameter value using
URLDecoder.decode(request.getQueryString(), "UTF-8")
and then pass it to the query and I used prepared statement which encodes it according to the connection properties as you mentioned
Thank you

Converting string encoding in java

Related to this question: "Fix" String encoding in Java
My project encoding is UTF-8.
I need to make a query to a DB that uses a particular varchar encoding (apparently EUC-KR).
I take the input as UTF-8, and I want to make the DB query with the EUC-KR encoded version of that string.
First of all, I can select and display the encoded strings using the following:
ResultSet rs = stmt.executeQuery("SELECT name FROM mytable");
while(rs.next())
System.out.println(new String(rs.getBytes(1), "EUC-KR"));
I want to do something like:
PreparedStatement ps = conn.prepareStatement("SELECT * FROM MYTABLE WHERE NAME=?");
ps.setString(1,input);
ResultSet rs = ps.executeQuery();
Which obviously won't work, because my Java program is not using the same encoding as the DB. So, I've tried replacing the middle line with each of the following, to no avail:
ps.setString(1,new String(input.getBytes("EUC-KR")));
ps.setString(1,new String(input.getBytes("EUC-KR"), "EUC-KR"));
ps.setString(1,new String(input.getBytes("UTF-8"), "EUC-KR"));
ps.setString(1,new String(input.getBytes("EUC-KR"), "UTF-8"));
I am using Oracle 10g 10.1.0
More details of my attempts follow:
What does seem to work is saving the name from the first query into a string without any other manipulation, and passing that back as a parameter. It matches itself.
That is,
ResultSet rs = stmt.executeQuery("SELECT name FROM mytable");
rs.next();
String myString = rs.getString(1);
PreparedStatement ps = conn.prepareStatement("SELECT * FROM mytable WHERE name=?");
ps.setString(1, myString);
rs = ps.executeQuery();
... will result with the 1 correct entry in rs. Great, so now I just need to convert my input to whatever format that thing seems to be in.
However, nothing I have tried seems to match the "correct" string when I try reading their bytes using
byte[] mybytearray = myString.getBytes();
for(byte b : mybytearray)
System.out.print(b+" ");
In other words, I can turn °í»ê into 고산 but I can't seem to turn 고산 into °í»ê.
The byte array given by
rs.getBytes(1)
is different from the byte array given by any of the following:
rs.getString(1).getBytes()
rs.getString(1).getBytes("UTF8")
rs.getString(1).getBytes("EUC-KR")
Unhappiness: it turns out that for my DB, NLS_CHARACTERSET = US7ASCII
Which means that what I'm trying to do is unsupported. Thanks for playing everyone :(

You can't accomplish anything with a String constructor. String is always UTF-16 inside. Converting UTF-16 chars to EUC-KR and back again won't help you.
Putting invalid Unicode into String values in the hopes that they will then be converted to EUC-KR is a really bad idea.
What you are doing is supposed to 'just work'. The oracle driver is supposed to talk to the server, find out the desired charset, and go from there.
What, however, is the database charset? If someone is storing EUC-KR without having set the charset to EUC-KR, you are more or less up a creek.
What you need to do is to tell your jdbc driver what charset to use to communicate with the server. You haven't mentioned if you are using Thin or OCI, the answer might be different.
Judging from http://download.oracle.com/docs/cd/E14072_01/appdev.112/e13995/oracle/jdbc/OracleDriver.html, you might want to try turning on defaultNChar.
In general, it's the job of the jdbc driver to transcode String to what the Oracle server wants. You may need tnsnames.ora options if you are using 'OCI'.
edit
OP reports that the nls_charset of the database is US7ASCII. That means that all JDBC drivers will think that it is their job to convert Unicode String values to ASCII. Korean characters will be reduced to ? at best. Officially, then, your are up a creek.
There are some possible tricks to try. One is the very dangerous trick of
new String(string.getBytes("EUC-KR"), "ascii")
that will try to make a string of Unicode chars that just so happens to have the values of EUC-KR in their low bytes. My belief is that this will corrupt data, but you could experiment.
Or, perhaps, ps.setBytes(n, string.getBytes("EUC-KR")), but I myself do not know if Oracle defines the conversion of bytes to chars as a binary copy. It might. Or, perhaps, adding a stored proc that takes a blob as an argument.
Really, what's called for here is to repair the database to use an nls_charset of UTF-8 or EUC-KR, but that's a whole other job.

Have you looked at the correct name for the charset ? Maybe you should be using UTF8 and EUC_KR ..
http://download.oracle.com/javase/1.4.2/docs/guide/intl/encoding.doc.html

Hopefully this is not a stupid answer but have you made sure that charsets.jar is in your classpath. It is NOT by default see this page for more...
The charsets.jar file is an optional feature of the JRE. To install it, you must choose the "custom installation" and select the "Support for additional locales" feature.

JDBC DatabaseMetaData.getColumns() returns duplicate columns

I'm busy on a piece of code to get alle the column names of a table from an Oracle database. The code I came up with looks like this:
DriverManager.registerDriver (new oracle.jdbc.driver.OracleDriver());
Connection conn = DriverManager.getConnection(
"jdbc:oracle:thin:#<server>:1521:<sid>", <username>, <password>);
DatabaseMetaData meta = conn.getMetaData();
ResultSet columns = meta.getColumns(null, null, "EMPLOYEES", null);
int i = 1;
while (columns.next())
{
System.out.printf("%d: %s (%d)\n", i++, columns.getString("COLUMN_NAME"),
columns.getInt("ORDINAL_POSITION"));
}
When I ran this code to my surprise too many columns were returned. A closer look revealed that the ResultSet contained a duplicate set of all the columns, i.e. every column was returned twice. Here's the output I got:
1: ID (1)
2: NAME (2)
3: CITY (3)
4: ID (1)
5: NAME (2)
6: CITY (3)
When I look at the table using Oracle SQL Developer it shows that the table only has three columns (ID, NAME, CITY). I've tried this code against several different tables in my database and some work just fine, while others exhibit this weird behaviour.
Could there be a bug in the Oracle JDBC driver? Or am I doing something wrong here?
Update: Thanks to Kenster I now have an alternative way to retrieve the column names. You can get them from a ResultSet, like this:
DriverManager.registerDriver (new oracle.jdbc.driver.OracleDriver());
Connection conn = DriverManager.getConnection("jdbc:oracle:thin:#<server>:1521:<sid>", <username>, <password>);
Statement st = conn.createStatement();
ResultSet rset = st.executeQuery("SELECT * FROM \"EMPLOYEES\"");
ResultSetMetaData md = rset.getMetaData();
for (int i=1; i<=md.getColumnCount(); i++)
{
System.out.println(md.getColumnLabel(i));
}
This seems to work just fine and no duplicates are returned! And for those who wonder: according to this blog you should use getColumnLabel() instead of getColumnName().

In oracle, Connection.getMetaData() returns meta-data for the entire database, not just the schema you happen to be connected to. So when you supply null as the first two arguments to meta.getColumns(), you're not filtering the results for just your schema.
You need to supply the name of the Oracle schema to one of the first two parameters of meta.getColumns(), probably the second one, e.g.
meta.getColumns(null, "myuser", "EMPLOYEES", null);
It's a bit irritating having to do this, but that's the way the Oracle folks chose to implement their JDBC driver.

This doesn't directly answer your question, but another approach is to execute the query:
select * from tablename where 1 = 0
This will return a ResultSet, even though it doesn't select any rows. The result set metadata will match the table that you selected from. Depending on what you're doing, this can be more convenient. tablename can be anything that you can select on--you don't have to get the case correct or worry about what schema it's in.

In the update to your question I noticed that you missed one key part of Kenster's answer. He specified a 'where' clause of 'where 1 = 0', which you don't have. This is important because if you leave it off, then oracle will try and return the ENTIRE table. And if you don't pull all of the records over, oracle will hold unto them, waiting for you to page through them. Adding that where clause still gives you the metadata, but without any of the overhead.
Also, I personally use 'where rownum < 1', since oracle knows immediately that all rownums are past that, and I'm not sure if it's smart enough to not try and test each record for '1 = 0'.

In addition to skaffman's answer -
use the following query in Oracle:
select sys_context( 'userenv', 'current_schema' ) from dual;
to access your current schema name if you are restricted to do so in Java.

This is the behavior mandated by the JDBC API - passing nulls as first and second parameter to getColumns means that neither catalog name nor schema name are used to narrow the search.
Link to the documentation . It is true that some other JDBC drivers have different behavior by default (e.g MySQL's ConnectorJ by default restricts to the current catalog), but this is not standard, and documented as such