I have actually found this problem multiple times on Stackoverflow, but the solutions would not help me.
I have a chat module in my android app and want to persist the messages in my server db, which works fine until special characters like emojis appear.
ERROR: Incorrect string value: '\xF0\x9F\x98\x81' for column 'message' at row 1
...
...
Caused by: java.sql.SQLException: Incorrect string value: '\xF0\x9F\x98\x81' for column 'message' at row 1
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1084)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4232)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4164)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2615)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2776)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2838)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2082)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2334)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2262)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2246)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:187)
... 23 more
My environment is:
-Mysql 5.6
-Tomcat 8.0.8
-Hibernate 4.3.5
-JDK 1.8.0_05
This is the used table with the column in question, 'message':
These are my properties in the persistence.xml (version 2.1):
<property name="javax.persistence.jdbc.driver" value="com.mysql.jdbc.Driver" />
<property name="javax.persistence.jdbc.url" value="jdbc:mysql://localhost:3306/gamedb?useUnicode=true&characterEncoding=UTF-8" />
<property name="javax.persistence.jdbc.user" value="*********" />
<property name="javax.persistence.jdbc.password" value="**************" />
<property name="hibernate.hbm2ddl.auto" value="update" />
<property name="hibernate.dialect" value="org.hibernate.dialect.MySQL5Dialect" />
<property name="hibernate.connection.useUnicode" value="true" />
<property name="hibernate.connection.characterEncoding" value="utf8" />
Now I tried the following solutions without effect:
-Change datatype of 'message' from varchar to longtext
-Change collation of 'message' to utf8mb4
-Change collation of table to utf8mb4
-Append url with "?useUnicode=true&characterEncoding=UTF-8"
-Set character-set-server of mysql to utf8mb4
I think the emoji is correctly transmitted to the server, before it persists the message it broadcasts it back to the app and it gets correctly displayed.
I once had the same problem. I don't know a pretty solution but this had worked for me.
After I created the Session object I changed the connection collation by hand:
s.doReturningWork(new ReturningWork<Object>() {
#Override
public Object execute(Connection conn) throws SQLException
{
try(Statement stmt = conn.createStatement()) {
stmt.executeQuery("SET NAMES utf8mb4");
}
return null;
}
});
if you are using hibernate with c3p0, you can use c3p0 config connectionCustomizerClassName whitch you can set to a class doing work with connections c3p0 got.
example:
hibernate.cfg.xml
<property name="hibernate.c3p0.connectionCustomizerClassName">com.hzmoyan.newlyappserver.db.C3p0UseUtf8mb4</property>
C3p0UseUtf8mb4 class
public class C3p0UseUtf8mb4 extends AbstractConnectionCustomizer{
#Override
public void onAcquire(Connection c, String parentDataSourceIdentityToken)
throws Exception {
super.onAcquire(c, parentDataSourceIdentityToken);
try(Statement stmt = c.createStatement()) {
stmt.executeQuery("SET NAMES utf8mb4");
}
}
}
The solution is to use utf8mb4 rather than utf8 in MySQL. The blog post I linked to explains how to do just that.
I just found a nice little hack to get this to work without having to add any code. If you set the validation query to be SET NAMES utf8mb4 then it will execute this when it gets the connection, therefore setting the parameter each time a new connection is retrieved. You also need test on borrow to make this work.
Note I have found that this can sometimes take a few seconds to work after a restart so could have failures if you have records waiting to be processed on startup
So in your application.properties you would add something like
datasource.test-on-borrow=true
datasource.validation-query=SET NAMES utf8mb4
I was able to fix the issue by providing the following in my connection URL;
useUnicode=true&characterEncoding=UTF-8
Example;
jdbc:mysql://localhost/database?useUnicode=true&characterEncoding=UTF-8
Maybe you need to modify hibernate configuration as
<property name="hibernate.connection.characterEncoding" value="utf8mb4" />
If your datasource is org.apache.commons.dbcp.BasicDataSource you can set connectionInitSqls parameter
<bean id="dataSource"
class="org.apache.commons.dbcp.BasicDataSource"
p:driverClassName="${jdbc.driverClassName}"
p:url="${jdbc.databaseurl}"
p:username="${jdbc.username}"
p:password="${jdbc.password}"
p:testOnBorrow="true"
p:maxActive="1000"
p:testWhileIdle="true"
p:validationQuery="SELECT 1"
p:validationQueryTimeout="5">
<property name="connectionInitSqls">
<list>
<value>SET NAMES 'utf8mb4' COLLATE 'utf8mb4_unicode_ci'</value>
</list>
</property>
</bean>
After not receiving further answers to my comments I found an alternative solution: Base64.
Instead of teaching my DB to understand utf8mb4 I encode all critical messages to Base64 before storing them and decode them when retrieving them from the database.
Pro:
- Works great
- Libraries are already given for java and android
Contra:
- Base64 strings take up more space than the pure utf8mb4 strings (33%-36% more)
- May cost some performance encoding and decoding
Related
I'd like to configure LocalDatastoreServiceTestConfig such that queries will fail if a compound index is needed (e.g., a query with a sort on multiple properties). Is there a way to do this?
I tried new LocalDatastoreServiceTestConfig().setNoIndexAutoGen(true) but it had no effect.
(There is a corresponding way to do this with the Python SDK.)
I assume by "fail" you mean "throw an exception" or something similar.
If so, you should set the autoGenerate attribute in your WEB-INF/datastore-indexes.xml to false.
Example WEB-INF/datastore-indexes.xml:
<datastore-indexes autoGenerate="false">
</datastore-indexes>
Setting autoGenerate to false will make a query that requires a composite index throw an exception.
Example code:
try {
Query q = new Query("Action")
.addSort("encrypter", Query.SortDirection.ASCENDING)
.addSort("requester", Query.SortDirection.ASCENDING)
.addSort("time", Query.SortDirection.DESCENDING);
//...snip...
} catch (Exception e) {
log.severe(e.toString());
}
I tested this and got an exception logged as expected:
SEVERE: com.google.appengine.api.datastore.DatastoreNeedIndexException: Query com.google.appengine.api.datastore.dev.LocalCompositeIndexManager$IndexComponentsO
nlyQuery#f9f81ad3 requires a composite index that is not defined. You must update C:\appengine-java-sdk\dev\core1\war\WEB-INF\datastore-indexes.xml or enable au
toGenerate to have it automatically added.
The suggested index for this query is:
<datastore-index kind="Action" ancestor="false" source="manual">
<property name="encrypter" direction="asc"/>
<property name="requester" direction="asc"/>
<property name="time" direction="desc"/>
</datastore-index>
For more information, see datastore-indexes.xml reference.
In my application, I use a pre determined log messages, something like this:
MESSAGE_ID PROCESS_ID This process is running
I would like to use a JDBC Appender and split my message in three, to get put each part in a particular SQL column.
Is this possible? And How?
Thanks a lot!
One idea is to use the ThreadContext to carry the message ID and process ID, and declare jdbc columns for each item like this:
<Column name="MESSAGE" pattern="%message" />
<Column name="MESSAGE_ID" pattern="%X{messageID}" />
<Column name="PROCESS_ID" pattern="%X{processID}" />
In your code you set the values like this:
ThreadContext.put("messageID", UUID.randomUUID().toString();
ThreadContext.put("processID", getProcessId());
...
logger.debug("this process is running");
...
I need to map in Java with hibernate a table in sqlserver that has spaces in the column names.
I use Sqlserver 2008 jre7 com.microsoft.sqlserver.sqljdbc4 with Hibernate 3.3.2
I found a lot of similar questions in different forums but none of the solutions I read worked.
I tried using the backtick
<property name="nameSpaces" type="string" >
<column name="`Name with spaces`" not-null="false" />
</property>
this is the error
com.microsoft.sqlserver.jdbc.SQLServerException: Incorrect syntax near '`'.
I also tried to use the [] brakets but hibernate converted it in ` again.
<column name="[Name with spaces]" not-null="false" />
com.microsoft.sqlserver.jdbc.SQLServerException: Incorrect syntax near '`'.
Finally I also tried both of them
<column name="`[Name with spaces]`" not-null="false" />
or
without success.
Can someone help me? Is there any hibernate configuration I can change to work with these column names.
Thanks
The answer (by BilalDja) is hidden above in a comment. I found his answer helpful so I thought I would add the answer here for others with the same problem.
This issue is hard to debug, I had to dig really deep into the hibernate and jdbc code to finally find the "real" SQL statement that is created to get data from the database. I then copied its value and ran it manually on my SQL Server database. Straight away the syntax error was obvious and sure enough it was cause by a space in a column name.
To fix this I just escaped the double quotes around the problematic column name as explained in BilalDja's comments and expanded slightly here...
#Column(name = "\"Column name\"")
private String variableColumnWillMapTo;
That's it, problem solved, no more syntax error.
I have a sample application which loads 2 records ro database and then fetches all the records from DB and prints info from every record.
My problem is follow: when I run program, it fetches only records which inserted on this running, but not before, Also, when I'm opening DB with SQirrel, there is no such database.
Here is my Hibernate config:
<?xml version='1.0' encoding='utf-8'?>
<!DOCTYPE hibernate-configuration PUBLIC
"-//Hibernate/Hibernate Configuration DTD//EN"
"http://www.hibernate.org/dtd/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<property name="dialect">org.hibernate.dialect.HSQLDialect</property>
<property name="connection.driver_class">org.hsqldb.jdbcDriver</property>
<property name="connection.url">jdbc:hsqldb:hibertest</property>
<property name="connection.username">sk</property>
<property name="connection.password">0000</property>
<property name="show_sql">true</property>
<property name="current_session_context_class">thread</property>
<property name="hbm2ddl.auto">update</property>
<mapping class="models.Work" />
</session-factory>
</hibernate-configuration>
What I'm doing wrong?
I am not experienced with HSQLDB but try the following URL:
<property name="connection.url">jdbc:hsqldb:file:hibertest;shutdown=true</property>
See: Connection URL documentation.
I believe the default configuration of hsqldb is to not save data on disk for created tables. Please check the hsqldb documentation for how to change this behavior and make tables persistent.
http://hsqldb.org/doc/guide/guide.html#sgc_persist_tables
All the provided answers didn't help me at all.
Currently I use HSQLDB-2.3.2 and hibernate-core-3.5.6 with annotations-3.2.0
In addition to the answer of Tomasz, I needed to manually edit the database script file setting this:
SET WRITE_DELAY 0
You can persist the Data using the HSQL by doing the following,
1.
Connection c = DriverManager.getConnection("jdbc:hsqldb:file:/opt/db/testdb", "SA", "");
"/opt/db/testdb" here is the DB location in the disk.
2. CREATE **TEXT** TABLE <tablename> (<column definition> [<constraint definition>])
3. Assign the source for the Table SET TABLE mytable SOURCE "myfile;fs=|"
Ref: http://hsqldb.org/doc/2.0/guide/texttables-chapt.html
My Github Repository shows the sample https://github.com/Ayyamperumal/HSQL/tree/master/SampleHSQLdb
you can use "Database Development" perspective from Eclipse to see the data. (Supply hsql jar as the driver) Or Execute org.hsqldb.util.DatabaseManager from the hsqldb-*.jar to get the GUI for seeing the data.
I want to dump my test db into the raw xml file using dbunit and I'm getting ClassCastException. Below the code:
new FlatXmlWriter(new FileOutputStream("expected_ds.xml")).
write(getDbunitConnection().createDataSet(new String[]{"TAB1","TAB2"}));
and as a result :
java.lang.ClassCastException: org.apache.commons.dbcp.DelegatingResultSet cannot be cast to oracle.jdbc.OracleResultSet
I'm using ojdbc14-10.2.0.3.0.jar, commons-dbcp-1.2.2.jar and dbunit-2.4.7.jar.
Is that a bug in oracle jdbc ?
In the ojdbc driver I have found sth like that:
/**
*
* TODO UnitTests are completely missing
* #author Phil Barr
* #author Last changed by: $Author: jbhurst $
* #version $Revision: 1072 $ $Date: 2009-10-12 19:46:45 +0200 (lun, 12 ott 2009) $
* #since 2.4.0
*/
public class OracleXMLTypeDataType extends BlobDataType
{
public Object getSqlValue(int column, ResultSet resultSet) throws SQLException, TypeCastException
{
byte[] data = new byte[0];
OracleResultSet oracleResultSet = (OracleResultSet) resultSet;
... some other stuf ...
}
...
}
It looks like oracle issue and from the javadoc it seems that it was not tested at all.
Has anyone had similar problem?
The Oracle JDBC driver is not at fault here.
It looks like DBUnit assumes it can cast the ResultSet to the Oracle-specific type. That's a bad idea in the first place (but can't be avoided in some cases).
Since you're using a connection pool, DbUnit doesn't actually get access to the Oracle-specific object, but to a wrapper provided by the pool instead.
Either stop using a pool for the tests or get the underlying connection from the pooled connection and pass that to DBUnit (this risks that DBUnit closes the physical connection, which is what the pool tries to avoid by only providing the wrapper).
Another option is "Do Not Use Apache DBCP". Instead of Apache use Oracle Datasource
<bean id="dataSource" class="oracle.jdbc.pool.OracleDataSource" destroy-method="close">
<property name="URL" value="${test.db.url}" />
<property name="user" value="${test.db.username}"/>
<property name="password" value="${test.db.password}"/>
<property name="connectionCachingEnabled" value="true"/>
</bean>