We Keep Coding

Is there a way to configure tests for the Java Google App Engine SDK to fail if a compound index is needed?

I'd like to configure LocalDatastoreServiceTestConfig such that queries will fail if a compound index is needed (e.g., a query with a sort on multiple properties). Is there a way to do this?
I tried new LocalDatastoreServiceTestConfig().setNoIndexAutoGen(true) but it had no effect.
(There is a corresponding way to do this with the Python SDK.)

I assume by "fail" you mean "throw an exception" or something similar.
If so, you should set the autoGenerate attribute in your WEB-INF/datastore-indexes.xml to false.
Example WEB-INF/datastore-indexes.xml:
<datastore-indexes autoGenerate="false">
</datastore-indexes>
Setting autoGenerate to false will make a query that requires a composite index throw an exception.
Example code:
try {
Query q = new Query("Action")
.addSort("encrypter", Query.SortDirection.ASCENDING)
.addSort("requester", Query.SortDirection.ASCENDING)
.addSort("time", Query.SortDirection.DESCENDING);
//...snip...
} catch (Exception e) {
log.severe(e.toString());
}
I tested this and got an exception logged as expected:
SEVERE: com.google.appengine.api.datastore.DatastoreNeedIndexException: Query com.google.appengine.api.datastore.dev.LocalCompositeIndexManager$IndexComponentsO
nlyQuery#f9f81ad3 requires a composite index that is not defined. You must update C:\appengine-java-sdk\dev\core1\war\WEB-INF\datastore-indexes.xml or enable au
toGenerate to have it automatically added.
The suggested index for this query is:
<datastore-index kind="Action" ancestor="false" source="manual">
<property name="encrypter" direction="asc"/>
<property name="requester" direction="asc"/>
<property name="time" direction="desc"/>
</datastore-index>
For more information, see datastore-indexes.xml reference.

Can't persist emojis with mysql and hibernate

I have actually found this problem multiple times on Stackoverflow, but the solutions would not help me.
I have a chat module in my android app and want to persist the messages in my server db, which works fine until special characters like emojis appear.
ERROR: Incorrect string value: '\xF0\x9F\x98\x81' for column 'message' at row 1
...
...
Caused by: java.sql.SQLException: Incorrect string value: '\xF0\x9F\x98\x81' for column 'message' at row 1
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1084)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4232)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4164)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2615)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2776)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2838)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2082)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2334)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2262)
at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2246)
at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:187)
... 23 more
My environment is:
-Mysql 5.6
-Tomcat 8.0.8
-Hibernate 4.3.5
-JDK 1.8.0_05
This is the used table with the column in question, 'message':
These are my properties in the persistence.xml (version 2.1):
<property name="javax.persistence.jdbc.driver" value="com.mysql.jdbc.Driver" />
<property name="javax.persistence.jdbc.url" value="jdbc:mysql://localhost:3306/gamedb?useUnicode=true&characterEncoding=UTF-8" />
<property name="javax.persistence.jdbc.user" value="*********" />
<property name="javax.persistence.jdbc.password" value="**************" />
<property name="hibernate.hbm2ddl.auto" value="update" />
<property name="hibernate.dialect" value="org.hibernate.dialect.MySQL5Dialect" />
<property name="hibernate.connection.useUnicode" value="true" />
<property name="hibernate.connection.characterEncoding" value="utf8" />
Now I tried the following solutions without effect:
-Change datatype of 'message' from varchar to longtext
-Change collation of 'message' to utf8mb4
-Change collation of table to utf8mb4
-Append url with "?useUnicode=true&characterEncoding=UTF-8"
-Set character-set-server of mysql to utf8mb4
I think the emoji is correctly transmitted to the server, before it persists the message it broadcasts it back to the app and it gets correctly displayed.

I once had the same problem. I don't know a pretty solution but this had worked for me.
After I created the Session object I changed the connection collation by hand:
s.doReturningWork(new ReturningWork<Object>() {
#Override
public Object execute(Connection conn) throws SQLException
{
try(Statement stmt = conn.createStatement()) {
stmt.executeQuery("SET NAMES utf8mb4");
}
return null;
}
});

if you are using hibernate with c3p0, you can use c3p0 config connectionCustomizerClassName whitch you can set to a class doing work with connections c3p0 got.
example:
hibernate.cfg.xml
<property name="hibernate.c3p0.connectionCustomizerClassName">com.hzmoyan.newlyappserver.db.C3p0UseUtf8mb4</property>
C3p0UseUtf8mb4 class
public class C3p0UseUtf8mb4 extends AbstractConnectionCustomizer{
#Override
public void onAcquire(Connection c, String parentDataSourceIdentityToken)
throws Exception {
super.onAcquire(c, parentDataSourceIdentityToken);
try(Statement stmt = c.createStatement()) {
stmt.executeQuery("SET NAMES utf8mb4");
}
}
}

The solution is to use utf8mb4 rather than utf8 in MySQL. The blog post I linked to explains how to do just that.

I just found a nice little hack to get this to work without having to add any code. If you set the validation query to be SET NAMES utf8mb4 then it will execute this when it gets the connection, therefore setting the parameter each time a new connection is retrieved. You also need test on borrow to make this work.
Note I have found that this can sometimes take a few seconds to work after a restart so could have failures if you have records waiting to be processed on startup
So in your application.properties you would add something like
datasource.test-on-borrow=true
datasource.validation-query=SET NAMES utf8mb4

I was able to fix the issue by providing the following in my connection URL;
useUnicode=true&characterEncoding=UTF-8
Example;
jdbc:mysql://localhost/database?useUnicode=true&characterEncoding=UTF-8

Maybe you need to modify hibernate configuration as
<property name="hibernate.connection.characterEncoding" value="utf8mb4" />

If your datasource is org.apache.commons.dbcp.BasicDataSource you can set connectionInitSqls parameter
<bean id="dataSource"
class="org.apache.commons.dbcp.BasicDataSource"
p:driverClassName="${jdbc.driverClassName}"
p:url="${jdbc.databaseurl}"
p:username="${jdbc.username}"
p:password="${jdbc.password}"
p:testOnBorrow="true"
p:maxActive="1000"
p:testWhileIdle="true"
p:validationQuery="SELECT 1"
p:validationQueryTimeout="5">
<property name="connectionInitSqls">
<list>
<value>SET NAMES 'utf8mb4' COLLATE 'utf8mb4_unicode_ci'</value>
</list>
</property>
</bean>

After not receiving further answers to my comments I found an alternative solution: Base64.
Instead of teaching my DB to understand utf8mb4 I encode all critical messages to Base64 before storing them and decode them when retrieving them from the database.
Pro:
- Works great
- Libraries are already given for java and android
Contra:
- Base64 strings take up more space than the pure utf8mb4 strings (33%-36% more)
- May cost some performance encoding and decoding

AppEngine - No Indexes Serving

I have been trying to use Query.setOrdering(xxxx desc) but AppEngine gives ServerError : Unable to process the request.
Admin Console :
com.google.appengine.api.datastore.DatastoreNeedIndexException: no matching index found.
The suggested index for this query is:
<datastore-index kind="Chat_messages" ancestor="false" source="manual">
<property name="FROM_TO" direction="asc"/>
<property name="TIME" direction="desc"/>
</datastore-index>
My code in servlet :
Query q=pm.newQuery(Chat_messages.class);
q.setFilter("FROM_TO == email_emailto || FROM_TO == emailto_email");
q.declareParameters("String email_emailto,String emailto_email");
q.setRange(0, 50);
q.setOrdering("TIME desc");

With the Python runtime, the process of running the app locally with the dev server generates the index file.

Are you actually getting an error about not having the correct index? The index that you have put in your datastore-index.xml is a single property index. This means that it is automatically built by Datastore and will not show up in your console.
You can see this page which describes which queries you can perform without needing to upload any indexes.

HSQLDB with Hibernate not saving records after application end

I have a sample application which loads 2 records ro database and then fetches all the records from DB and prints info from every record.
My problem is follow: when I run program, it fetches only records which inserted on this running, but not before, Also, when I'm opening DB with SQirrel, there is no such database.
Here is my Hibernate config:
<?xml version='1.0' encoding='utf-8'?>
<!DOCTYPE hibernate-configuration PUBLIC
"-//Hibernate/Hibernate Configuration DTD//EN"
"http://www.hibernate.org/dtd/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<property name="dialect">org.hibernate.dialect.HSQLDialect</property>
<property name="connection.driver_class">org.hsqldb.jdbcDriver</property>
<property name="connection.url">jdbc:hsqldb:hibertest</property>
<property name="connection.username">sk</property>
<property name="connection.password">0000</property>
<property name="show_sql">true</property>
<property name="current_session_context_class">thread</property>
<property name="hbm2ddl.auto">update</property>
<mapping class="models.Work" />
</session-factory>
</hibernate-configuration>
What I'm doing wrong?

I am not experienced with HSQLDB but try the following URL:
<property name="connection.url">jdbc:hsqldb:file:hibertest;shutdown=true</property>
See: Connection URL documentation.

I believe the default configuration of hsqldb is to not save data on disk for created tables. Please check the hsqldb documentation for how to change this behavior and make tables persistent.
http://hsqldb.org/doc/guide/guide.html#sgc_persist_tables

All the provided answers didn't help me at all.
Currently I use HSQLDB-2.3.2 and hibernate-core-3.5.6 with annotations-3.2.0
In addition to the answer of Tomasz, I needed to manually edit the database script file setting this:
SET WRITE_DELAY 0

You can persist the Data using the HSQL by doing the following,
1.
Connection c = DriverManager.getConnection("jdbc:hsqldb:file:/opt/db/testdb", "SA", "");
"/opt/db/testdb" here is the DB location in the disk.
2. CREATE **TEXT** TABLE <tablename> (<column definition> [<constraint definition>])
3. Assign the source for the Table SET TABLE mytable SOURCE "myfile;fs=|"
Ref: http://hsqldb.org/doc/2.0/guide/texttables-chapt.html
My Github Repository shows the sample https://github.com/Ayyamperumal/HSQL/tree/master/SampleHSQLdb
you can use "Database Development" perspective from Eclipse to see the data. (Supply hsql jar as the driver) Or Execute org.hsqldb.util.DatabaseManager from the hsqldb-*.jar to get the GUI for seeing the data.

Basic HQL statement always fails

I've recently started trying to use Hibernate, but am doing so in Netbeans. This has left me having to use this example project to try and get me up and running.
Unfortunately, at the step "Enumerating Film Titles and Retrieving Actors Using an HQL Query" my HQL queries do not give results and instead fail, with the exception:
org.hibernate.exception.SQLGrammarException: could not execute query
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'from limit 100' at line 1
It seems that the HQL isn't generating proper MySQL statements but I can't seem to see why, as I've followed the example to the letter thus far.
I am attempting to connect to a local MySQL database named 'sakila', with the following details:
jdbc:mysql://localhost:3306/sakila
which seems to work correctly as I am able to browse the tables from within Netbeans no problem.
My hibernate.cfg.xml is as follows:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE hibernate-configuration PUBLIC "-//Hibernate/Hibernate Configuration DTD 3.0//EN" "http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">
<hibernate-configuration>
<session-factory>
<property name="hibernate.dialect">org.hibernate.dialect.MySQLInnoDBDialect</property>
<property name="hibernate.connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="hibernate.connection.url">jdbc:mysql://localhost:3306/sakila</property>
<property name="hibernate.connection.username">root</property>
<property name="hibernate.show_sql">true</property>
<property name="transaction.factory_class">org.hibernate.transaction.JDBCTransactionFactory</property>
<property name="hibernate.current_session_context_class">thread</property>
<property name="hibernate.query.factory_class">org.hibernate.hql.classic.ClassicQueryTranslatorFactory</property>
<mapping resource="dvdrental/Language.hbm.xml"/>
<mapping resource="dvdrental/FilmActor.hbm.xml"/>
<mapping resource="dvdrental/FilmCategory.hbm.xml"/>
<mapping resource="dvdrental/Category.hbm.xml"/>
<mapping resource="dvdrental/Film.hbm.xml"/>
<mapping resource="dvdrental/Actor.hbm.xml"/>
</session-factory>
</hibernate-configuration>
When I'm using the HQL Query window, the SQL it seems to generate only ever says 'select from ' which is obviously wrong, but I can't see why this is being caused?
Edit 1: The HQL Query tab showing my input

Completely stupid reason- I hadn't even thought about the fact I hadn't built the project, so the example code from the bottom of the example ran properly (cause it came pre-compiled). Works no problem now.

I think the main cause of the error is as stated earlier from User, the U in User should be uppercase/capitalized as hibernate uses the POJO. I tried recompiling the code and it works correctly for the sample Swing application: https://netbeans.org/kb/docs/java/hibernate-java-se.html.
However the HQL Query editor still gives this error. It is strange but seems like this editor is not reliable.

The error message 'from limit 100' from your posted exception tells me, that you forgot to add Film after from. Please provide the HQL query that throws the exception.
BTW: the examples in the example project are bad practice. You should never use inline arguments in HQL queries:
from Film as film where film.filmId between '"+startID+"' and '"+endID+"'
You should use query parameters:
Query q = session.createQuery("from Film as film where film.filmId between :startID and :endID");
q.setParameter("startID", startID);
q.setParameter("endID", endID);

When I'm using the HQL Query window, the SQL it seems to generate only ever says 'select from ' which is obviously wrong, but I can't see why this is being caused?
A simple
from Film
is valid HQL and should work. Could you try without the following line:
<property name="hibernate.query.factory_class">org.hibernate.hql.classic.ClassicQueryTranslatorFactory</property>
Not sure this will solve the problem but there is no reason to use the "old" parser anyway.
That replaces the problem with a new one I'm afraid. Now the SQL tab of the HQL pane reports 'Invalid Query', and if I run the query I get org.hibernate.hql.ast.QuerySyntaxException. Any further pointers?
As I said, I was not sure this would solve the problem. What I would do is write a "simple" JUnit test to validate the setup is working. But an obvious bug like this in NetBeans HQL Query window is unlikely, IMO.

Completely stupid reason- I hadn't even thought about the fact I hadn't built the project, so the example code from the bottom of the example ran properly (cause it came pre-compiled). Works no problem now.
I experienced exactly the same problem but I think your stated solution is not correct. Even your project is not compiled HQL will show you results.
One should be careful of Table name in hibernate.cfg.xml file
as it should be same in query (case sensitive) also for example in your case it has table name as "Actor" so
from Actor will work, not from actor