We Keep Coding

Is there a way to configure tests for the Java Google App Engine SDK to fail if a compound index is needed?

I'd like to configure LocalDatastoreServiceTestConfig such that queries will fail if a compound index is needed (e.g., a query with a sort on multiple properties). Is there a way to do this?
I tried new LocalDatastoreServiceTestConfig().setNoIndexAutoGen(true) but it had no effect.
(There is a corresponding way to do this with the Python SDK.)

I assume by "fail" you mean "throw an exception" or something similar.
If so, you should set the autoGenerate attribute in your WEB-INF/datastore-indexes.xml to false.
Example WEB-INF/datastore-indexes.xml:
<datastore-indexes autoGenerate="false">
</datastore-indexes>
Setting autoGenerate to false will make a query that requires a composite index throw an exception.
Example code:
try {
Query q = new Query("Action")
.addSort("encrypter", Query.SortDirection.ASCENDING)
.addSort("requester", Query.SortDirection.ASCENDING)
.addSort("time", Query.SortDirection.DESCENDING);
//...snip...
} catch (Exception e) {
log.severe(e.toString());
}
I tested this and got an exception logged as expected:
SEVERE: com.google.appengine.api.datastore.DatastoreNeedIndexException: Query com.google.appengine.api.datastore.dev.LocalCompositeIndexManager$IndexComponentsO
nlyQuery#f9f81ad3 requires a composite index that is not defined. You must update C:\appengine-java-sdk\dev\core1\war\WEB-INF\datastore-indexes.xml or enable au
toGenerate to have it automatically added.
The suggested index for this query is:
<datastore-index kind="Action" ancestor="false" source="manual">
<property name="encrypter" direction="asc"/>
<property name="requester" direction="asc"/>
<property name="time" direction="desc"/>
</datastore-index>
For more information, see datastore-indexes.xml reference.

Objectify multiple filters doesn´t work with cron job

I´m working with objectify on appengine, I tried to add a cron job to delete all temp entities which are older than an hour:
Iterable<Key<Entry>> allKeys = ofy().load().type(Entry.class)
.filter("temporary", true)
.filter("createdAt", oneHourAgo).keys();
if(allKeys != null){
ofy().delete().keys(allKeys);
}
but i always get an Exception when executing the cron job on the appengine server:
com.google.appengine.api.datastore.DatastoreNeedIndexException: no matching index found.
The suggested index for this query is:
<datastore-index kind="Entry" ancestor="false" source="manual">
<property name="temporary" direction="asc"/>
<property name="createdAt" direction="asc"/>
</datastore-index>
does anybody know why this happens? The job works if I remove:
.filter("createdAt", oneHourAgo)

When you use your app with a Development server, the development server tries to figure out which indexes you need and places them in the index-definition file automatically. Since you use this query in a cron job, the development server cannot help you here. You will need to add a definition for this index manually.
Java Datastore Index Configuration

index was not created even indicated in datastore-index.xml

My app id is gusclabs.
Index entry in datastore-index.xml:
<datastore-index kind="ItemGroupOptionalQuantities" ancestor="true" source="manual">
<property name="quantityFrom" direction="asc"/>
</datastore-index>
Error in appspot-log:
Type 'com.google.appengine.api.datastore.DatastoreNeedIndexException' was not included in the set of types which can be serialized by this SerializationPolicy or its Class object could not be loaded. For security purposes, this type will not be serialized.: instance = com.google.appengine.api.datastore.DatastoreNeedIndexException: no matching index found.
The suggested index for this query is:
<datastore-index kind="ItemGroupOptionalQuantities" ancestor="true" source="manual">
<property name="quantityFrom" direction="asc"/>
</datastore-index>
I updated the entities after deploy the app in a new version.
When I use a query using quantityFrom to order, I have this error.
--- start query ----
Query query = new Query("ItemGroupOptionalQuantities");
query.setAncestor(ancestorKey);
query.addSort("quantityFrom", Query.SortDirection.ASCENDING)
PreparedQuery preparedQuery = datastore.prepare(query);
FetchOptions fetchOptions = FetchOptions.Builder.withLimit(10);
QueryResultList<Entity> entities = preparedQuery.asQueryResultList(fetchOptions);
--- end query ---

If you just update the app then you will have to wait a few minutes for these indexes to be built. From the dashboard of your app if you visit the Datastore Indexes pages you should be able to see what kind of indexes you have and in what state they are.

Solved.
The right place for the datastore-indexes.xml file is "/war/WEB-INF", not "/war/WEB-INF/appengine-generated"
Thanks Patrick and Lipis.

Low priority updates on MySQL using JDBC - how to check if they work

I have a basic setup where the database is read by multiple web applications, and periodically I have a batch application which does a lot of writing. During the writing, the peroformance of the web-apps degrade heavily (their database reads are very slow).
The env. is MySQL db using MYISAM engine, the batch application is a Java SE app, using spring-batch and SimpleJDBCTemplate to issue SQL commands via JDBC. I found that MySQL has a parameter that lowers the priority of write operations on MYISAM engine: low_priority_updates. To quote the docs, amongs others, you can "SET LOW_PRIORITY_UPDATES=1 to change the priority in one thread". I opted for this because it's easiest from the config standpoint of my application. What I've done is configured my DataSource such that it exectutes that "SET ..." for each connection it opens, like so:
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<!-- other props omitted -->
<property name="connectionInitSqls">
<list>
<value>SET low_priority_updates="ON"</value>
</list>
</property>
</bean>
Now my question is, how do I actually check that an SQL issued via this datasource does actually run with low priority? If I do SHOW FULL PROCESSLIST in MySQL while the inserts are happening it just tell me what SQL they're executing, nothing about the priority:
If I check the server variables low_priority_updates is "OFF" but that's just the server variable, it sais nothing about the thread-local value.
So again, is there any actual way to check if per query/thread low_priority_updates's values are taken into account?

By issuing SET LOW_PRIORITY_UPDATES=1 command, you are affecting the variable value for the session. Therefore it is possible to see this by checking the value of the variable in the session.
I know of two ways to do it:
1- SHOW SESSION VARIABLES LIKE 'low_priority_dapdates'
this shows ON/OFF
2- select ##session.low_priority_updates
this gives 0/1
Important: the above statements/calls will show you the values of the variables in the session where they run.
Therefore, you will need to run them using the connections themselves in order to see the values. I don't know of a way in MySQL where you can select values for variables that belong to another session.
If you would like to see them as a list, you might need to do a work around by creating a table and logging that info yourself. for example:
CREATE TABLE `mydb`.`my_low_priority_updates` (
`connection_id` INT ,
`low_priority_updates_value` INT NOT NULL
)
ENGINE = MyISAM;
then you need a statement that inserts the connection id and the value into the table:
insert into my_low_priority_updates(connection_id,low_priority_updates_value)
select connection_id(),##session.low_priority_updates
from dual
where not exists (select 1 from my_low_priority_updates where connection_id=connection_id())
you can put this statement in a procedure and make sure its called, or add it in a trigger on a table that you know gets updated/inserted into.
after that, querying the my_low_priority_updates table later will show you the values of the variable in each connection.

Google App Engine Error: No matching index found. (Java)

I am writing a query but it always says "No matching index found". I don't know why. My code is as below:
Query query = pm.newQuery(Classified.class);
query.setFilter("emp_Id == emp");
query.setOrdering("upload_date desc");
query.declareParameters("String emp");
List<Classified> results = (List<Classified>)query.execute(session.getAttribute("emp_Id").toString());
<?xml version="1.0" encoding="utf-8"?>
<datastore-indexes autoGenerate="true">
<datastore-index kind="Classified" ancestor="false">
<property name="emp_Id" direction="asc" />
<property name="category" direction="asc" />
<property name="upload_date" direction="desc" />
</datastore-index>
</datastore-indexes>
I have added the above index, but it did not help.

I believe you need to configure a Datastore Index. There's probably one already generated for you in Eclipse at WEB-INF/appengine-generated/datastore-indexes-auto.xml that you just need to copy to WEB-INF/datastore-indexes.xml and deploy again.

Because this needs to be somewhere on the internet...
I kicked myself when I found this out
The error is you do not have a index matching what the query would like to perform. You can have multiple indexes for each entity.
In the Logcat, error, it will tell you exactly what index to set and what order the elements need to be.
ie, if the error says it wants (it wont be nicely formatted):
<datastore-index kind="Classified" ancestor="false">
<property name="category" direction="desc" />
<property name="upload_date" direction="desc" />
</datastore-index>
then Project -> war -> WEB-INF -> appengine-generated -> datastore-indexes-auto.xml and add exactly that. Then, redeploy the project.
Next go into your Google Cloud Console and look at Datastore -> indexes. It should say that the index is being prepared (This goes quicker if you can kill all apps connected and shut down the instance in the console).
Once this has moved into the list of other indexes, rerun the your application and it wont error out with regards to the index anymore.
Go get it Gentlemen/Ladies

The index you define must hold all possible results in the order they will be returned. Your query asks for a particular emp_Id, ordered by upload_date, but your index is ordered primarily by category.
Try removing the category line from your index definition, or swapping the order of category and upload_date, to make upload_date the primary sort order for the index. If another part of your code relies on the category line, you may have to make two separate indices (which incurs some computational cost).
Edit: see comment below by Nick Johnson re. extra parameters.

I am running into this issue at the moment when doing a single property query such as:
const query = datastore
.createQuery('Emailing_dev')
.filter('status', '=', 'Scheduled')
In my case, I should not be getting any errors, however I get Error 9: No matching index found.
If I defined the single property index twice in the yaml, it works:
indexes:
- kind: Emailing_dev
properties:
- name: status
- name: status
but this for sure must be a bug..!!