We Keep Coding

Why does the Google Datastore console behave differently to the GAE Java library for Datastore?

I have a Google App Engine + Java app which has been happily running for many years (using JDO + datastore for persistence) and I have had no problem (occasionally, and reluctantly) updating a property of an entity in the Google Datastore console manually.
Recently (maybe the last 2-3 months) I have noticed a change in behaviour which breaks our app. I do not understand exactly what's going wrong or how we could handle it.
So my question is:
Why is it behaving differently and what can I do about it?
Let me first try to explain the behaviour I am seeing and then show my smallest possible replicating test case.
Suppose you had a simple persistence class:
#PersistenceCapable
public class Account implements Serializable {
#Persistent private ShortBlob testShortBlob;
#Persistent private String name;
// ...etc...
}
If I edited the name via the Datastore web console in the past, it would work as expected, the name field would change and everything else would work fine.
The behaviour I am seeing now is that after saving the entity via the console, I can no longer query and load the entity in JDO, I get:
java.lang.ClassCastException: com.google.appengine.api.datastore.Blob cannot be cast to com.google.appengine.api.datastore.ShortBlob
Which points to some underlying datastore change that means that ShortBlob field is having it's type change from ShortBlob to Blob (even though I make no edits to that field via the console).
This test case will replicate the issue:
DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
// this one really is a ShortBlob - will load fine in JDO
Entity account = new Entity("Account", "123");
account.setProperty("name", "Test Name");
account.setUnindexedProperty("testShortBlob", new ShortBlob("blah".getBytes()));
datastore.put(account);
// this one really is not a ShortBlob, its a blob - it will fail for the same reason I am seeing in production.
account = new Entity("Account", "124");
account.setProperty("name", "Test Name 2");
account.setUnindexedProperty("testShortBlob", new Blob("blah".getBytes()));
datastore.put(account);
// then load the entity via JDO
try {
accountFromJdo = pm.getObjectById(Account.class, key);
} catch (Exception ex) {
System.out.println("We get here, the object won't load with the ClassCast Exception");
}
So that's the issue, but why would saving via the cloud datastore console be changing the ShortBlob's to Blob?
My workaround currently is to set the ShortBlob fields to null in the Datastore console - that will then allow the entity to load. But that sucks if the data in the blob is important!
Update:
I have been doing more testing on this, using the low-level JSON API to see if I could se a difference in the raw JSON responses before and after saving the entity via console. The good news is, I can!
Before editing the entity via the console, a shortBlob field saved to the Datastore via the JDO App Engine interface will look like this:
},
"testShortBlob": {
"blobValue": "tNp7MfsjhdfjkahsdvfkjhsdvfIItWyzy6glmIrow4WWhRPbhQ/U+MGX3opVvpxu"
},
But if I go in to the Datastore console, and edit the entity (leave the blob field unchanged, edit an unrelated field, such as name. Now when I run the same query I get:
},
"testShortBlob": {
"blobValue": "tNp7MfsjhdfjkahsdvfkjhsdvfIItWyzy6glmIrow4WWhRPbhQ/U+MGX3opVvpxu",
"excludeFromIndexes": true
},
Subtle difference, but I think it's important, according to the Java docs ShortBlob are indexed, and Blob are not.
So I think my question now is: why does editing an entity via the Cloud Datastore console change the indexed status of blob fields?

Thanks for the detailed question and debugging. This seems fishy. I will make sure https://issuetracker.google.com/issues/79547492 gets assigned to the correct team.
As far as workarounds go:
The JSON API you noticed is Cloud Datastore API v1 there are a variety of client libraries to help make it easy to access.
It is possible to use that API to transactionally read/modify/write entities. In your case it would allow you to perform the desired transforms. Alternatively, making modifications through JDO would also work.

Searchanalytics Query JAVA API v3 (Google WebMaster Tools)

I am trying to implement Google Webmasters Searchanalytics Query using using the Java API but i did not found any Java sample to use , in Google website here there is only Python samples for Searchanalytics Query , and they did not say that it's not available in Java API.
I found this class Webmasters.Searchanalytics.Query in the Java API which I assume that is equivalent to the Python function searchanalytics.query() but i did not found any implementation of it.
My question if it is possible to query data from Google Search Console using the Java API??
if yes i wounder if there is someone who can provide a Java sample, something like the Python sample provided by Google here.
Thank you in advance.

I succeded to implement Webmasters.Searchanalytics.Query
as follow
first you need to create your QueryRequest using the SearchAnalyticsQueryRequest class example:
private static SearchAnalyticsQueryRequest createSearchAnalyticsQueryRequest() {
SearchAnalyticsQueryRequest searQueryRequest = new SearchAnalyticsQueryRequest();
searQueryRequest.setStartDate("2016-04-10");
searQueryRequest.setEndDate("2016-04-20");
List<String> dimensions = new ArrayList<String>();
dimensions.add("page");
dimensions.add("query");
dimensions.add("country");
dimensions.add("device");
dimensions.add("date");
searQueryRequest.setDimensions(dimensions);
return searQueryRequest;
}
then executing the query as follow :
public static String Query(String site,
SearchAnalyticsQueryRequest searQueryRequest) throws Exception {
Webmasters.Searchanalytics.Query query = service.searchanalytics()
.query(site, searQueryRequest);
SearchAnalyticsQueryResponse queryResponse = query.execute();
return queryResponse.toPrettyString();
}

I think You missed it. here. Actually all you need to do is to click the Java link on the left.

financequotes API returning null values

I have downloaded an API called "financequotes" for Java (Link: http://financequotes-api.com/) and have attempted to use it for a project. It has been imported into my class path and all the methods run, however when I ask for a stocks details
Stock s = new Stock("INTC");
s.print();
I am given back all the details which should have been obtained online as null including name, currency, quote, etc.
Why is this?
ALTERNATIVELY - Could you suggest another finance API which is relatively simple to use to gather basic financial data?
Thanks

The creator of the API has answered - Here was the problem
The code doesn't have a request to Yahoo Finance yet. There's 2 alternative ways to fix this.
Request it through the YahooFinance static methods
Stock stock = YahooFinance.get("INTC");
stock.print();
Force a refresh of the stock's quote by using the getQuote(boolean refresh) method
Stock stock = new Stock("INTC");
stock.getQuote(true);
stock.print();
This will automatically also load/refresh the statistics and dividend data.

Intrinio provides a simple to use API for financial information. It looks like you are a Java user, there are packages on for connecting via rest API and for connecting to real time prices via websocket.
The API is easy to use for stock prices, fundamentals, options, analyst estimates, etc. This tutorial will get you started, but here is an example in curl:
curl "https://api.intrinio.com/prices?ticker=AAPL" -u "API_Password:API_Username"

MongoDB: How to measure latency or other parameters of DB

Situation: App has an API part and one of the API calls returns status of MongoDB. Right now it returns only "OK" or "DOWN".
final DB defaultDb = dbFactory.getDb(dbName);
Getting general status of DB from DB object is not a problem. But how I can get some more information? Like latency or other DB parameters? And is it possible to get more?

Take a look at the diagnostic commands in the reference: http://docs.mongodb.org/manual/reference/command/nav-diagnostic/
You can run any of those using DB.command(String) method.
EDIT: I have also found a CommandResult DB.getStats() method in the Java API.

You can use the ELK stack. For e.g., look here:
https://logz.io/blog/mongodb-performance-monitoring-elk-stack/

How can I fill out an online form with Java?

My cell phone provider offers a limited number of free text messages on their website. I frequently use the service although I hate constantly having a tab open in my browser.
Does anyone know/point me in the right direction of how I could create a jar file/command line utility so I can fill out the appropriate forms on the site. I've always wanted to code up a project like this in Java, just in case anyone asks why I'm not using something else.
Kind Regards,
Lar

Try with Webdriver from Google or Selenium.

Sounds like you need a framework designed for doing functional testing. These act as browsers and can navigate web sites for testing and automation. You don't need the testing functionality, but it would still serve your needs.
Try HtmlUnit, or LiFT, which is a higher-level abstraction built on HtmlUnit.

Use Watij with the Eclipse IDE. When your done, compile as an .exe or run with a batch file.
Here is some sample code I wrote for filling in fields for a Google search, which can be adjusted for the web form you want to control :
package goog;
import junit.framework.TestCase;
import watij.runtime.ie.IE;
import static watij.finders.SymbolFactory.*;
public class GTestCases extends TestCase {
private static watij.runtime.ie.IE activeIE_m;
public static IE attachToIE(String url) throws Exception {
if (activeIE_m==null)
{
activeIE_m = new IE();
activeIE_m.start(url);
} else {
activeIE_m.goTo(url);
}
activeIE_m.bringToFront();
return (activeIE_m);
}
public static String getActiveUrl () throws Exception {
String currUrl = activeIE_m.url().toString();
return currUrl;
}
public void testGoogleLogin() throws Exception {
IE ie = attachToIE("http://google.com");
if ( ie.containsText("/Sign in/") ) {
ie.div(id,"guser").link(0).click();
if ( ie.containsText("Sign in with your") ||
ie.containsText("Sign in to iGoogle with your")) {
ie.textField(name,"Email").set("test#gmail.com");
ie.textField(name,"Passwd").set("test");
if ( ie.checkbox(name,"PersistentCookie").checked() ){
ie.checkbox(name,"PersistentCookie").click();
}
ie.button(name,"signIn").click();
}
}
System.out.println("Login finished.");
}
public void testGoogleSearch() throws Exception {
//IE ie = attachToIE( getActiveUrl() );
IE ie = attachToIE( "http://www.google.com/advanced_search?hl=en" );
ie.div(id,"opt-handle").click();
ie.textField(name,"as_q").set("Watij");
ie.selectList(name,"lr").select("English");
ie.button(value,"Advanced Search").click();
System.out.println("Search finished.");
}
public void testGoogleResult() throws Exception {
IE ie = attachToIE( getActiveUrl() );
ie.link(href,"http://groups.google.com/group/watij").click();
System.out.println("Followed link.");
}
}

It depends on how they are sending the form information.
If they are using a simple GET request, all you need to do is fill in the appropriate url parameters.
Otherwise you will need to post the form information to the target page.

You could use Watij, which provides a Java/COM interface onto Internet Explorer. Then write a small amount of Java code to navigate the form, insert values and submit.
Alternatively, if it's simple, then check out HttpClient, which is a simple Java HTTP client API.
Whatever you do, watch out that you don't contravene your terms of service (easy during testing - perhaps you should work against a mock interface initially?)

WebTest is yet another webapp testing framework that may be easier to use than the alternatives cited by others.

Check out the Apache Commons Net Package. There you can send a POSt request to a page. This is quite low level but may do what you want (if not you might check out the functional testing suites but it is probably not as easy to dig into).

As jjnguy says, you'll need to dissect the form to find out all the parameters.
With them you can form your own request using Apache's HTTP Client and fire it off.