How to browse local Java App Engine datastore? - java

It seems there is no equivalent of Python App Engine's _ah/admin for the Java implementation of Google App Engine.
Is there a manual way I can browse the datastore? Where are the files to be found on my machine? (I am using the App Engine plugin with Eclipse on OS X).

http://googleappengine.blogspot.com/2009/07/google-app-engine-for-java-sdk-122.html: "At long last, the dev appserver has a data viewer. Start your app locally and point your browser to http://localhost:8888/_ah/admin http://localhost:8000/datastore* to check it out."
* as of 1.7.7

There's currently no datastore viewer for the Java SDK - one should be coming in the next SDK release. In the meantime, your best bet is to write your own admin interface with datastore viewing code - or wait for the next SDK release.
Java App Engine now has a local datastore viewer, accessible at http://localhost:8080/_ah/admin.

I have local datastore on my Windows+Eclipse environment on \war\WEB-INF\appengine-generated\local_db.bin
As far as I understood it uses internal format named "protocol buffers". I don't have external tools to present the file in human-readable format.
I'm using simple "viewer" code like this:
public void doGet(HttpServletRequest req, HttpServletResponse resp)
throws IOException
{
resp.setContentType("text/plain");
final DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
final Query query = new Query("Table/Entity Name");
//query.addSort(Entity.KEY_RESERVED_PROPERTY, Query.SortDirection.DESCENDING);
for (final Entity entity : datastore.prepare(query).asIterable()) {
resp.getWriter().println(entity.getKey().toString());
final Map<String, Object> properties = entity.getProperties();
final String[] propertyNames = properties.keySet().toArray(
new String[properties.size()]);
for(final String propertyName : propertyNames) {
resp.getWriter().println("-> " + propertyName + ": " + entity.getProperty(propertyName));
}
}
}

In the newest versions of the SDK (1.7.6+) the admin part of the dev server comes with it changed its location
Analyzing the server output logs we can see that it is accessible at:
http://localhost:8000
And the Datastore viewer:
http://localhost:8000/datastore
Looks pretty neat - according to google's new design guidlines.

Because Google App Engines Datastore viewer does not support displaying collections of referenced entities, I modified Paul's version to display all descendant entities:
public void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException {
String entityParam = req.getParameter("e");
resp.setContentType("text/plain");
final DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
// Original query
final Query queryOrig = new Query(entityParam);
queryOrig.addSort(Entity.KEY_RESERVED_PROPERTY, Query.SortDirection.ASCENDING);
for (final Entity entityOrig : datastore.prepare(queryOrig).asIterable()) {
// Query for this entity and all its descendant entities and collections
final Query query = new Query();
query.setAncestor(entityOrig.getKey());
query.addSort(Entity.KEY_RESERVED_PROPERTY, Query.SortDirection.ASCENDING);
for (final Entity entity : datastore.prepare(query).asIterable()) {
resp.getWriter().println(entity.getKey().toString());
// Print properties
final Map<String, Object> properties = entity.getProperties();
final String[] propertyNames = properties.keySet().toArray(new String[properties.size()]);
for(final String propertyName : propertyNames) {
resp.getWriter().println("-> " + propertyName + ": " + entity.getProperty(propertyName));
}
}
}
}
It should be noted that nothing is displayed for empty collections/referenced entities.

Open the \war\WEB-INF\appengine-generated\local_db.bin file with a text editor, like Notepad++.
The data is scrambled but at least you can read it and you can copy to extract it.

For me the fix was to do the login using below gcloud command
gcloud auth application-default login

Related

Saving documents using OpenOffice java api throws Exception

FYI / Context: I am running a portable installation of libreoffice on windows 10 (getting the same exception on a mac with normal installation though).
Reading the document
Maybe this is important... I am reading the document through an InputStream, because the other method fails due to a different exception (probably a story for another time).
public XComponent openFileViaStream(File file) throws CommandAbortedException, Exception {
Object fileAccess = this.componentFactory.createInstanceWithContext(SimpleFileAccessClass, this.context);
XSimpleFileAccess xSimpleFileAccess = (XSimpleFileAccess) UnoRuntime.queryInterface(XSimpleFileAccess.class,
fileAccess);
XStream xInputStream = xSimpleFileAccess.openFileReadWrite(file.getAbsolutePath());
PropertyValue[] loadProps = new PropertyBuilder().add("InputStream", xInputStream).build();
return loader.loadComponentFromURL("private:stream", "_blank", 0, loadProps);
}
Writing the document
PropertyBuilder is a utility class that just builds an Array of PropertyValues for ease of use.
public void save(Object storeMe, File destination) throws IOException, MalformedURLException {
//#formatter:off
PropertyValue[] propertyValue = new PropertyBuilder()
.add("Overwrite", Boolean.TRUE)
.add("FilterName", "StarOffice XML")
.build();
//#formatter:on
XStorable2 st = UnoRuntime.queryInterface(XStorable2.class, storeMe);
// already tried
// st.storeAsURL(destination.toURI().toURL().toString(), propertyValue);
// st.storeToURL(destination.toURI().toString(), propertyValue);
// st.storeToURL(destination.toURI().toURL().toString(), propertyValue);
st.storeAsURL(destination.toURI().toString(), propertyValue);
}
The exception
I couldn't find a solution while searching on stackoverflow...
com.sun.star.task.ErrorCodeIOException: SfxBaseModel::impl_store <file:/E:/test/abc.odt> failed: 0x81a(Error Area:Io Class:Parameter Code:26)
at com.sun.star.lib.uno.environments.remote.Job.remoteUnoRequestRaisedException(Job.java:173)
at com.sun.star.lib.uno.environments.remote.Job.execute(Job.java:139)
at com.sun.star.lib.uno.environments.remote.JobQueue.enter(JobQueue.java:334)
at com.sun.star.lib.uno.environments.remote.JobQueue.enter(JobQueue.java:303)
at com.sun.star.lib.uno.environments.remote.JavaThreadPool.enter(JavaThreadPool.java:87)
at com.sun.star.lib.uno.bridges.java_remote.java_remote_bridge.sendRequest(java_remote_bridge.java:636)
at com.sun.star.lib.uno.bridges.java_remote.ProxyFactory$Handler.request(ProxyFactory.java:146)
at com.sun.star.lib.uno.bridges.java_remote.ProxyFactory$Handler.invoke(ProxyFactory.java:128)
at com.sun.proxy.$Proxy10.storeAsURL(Unknown Source)
at DocumentHandler.save(DocumentHandler.java:54)
at Main.test(Main.java:14)
at Main.main(Main.java:19)
I really have no idea what I am doing wrong. I've looked at examples from api.libreoffice.org etc.
Am I missing something? A PropertyValue?
Thank you in advance!
See if any of these ideas help.
The URI should look like file:///E:/test/abc.odt.
Set the filter name to StarOffice XML (Writer) or writer8. Or don't set it at all; pass one property instead of two.
Verify you have the authorization to write to the file, for example by using standard Java libraries to create a file in that location. Be sure the file is not locked by some other process.

Azure BlobNotFound when downloading in SpringBoot app

I have a task to make possibility to download simple .txt files from the application using Azure Blob Storage. The code is supposed to work. I didn't write it, but it looks OK to me and from what I'll show later in this post, it really connects to the Azure, and, what's more important, it really works only when I'm testing the app on localhost, but not on the publicly available site.
These are the steps I made:
uploaded files to the storage (the underlined is one of them):
added proper link to the button that should download the attachment via REST API
of course, I've also added reference to the attachment in the database (its ID, name etc.)
here's how it looks on frontend:
And this is what I get:
I've seen somewhere that it might be caused by Azure CORS settings that don't allow the app to access the storage. Here's what I've done so far:
went to portal.azure.com and changed CORS settings like this:
found something about putting some code into the app under this Microsoft link, but it's not Java. I guess there are some analogical ways in Java:
https://blogs.msdn.microsoft.com/windowsazurestorage/2014/02/03/windows-azure-storage-introducing-cors/ . Is it necessary after the CORS rules have been added in the Azure Portal?
Also, I've found information that it may be caused by the storage access permissions. The Public Access Level is set to Container:
Not sure if it gives anything, but these are the container's properties:
What else can be the problem with the BlobNotFound error I receive? Hope I've put enough information here, but if some more is needed say in comment and I'll provide it.
This is the code that's supposed to download the attachment of this method, contained in 3 classes:
Controller class part:
#GetMapping("/download/{id}")
#ResponseStatus(HttpStatus.OK)
public void downloadAttachment(#PathVariable long id, HttpServletResponse response) throws IOException {
dataUploadRequestAttachmentService.downloadStaticAttachment(response, id);
}
Controller service class part:
public void downloadStaticAttachment(HttpServletResponse response, long id) throws IOException {
ArticleAttachment articleAttachment = this.findAttachment(id);
String mimeType = URLConnection.guessContentTypeFromName(articleAttachment.getName());
if (mimeType == null){
mimeType = "application/octet-stream";
}
response.setContentType(mimeType);
response.setHeader("Content-Disposition", String.format("attachment; filename=\"%s\"", articleAttachment.getName()));
azureBlobStorageArticleAttachmentService.downloadArticleAttachment(
articleAttachment.getName(),
articleAttachment.getId(),
response.getOutputStream()
);
}
And the AzureBlobStorageArticleAttachmentService class:
public void downloadArticleAttachment(String attachmentName, Long articleId, OutputStream outputStream) {
try {
CloudBlockBlob blob = container.getBlockBlobReference(String.format("%s_%s", articleId, attachmentName));
blob.download(outputStream);
} catch (URISyntaxException | StorageException e) {
e.printStackTrace();
log.error(String.format("Download article attachment %s error", attachmentName));
}
}
According to your description, please debug to check if you get the correct blob name in the code: CloudBlockBlob blob = container.getBlockBlobReference(String.format("%s_%s", articleId, attachmentName));
Here is a demo about how to download blobs using Java SDK for your reference:
/// <summary>
/// download blob to memory
/// </summary>
/// <param name="containerName">blob container name</param>
/// <param name="blobName">blob Name</param>
public static ByteArrayOutputStream downloadBlobToMemory(String containerName, String blobName) {
CloudStorageAccount account = null;
CloudBlobContainer container = null;
ByteArrayOutputStream byteArrayOutputStream = null;
try {
account = CloudStorageAccount.parse(ConnString);
CloudBlobClient client = account.createCloudBlobClient();
container = client.getContainerReference(containerName);
container.createIfNotExists();
CloudBlockBlob cloudBlockBlob = container.getBlockBlobReference(blobName);
byteArrayOutputStream=new ByteArrayOutputStream();
cloudBlockBlob.download(byteArrayOutputStream);
}catch(Exception ex) {
ex.printStackTrace();
}
return byteArrayOutputStream;
}
/// <summary>
/// download blob to local disk
/// </summary>
/// <param name="containerName">blob container name</param>
/// <param name="blobName">blob Name</param>
/// <param name="filePath"> for example: C:\\Test\test.txt</param>
public static void downloadBlobToDisk(String containerName, String blobName, String filePath) {
CloudStorageAccount account = null;
CloudBlobContainer container = null;
try {
account = CloudStorageAccount.parse(ConnString);
CloudBlobClient client = account.createCloudBlobClient();
container = client.getContainerReference(containerName);
container.createIfNotExists();
CloudBlockBlob cloudBlockBlob = container.getBlockBlobReference(blobName);
FileOutputStream fileOutputStream=new FileOutputStream(filePath);
cloudBlockBlob.download(fileOutputStream);
}catch(Exception ex) {
ex.printStackTrace();
}
}
Lee Liu's suggestion about the Blob name was correct when I managed to find out the correct application address. It turned out that domain address visible by user was ending with "azureedge.net", but there's a different one when I went into portal.azure.com. It caused the main problem. After that, I indeed found problem with correct Blob names in storage - because of String.format, I had to add their ID in database with a "_" sign, then they started to be downloaded with content instead of empty files.
It seems that the code was OK, it was the problem with improper address and file names.

azure java sdk authentication

I would like to list available IP VM's in the new Azure portal using Java SDK.
Couple of years back in the good old classic portal, I had followed the usual management certificate procedure to access vm's,create vm's and work with Azure Endpoints.
Fast fwd now I see that they have used a new portal and new mechanisms to interact with Java SDK. I read somewhere in the above link that with the old way with certificates, I can manage only the class portal resources.
I'm trying to code a simple program which authenticates and lists the vm's of the new portal as a start. Seems like they have complicated it a lot.
I followed the below link to "Create service principal with password"
https://azure.microsoft.com/en-us/documentation/articles/resource-group-authenticate-service-principal/
Then I went to this link
https://azure.microsoft.com/en-us/documentation/samples/resources-java-manage-resource-group/
which asked me go the "See how to create an Auth file" link in above page
(mine is not a webapp and when I try to create the AD as a native client application, it is not allowing me to save keys in configure tab, so I had to create a web app)
After doing all this, I got stuck with this below error
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
'authority' Uri should have at least one segment in the path (i.e.https://<host>/<path>/...)
java.lang.IllegalArgumentException: 'authority' Uri should have at least one segment in the path (i.e. https://<host>/<path>/...)
at com.microsoft.aad.adal4j.AuthenticationAuthority.detectAuthorityType(AuthenticationAuthority.java:190)
at com.microsoft.aad.adal4j.AuthenticationAuthority.<init>(AuthenticationAuthority.java:73)
When I checked it says that the error is because I don't have a valid client application id in your Azure Active Directory.
Is there any simple way to authenticate and start using the API's?
#Vikram, I suggest that you can try to refer to the article to create an application on AAD.
Then you can follow the code below to get the access token for authentication.
// The parameters include clientId, clientSecret, tenantId, subscriptionId and resourceGroupName.
private static final String clientId = "<client-id>";
private static final String clientSecret = "<key>";
private static final String tenantId = "<tenant-id>";
private static final String subscriptionId = "<subscription-id>";
// The function for getting the access token via Class AuthenticationResult
private static AuthenticationResult getAccessTokenFromServicePrincipalCredentials()
throws ServiceUnavailableException, MalformedURLException, ExecutionException, InterruptedException {
AuthenticationContext context;
AuthenticationResult result = null;
ExecutorService service = null;
try {
service = Executors.newFixedThreadPool(1);
// TODO: add your tenant id
context = new AuthenticationContext("https://login.microsoftonline.com/" + tenantId, false, service);
// TODO: add your client id and client secret
ClientCredential cred = new ClientCredential(clientId, clientSecret);
Future<AuthenticationResult> future = context.acquireToken("https://management.azure.com/", cred, null);
result = future.get();
} finally {
service.shutdown();
}
if (result == null) {
throw new ServiceUnavailableException("authentication result was null");
}
return result;
}
String accessToken = getAccessTokenFromServicePrincipalCredentials().getAccessToken();
If you want to list the VMs on new portal, you can try to use the REST API List the resources in a subscription to get all resources and filter the VMs via the resource type Microsoft.Compute/virtualMachines.
Hope it helps.

Java - How to Retrieve and use a single value from azure mobile services in Android

I am new to azure but i know certain things like how to retrieve and store data to azure , i followed azure official documentation for this purpose.
Link is Here - https://azure.microsoft.com/en-in/documentation/articles/mobile-services-android-get-started-data/
But the problem is, this tutorial is only showing How to retrieve and use Data from azure using Adapters and Lists . I want to know , How can i retrieve a single value from azure mobile services and how to use it in android.
Plzz provide me both backend code (if there is any) and java code for this . THANKS in advance
I got it solved. No need to create a custom API.
Just follow the basics , Here is the code :-
final String[] design = new String[1];
private MobileServiceTable<User> mUser;
mUser = mClient.getTable(User.class);
new AsyncTask<Void, Void, Void>() {
#Override
protected Void doInBackground(Void... params) {
try {
final MobileServiceList<User> result =
mUser.where().field("name").eq(x).execute().get();
for (User item : result) {
// Log.i(TAG, "Read object with ID " + item.id);
desig[0] = item.getDesignation(); //getDesignation() is a function in User class ie- they are getters and setters
Log.v("FINALLY DESIGNATION IS", desig[0]);
}
} catch (Exception exception) {
exception.printStackTrace();
}
return null;
}
#Override
protected void onPostExecute(Void aVoid) {
super.onPostExecute(aVoid);
designation.setText(desig[0]);
}
}.execute();
DON'T forget to create a class User for serialization and all. Also you should define the array .
FEEL FREE to write if you got it not working.
EDIT :-
design[0] is an array with size 1.
eq(x) is equal to x where , x variable contains username for which i want designation from database (azure).
You can do this with a custom API. See this link: https://azure.microsoft.com/en-us/documentation/articles/mobile-services-how-to-use-server-scripts/#custom-api
Code looks like this:
exports.post = function(request, response) {
response.send(200, "{ message: 'Hello, world!' }");
}
It's then reachable at https://todolist.azure-mobile.net/api/APIFILENAME.
If you want to access a table you can do something like:
exports.post = function(request, response) {
var userTable = tables.getTable('users');
permissionsTable
.where({ userId: user.userId})
.read({ success: sendUser });
}
function sendUser(results){
if(results.length <= 0) {
res.send(200, {});
} else {
res.send(200, {result: results[0]});
}
}
You can then follow the instructions for using the API on your Android client here: https://azure.microsoft.com/en-us/documentation/articles/mobile-services-android-call-custom-api/
How your app is written will change how this code works/looks, but it looks something like:
ListenableFuture<MarkAllResult> result = mClient.invokeApi( "UsersAPI", MarkAllResult.class );
That invokes the API. You need to write the class and Future to handle the results. The above page explains this in great detail.
The most optimal solution would be to create an api on your server which accepts an ID to return an single object/tablerow.
In your android app, you only have to call:
MobileServiceTable<YourClass> mYourTable;
mClient = new MobileServiceClient(
"https://yoursite.azurewebsites.net/",
mContext);
mYourTable = mClient.getTable(YourClass.class);
YourClass request = mYourTable.lookUp(someId).get();
// request -> https://yoursite.azurewebsites.net/tables/yourclass/someId
YourClass should have the same properties as the object on the server.

How to retrieve data from the Google Datastore?

I use java servlets. I wrote a code which has no error,But it returns me an empty list always.
I dint want to update Datastore from the servlet. i just want to read entities. i'll enclose my code tell me where is the problem.
I always get the data store is empty.This is just a test code.Even this dosent seem to work. searched internet for week . All for vain.
public class TestServlet extends HttpServlet {
public void doGet(HttpServletRequest req, HttpServletResponse resp)
throws IOException {
resp.setContentType("text/plain");
resp.getWriter().println("Hello, world");
DatastoreService datastoreService = DatastoreServiceFactory.getDatastoreService();
resp.getWriter().println(datastoreService.getIndexes());
if (datastoreService.getIndexes().isEmpty())
resp.getWriter().println("the data store is empty");
Query query = new Query("IMAGES");
PreparedQuery pq = datastoreService.prepare(query);
for (Entity entity :pq.asIterable())
{
resp.getWriter().println(entity.getKind() );
resp.getWriter().println(entity.getAppId() );
resp.getWriter().println(entity.getKey() );
}
if (!pq.asIterable().iterator().hasNext())
resp.getWriter().println("the data store is empty");
}
I rectified my problem . I forgot to add a namespace. here is a snippet on How to set namespace in GAE DataStore.
// Set the namepace temporarily to "abc"
String oldNamespace = NamespaceManager.get();
NamespaceManager.set("abc");
try {
... perform operation using current namespace ...
} finally {
NamespaceManager.set(oldNamespace);
}

Categories