I have a system which, pulls data from my server and stores it to a mobile SQL database via android studio. It works but it is painful slow like 30mins. I have around 86000 records in my database and want to pull all of them out of the server. What is the best way to do this?
Presently I get the count from the server and then query the server database until i find each ID and then send that result back to my mobile app.
app.post("/get_data", function(req, res)
{
var Id_request = req.body.Id_request;//The requested ID
var query = {Val_String : Id_request};//Query value
//console.log(query);
//Data.find({}, function(err, result) {//Finds all data
Data.findOne(query, function(err, result) {//Finds all data
if (err) {
//res.status(400).send(err);
console.log("Sending error");
res.status(200).send();
} else {
return res.json(result);
}
});
});
I use a recersive function in my pull request for each ID
private void call_method()
{
HashMap<String, String> map = new HashMap<>();
map.put("Id_request", Integer.toString(data_pointer));//The ID value
Call<Fetch_result> call = retrofitInterface.executeGet_data(map);//Run the post
call.enqueue(new Callback<Fetch_result>() {
//call.enqueue(new Callback<Fetch_result>() {
#Override
public void onResponse(Call<Fetch_result> call, Response<Fetch_result> response) {
if (response.code() == 200)//Successful login
{
D1= response.body().getT1_String();
D2= response.body().getT2_String();
data_pointer = data_pointer + 1;
boolean result = BLE_DB.addData_Downloaded(D1,D2);//Add data
if(data_pointer<= Total_entries) {//Call method again
call_method();//Recursive here
}else if (data_pointer > Total_entries){
Utils.toast(getApplicationContext(),"All data received");
}
} else if (response.code() == 404) {
Utils.toast(getApplicationContext(), "Get data fail");
}
}
#Override
public void onFailure(Call<Fetch_result> call, Throwable t) {
Utils.toast(getApplicationContext(), "Get data error");
}
});
}
How Can I speed this up or do it differently to speed it up?
Try to fetch as much data as possible at once ( limit the amount of queries you do ). It is hard to tell you how since I don't know your monogDB collection.
Try to do this with as little requests as possible. If you can return all the fetched data at once, this will save you some time.
JSON may be very slow when doing it on 86000 documents
Consider caching the data for future users
Right now i suspect what is limiting you is the fact that you are doing 86000 queries to the db... If you can get the entire mongoDB collection it may be a bit faster( look at notes )
Notes:
https://docs.mongodb.com/manual/reference/method/db.collection.find/#db-collection-find ( omit the query parameter will result in fetching the entire collection )
Related
I am a newbie (6 months going or so) and creating an app on Android (Java) that utilizes FireBase Auth with Google Sign In. (with only a few days of NodeJS exposure now) In other words my end user signs in with the Google Account. That part (I think) works pretty well so far. I use the Firestore Database heavily for a lot of things in the app.
So now I've gotten to the point where I want to use (Callable) Cloud Functions with HTTP Triggers. (never having done any of this before) I'm trying to get a proof of concept working at this time. The actual function works and I can trigger it from my app.
It appears that I cannot figure out how to make the function "private" though; as in "adding proper Members" to the Cloud function who have the right to invoke the function.
I have tried a few different things by trial error, but first let me show what I have.
This is the Cloud Function and I'm passing in an arbitrary String as a test, works nicely: (as long as "allUsers" have the role/right to invoke the function; in other words when the function is public.
exports.createTest = functions.https.onCall((data, context) => {
const text = data.text;
const uid = context.auth.uid;
const name = context.auth.token.name || null;
const email = context.auth.token.email || null;
console.log('UID: ', uid);
console.log('Name: ', name);
console.log('Email: ', email);
console.log('Message: ', text);
});
The above function gets triggered in my Android/Java code like this: (I think this code came from Google Doc/Sample/Example
private FirebaseFunctions mFunctions;
...
private void testing() {
mFunctions = FirebaseFunctions.getInstance();
Log.e(LOG_TAG, "Testing executed!");
String testMessage = "Hello Hello Testing 123 Mic Check";
createTest(testMessage)
.addOnCompleteListener(new OnCompleteListener<String>() {
#Override
public void onComplete(#NonNull Task<String> task) {
if (!task.isSuccessful()) {
Exception e = task.getException();
if (e instanceof FirebaseFunctionsException) {
FirebaseFunctionsException ffe = (FirebaseFunctionsException) e;
FirebaseFunctionsException.Code code = ffe.getCode();
Object details = ffe.getDetails();
Log.e(LOG_TAG, "FFE: " + ffe.getMessage() );
Log.e(LOG_TAG, "Code: " + code);
Log.e(LOG_TAG, "Details:" + details);
}
// ...
}
// ...
}
});
}
private Task<String> createTest(String text) {
// Create the arguments to the callable function.
Map<String, Object> data = new HashMap<>();
data.put("text", text);
data.put("push", true);
return mFunctions
.getHttpsCallable("createTest") //this is the function name
.call(data)
.continueWith(new Continuation<HttpsCallableResult, String>() {
#Override
public String then(#NonNull Task<HttpsCallableResult> task) throws Exception {
// This continuation runs on either success or failure, but if the task
// has failed then getResult() will throw an Exception which will be
// propagated down.
String result = (String) task.getResult().getData();
if (result != null) {
Log.e(LOG_TAG, "Result: " + result);
}
return result;
}
});
}
Only when I have "allUsers" added with the role/right to "invoke Cloud Function" then I get this working. My understanding of HTTP Requests and such is pretty limited, which is not making things easier.
I tried using the "allAuthenticatedUsers" options, which I figured would do the trick, because I actually authenticate my Users in the app through Firebase/Google Sign In. This Cloud Function shall only be available to either a) authenticated users or b) users of a specific domain. (I have a domain, let's say #testorganization.com) Or if I can identify my particular app (api key?) then that would work, too.
The moment I add a member "allAuthenticatedUsers" with role to invoke the function (and remove "allUsers) nothing happens. I also tried adding the entire domain, but that wouldn't work. (duh) Also tried adding my service account (trial and error at this point) and didn't seem to work.
In my Node JS code I am actually receiving the UID of the authenticated user, so it appears that some kind of user authentication information is already being exchanged.
With that knowledge, I can (successfully tried this) get the UID and cross check that against my database and verify a user that way, but seems unnecessary and I should be able to make the permissions work. (lock the function down entirely) Plus this took a really long time just finish this cross check. Or is this pretty standard procedure to do?
Like this-->
const usersRef = admin.firestore().collection('users').doc(uid)
usersRef.get()
.then((docSnapshot) => {
if (docSnapshot.exists) {
usersRef.onSnapshot((doc) => {
console.log('User Type logged in: ', doc.data().userCategory)
console.log('User Title: ', doc.data().userTitle)
});
} else {
console.log('User does not exist')
}
});
Edit:
So while not having figured out how to shut down the function entirely, I did discover that instead of cross checking my users, I can simple check for auth like this:
if (context.auth){
//user is auth'd
} else {
//no auth
}
That's a little bit better, I guess. (but still doesn't technically prevent access to the function?!)
Thank you so much for any help. Much appreciated.
Edit2:
Here is a screensshot of the area in the cloud console (for cloud function roles/privileges) that I am referring to:
https://imgur.com/cBsjaaL
With a Callable Cloud Function, if you want to ensure that only authenticated users can trigger it's business logic, you actually don't need to configure any extra "cloud function roles/privileges" as shown at the bottom of your question.
By default, with a Callable Cloud Function you will get, when available, "Firebase Authentication and FCM tokens automatically included in requests" and it will "automatically deserializes the request body and validates auth tokens", as explained in the doc.
So you just have to follow the doc and use the context parameter. As you have mentioned in your question, you can check the user is authenticated by doing:
if (context.auth) {
//...
}
If you want to verify the user email, you would do:
exports.addMessage = functions.https.onCall((data, context) => {
const uid = context.auth.uid;
return admin.auth().getUser(uid)
.then(userRecord => {
const userEmail = userRecord.email;
//....
})
.catch(function(error) {
console.log('Error fetching user data:', error);
// Send back an error to the front end
// See https://firebase.google.com/docs/functions/callable#handle_errors
});
});
You will find more examples on how to "work with" users with the Admin SDK here in the doc.
I'm currently using Firebase Firestore for an Android Project but I'm having some trouble retrieving data when the phone is on Airplane mode. Here's my code:
public void loadThings() {
FirebaseFirestore db = FirebaseFirestore.getInstance();
db.collection("medidas").whereEqualTo("id_user", mAuth.getCurrentUser().getUid()).get().addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
#Override
public void onComplete(#NonNull Task<QuerySnapshot> task) {
if (task.isSuccessful()) {
QuerySnapshot snapshot = task.getResult();
int tam = snapshot.getDocuments().size();
data = new String[tam];
StringBuilder temp;
DocumentSnapshot doc;
for (int i = 0; i < tam; i++) {
temp = new StringBuilder();
doc = snapshot.getDocuments().get(i);
temp.append("Date: ").append(doc.get("fecha")).append(";");
temp.append("Min: ").append(doc.get("min")).append(";");
temp.append("Max: ").append(doc.get("max")).append(";");
temp.append("Avg: ").append(doc.get("avg")).append(";");
data[i] = temp.toString();
}
if(tam==0)
{
noMeasures();
}
}
else
{
data=null;
}
mLoadingIndicator.setVisibility(View.INVISIBLE);
mMeasuresAdapter.setMeasuresData(data);
if (null == data) {
showErrorMessage();
} else {
showMeasuresDataView();
}
}
});
}
The specific problem is that sometimes it takes to long to show the data (more than 10 seconds) while some others it shows the data inmediatly. Since the phone is on airplane mode is obvious that the data I'm retrieving comes from the cache. However, I don't understand why it is taking so long sometimes. Is there some way to tell firestore explicitly to bring data from the cache instead of trying to fetch it from the server? Thanks in advance
Is there some way to tell firestore explicitly to bring data from the cache instead of trying to fetch it from the server?
Yes it is, starting with the 16.0.0 SDK version update, you can achieve this with the help of the DocumentReference.get(Source source) and Query.get(Source source) methods.
By default, get() attempts to provide up-to-date data when possible by waiting for data from the server, but it may return cached data or fail if you are offline and the server cannot be reached. This behavior can be altered via the Source parameter.
So you can pass as an argument to the DocumentReference or to the Query the source, so we can force the retrieval of data from the server only, chache only or attempt server and fall back to the cache.
So in code might look like this:
FirebaseFirestore db = FirebaseFirestore.getInstance();
DocumentReference docIdRef = db.collection("yourCollection").document("yourDocument");
docIdRef.get(Source.CACHE).addOnSuccessListener(new OnSuccessListener<DocumentSnapshot>() {
#Override
public void onSuccess(DocumentSnapshot documentSnapshot) {
//Get data from the documentSnapshot object
}
});
In this case, we force the data to be retrieved from the CACHE only. If you want to force the data to be retrieved from the SERVER only, you should pass as an argument to the get() method, Source.SERVER. More informations here.
I'm getting this exception in my logs: "clarifai2.exception.ClarifaiException: Maximum attempts reached of getting a default model." which is being generated by a large number of my android app users, but I am unable to replicate the exception or determine what is causing it. Any help on how to recreate or even better prevent this exception from occurring would be very helpful.
UPDATE:
I found the issue and am able to reproduce on demand, if no internet connection is available the Clarifai library throws this exception, there is no check for network connection state within the library. I can check for network connection in my app before building the clarifai client, but if the network connection is lost after the client is built this exception is generated, any ideas on how to prevent this? Thank you.
1 - Ensure data connection to internet is available on phone.
2 - Build clarifai client in onCreate
3 - Send clarifai predict request to food model
4 - Disable wifi and mobile data connections on phone
5 - Wait 10 to 15 seconds, can navigate to other activities, then clarifai throws "Clarifai Exception: Maximum attempts reached of getting a default model" and crashes the app.
Clarifai Library class which can throw this exception is available HERE
I call the buildClarifaiClient method below in my onCreate of the requesting activity.
private void buildClarifaiClient(){
if(clarifaiClient == null){
clarifaiClient = new ClarifaiBuilder("KeyString")
.client(new OkHttpClient.Builder()
.connectTimeout(60, TimeUnit.SECONDS)
.readTimeout(60, TimeUnit.SECONDS)
.writeTimeout(60, TimeUnit.SECONDS)
//.addInterceptor(new HttpLoggingInterceptor(logger::info).setLevel(HttpLoggingInterceptor.Level.BASIC))
.build()
)
.buildSync();
}
}
Request to Clarifai on ActivityResult after a picture is taken:
#Override
protected void onActivityResult(int requestCode, int resultCode, Intent data) {
if (requestCode == REQUEST_IMAGE_CAPTURE && resultCode == RESULT_OK) {
if(clarifaiClient != null) {
snapSearchActivityResult = true;
taskClarifaiRequest = new AsyncTask<Void, Void, ClarifaiResponse<List<ClarifaiOutput<Concept>>>>() {
#Override
protected void onPreExecute() {
}
#Override
protected ClarifaiResponse<List<ClarifaiOutput<Concept>>> doInBackground(Void... params) {
// The default Clarifai model that identifies concepts in images
// Use this model to predict, with the image that the user just selected as the input
return clarifaiClient.getDefaultModels().foodModel().predict()
.withInputs(ClarifaiInput.forImage(ClarifaiImage.of(getPicByteData())))
.executeSync();
}
#Override
protected void onPostExecute(ClarifaiResponse<List<ClarifaiOutput<Concept>>> response) {
//setBusy(false);
if (!response.isSuccessful()) {
showErrorSnackbar(getString(R.string.clarifaiAPIContactError));
return;
}
final List<ClarifaiOutput<Concept>> predictions = response.get();
if (predictions.isEmpty()) {
showErrorSnackbar(getString(R.string.clarifaiAPIResultsError));
return;
}
List<Concept> concepts = predictions.get(0).data();
int conceptsSize = concepts.size();
Log.d("conceptsSize", String.valueOf(conceptsSize));
for (Concept c : concepts) {
// Do something with the value
Log.d("foodName", String.valueOf(c.name()));
Log.d("foodProb", String.valueOf(c.value()));
}
}
private void showErrorSnackbar(String errorString) {
Snackbar.make(
parentLayout,
errorString,
Snackbar.LENGTH_LONG
).show();
}
};
taskClarifaiRequest.execute();
} else {
Snackbar.make(
parentLayout,
"Unable to connect to Image API, try again.",
Snackbar.LENGTH_LONG
).show();
buildClarifaiClient();
}
}
Stacktrace:
Exception clarifai2.exception.ClarifaiException: Maximum attempts
reached of getting a default model.
clarifai2.dto.model.DefaultModels.update ()
clarifai2.dto.model.DefaultModels.access$000 ()
clarifai2.dto.model.DefaultModels$1.run ()
java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1113)
java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:588)
java.lang.Thread.run (Thread.java:818)
I'd say that the API key you use here doesn't have permissions for doing get model request. When you access the food model via getDefaultModels(), the model is first updated (via get model request) and only after that the predict request is run. If the first one fails, the second one doesn't run.
In the Java API client 2.3 which is about to be released next week, the call getDefaultModels() will no longer initiate get model request since most of the time you require only an ID of a default model, not the complete model data which get model request returns. Since only predict request will be performed, you will not need to have get model request permissions on your API key.
In the meanwhile, you can do one of two things. Either:
Add Models:Get permission to your API key. You do this by going to the API keys page, editing your API key, clicking ADVANCED under Scopes and checking Models:Get. You probably already have Predict permission checked.
Or use the model ID directly on the predict call. I've inserted the food default model ID (which you can also see in the DefaultModels.java link you posted above) into the predict call for you: client.predict("bd367be194cf45149e75f01d59f77ba7").withInputs(...).executeSync();.
I am new to azure but i know certain things like how to retrieve and store data to azure , i followed azure official documentation for this purpose.
Link is Here - https://azure.microsoft.com/en-in/documentation/articles/mobile-services-android-get-started-data/
But the problem is, this tutorial is only showing How to retrieve and use Data from azure using Adapters and Lists . I want to know , How can i retrieve a single value from azure mobile services and how to use it in android.
Plzz provide me both backend code (if there is any) and java code for this . THANKS in advance
I got it solved. No need to create a custom API.
Just follow the basics , Here is the code :-
final String[] design = new String[1];
private MobileServiceTable<User> mUser;
mUser = mClient.getTable(User.class);
new AsyncTask<Void, Void, Void>() {
#Override
protected Void doInBackground(Void... params) {
try {
final MobileServiceList<User> result =
mUser.where().field("name").eq(x).execute().get();
for (User item : result) {
// Log.i(TAG, "Read object with ID " + item.id);
desig[0] = item.getDesignation(); //getDesignation() is a function in User class ie- they are getters and setters
Log.v("FINALLY DESIGNATION IS", desig[0]);
}
} catch (Exception exception) {
exception.printStackTrace();
}
return null;
}
#Override
protected void onPostExecute(Void aVoid) {
super.onPostExecute(aVoid);
designation.setText(desig[0]);
}
}.execute();
DON'T forget to create a class User for serialization and all. Also you should define the array .
FEEL FREE to write if you got it not working.
EDIT :-
design[0] is an array with size 1.
eq(x) is equal to x where , x variable contains username for which i want designation from database (azure).
You can do this with a custom API. See this link: https://azure.microsoft.com/en-us/documentation/articles/mobile-services-how-to-use-server-scripts/#custom-api
Code looks like this:
exports.post = function(request, response) {
response.send(200, "{ message: 'Hello, world!' }");
}
It's then reachable at https://todolist.azure-mobile.net/api/APIFILENAME.
If you want to access a table you can do something like:
exports.post = function(request, response) {
var userTable = tables.getTable('users');
permissionsTable
.where({ userId: user.userId})
.read({ success: sendUser });
}
function sendUser(results){
if(results.length <= 0) {
res.send(200, {});
} else {
res.send(200, {result: results[0]});
}
}
You can then follow the instructions for using the API on your Android client here: https://azure.microsoft.com/en-us/documentation/articles/mobile-services-android-call-custom-api/
How your app is written will change how this code works/looks, but it looks something like:
ListenableFuture<MarkAllResult> result = mClient.invokeApi( "UsersAPI", MarkAllResult.class );
That invokes the API. You need to write the class and Future to handle the results. The above page explains this in great detail.
The most optimal solution would be to create an api on your server which accepts an ID to return an single object/tablerow.
In your android app, you only have to call:
MobileServiceTable<YourClass> mYourTable;
mClient = new MobileServiceClient(
"https://yoursite.azurewebsites.net/",
mContext);
mYourTable = mClient.getTable(YourClass.class);
YourClass request = mYourTable.lookUp(someId).get();
// request -> https://yoursite.azurewebsites.net/tables/yourclass/someId
YourClass should have the same properties as the object on the server.
So I'm building a signup procedure that needs the user to verify their phone number by receiving a code by sms. I'm using Parse as the backend system and I'm using Twilio service which comes included in Parse to take care of the sms function. I have been successful in sending the verification code to user's number.
This is my parse cloud code:
var client = require('twilio')('ACb3....', '2b3....');
//Send an SMS text message
Parse.Cloud.define("sendVerificationCode", function(request, response) {
var verificationCode = Math.floor(Math.random()*999999);
client.sendSms({
From: "+61437877758",
To: request.params.phoneNumber,
Body: "Your verification code is " + verificationCode + "."
}, function(err, responseData) {
if (err) {
response.error(err);
} else {
response.success("Success");
}
});
});
This is the code from the app:
HashMap<String, Object> params = new HashMap<String, Object>();
params.put("phoneNumber", userNumber);
ParseCloud.callFunctionInBackground("sendVerificationCode", params, new FunctionCallback<String>() {
public void done(String result, ParseException e) {
if (e == null) {
Log.d("Parse", result);
Intent i = new Intent(SignupActivity.this, PhoneVerificationActivity.class);
startActivity(i);
} else {
Toast.makeText(SignupActivity.this, "there was a problem with connection", Toast.LENGTH_LONG).show();
}
}
});
Now I would like to know how can I send that verification code back to my android app from Parse Cloud after success, so tat I can check the verification code against the code user puts in the EditText
if (err) {
response.error(err);
} else {
*//So the code for sending the verification code back goes here:*
response.success("Success");
}
Do I need to use Json and Rest API?, how can I call and grab this verification code from the app?.
I would really appreciate your help. Thanks.
One way would be to return it in response.success...
response.success({ status: "success", verificationCode: ... });
Another way, a better way, is to not trust the client with this. Store a record of it on an object on the server... When the user enters the validation code, call back into another function to check if it is valid. An example of this type of system can be seen in this old out-dated GitHub login example: https://github.com/ParsePlatform/CloudCodeOAuthGitHubTutorial/blob/master/cloud/main.js#L116