Write to Firestore from inside Google Cloud Dataflow - java

The core issue I have right now, is when I run the Dataflow pipeline deployed to Google Cloud Dataflow, I get the error:
java.lang.IllegalStateException: FirebaseApp with name [DEFAULT] doesn't exist.
If I run the same pipeline locally, it all works. So I SUSPECT either an authentication issue, or an environment issue.
Code bits:
The DEPLOY and REAL variables are to control whether or not to push to Cloud (or run locally) and whether or not to use my Pub/Sub source, or use moc'd data. Switching between moc'd and pub/sub data doesn't seem to have an effect on the Firestore situation at all. Only the deploy or not does.
The main() piece where I'm initializing the Firestore application:
public class BreakingDataTransactions {
// When true, this pulls from the specified Pub/Sub topic
static Boolean REAL = true;
// when set to true the job gets deployed to Cloud Dataflow
static Boolean DEPLOY = true;
public static void main(String[] args) {
// validate our env vars
if (GlobalVars.projectId == null ||
GlobalVars.pubsubTopic == null ||
GlobalVars.gcsBucket == null ||
GlobalVars.region == null) {
System.out.println("You have to set environment variables for project (BREAKING_PROJECT), pubsub topic (BREAKING_PUBSUB), region (BREAKING_REGION) and Cloud Storage bucket for staging (BREAKING_DATAFLOW_BUCKET) in order to deploy this pipeline.");
System.exit(1);
}
// Initialize our Firestore instance
try {
GoogleCredentials credentials = GoogleCredentials.getApplicationDefault();
System.out.println("*************************");
System.out.println(credentials);
FirebaseOptions firebaseOptions =
new FirebaseOptions.Builder()
.setCredentials(credentials)
.setProjectId(GlobalVars.projectId)
.build();
FirebaseApp firebaseApp = FirebaseApp.initializeApp(firebaseOptions);
} catch (IOException e) {
e.printStackTrace();
}
// Start dataflow pipeline
DataflowPipelineOptions options =
PipelineOptionsFactory.create().as(DataflowPipelineOptions.class);
options.setProject(GlobalVars.projectId);
if (DEPLOY) {
options.setRunner(DataflowRunner.class);
options.setTempLocation(GlobalVars.gcsBucket);
options.setRegion(GlobalVars.region);
}
Pipeline p = Pipeline.create(options);
And the piece where I'm processing things:
PCollection<Data> dataCollection =
jsonStrings
.apply(ParDo.of(JSONToPOJO.create(Data.class)))
.setCoder(AvroCoder.of(Data.class));
PCollection<Result> result =
dataCollection
.apply(Window.into(FixedWindows.of(Duration.standardSeconds(1))))
.apply(WithKeys.of(x -> x.operation + "-" + x.job_id))
.setCoder(KvCoder.of(StringUtf8Coder.of(), AvroCoder.of(Data.class)))
.apply(Combine.<String, Data, Result>perKey(new DataAnalysis()))
.apply(Reify.windowsInValue())
.apply(MapElements.into(TypeDescriptor.of(Result.class))
.<KV<String, ValueInSingleWindow<Result>>>via(
x -> {
Result r = new Result();
String key = x.getKey();
r.query_action = key.substring(0, key.indexOf("-"));
r.job_id = key.substring(key.indexOf("-") + 1);
r.average_latency = x.getValue().getValue().average_latency;
r.failure_percent = x.getValue().getValue().failure_percent;
r.timestamp = x.getValue().getTimestamp().getMillis();
return r;
}));
// this node will (hopefully) actually write out to Firestore
result.apply(ParDo.of(new FireStoreOutput()));
And finally, the FireStoreOutput class:
public static class FireStoreOutput extends DoFn<Result, String> {
Firestore db;
#ProcessElement
public void processElement(#Element Result result) {
db = FirestoreClient.getFirestore();
DocumentReference docRef = db.collection("events")
.document("next2020")
.collection("transactions")
.document(result.job_id)
.collection("transactions")
.document();
//System.out.println(docRef.getId());
// Add document data with id "alovelace" using a hashmap
Map<String, Object> data = new HashMap<>();
data.put("failure_percent", result.failure_percent);
data.put("average_latency", result.average_latency);
data.put("query_action", result.query_action);
data.put("timestamp", result.timestamp);
// asynchronously write data
ApiFuture<WriteResult> writeResult = docRef.set(data);
try {
writeResult.get();
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
;
}
}
The error occurs on the line: db = FirestoreClient.getFirestore();
I'm deploying the Dataflow job with the --serviceAccount flag specifying a service account that has permissions to do all the things.
So unless the GoogleCredentials credentials = GoogleCredentials.getApplicationDefault(); somehow doesn't work (but you see the print statement there, and it does correctly print out the credentials on build) that isn't it.
BUT, that only happens at build time...so I'm wondering if I have a persistence problem, where it initializes fine at build time, but when the job is actually running in the Cloud, it loses the initialization between the deployment and the processing. And if that's the case, how do I solve that problem?
Thanks!

Okay, I found a solution... The biggest issue was that my DAG's PCollection was split into two thread paths. I have two types of operations "read" and "write" so those results were each sending a PCollection to my FirestoreOut class, which is where I was attempting to initialize the Firestore app, resulting in the already initialized problem.
HOWEVER, making my db object a synchronized static object, and instituting a synchronized getDB() method where I initialize only if it's not set yet worked. Final updated relevant code for the FireStoreOut piece:
public static class FireStoreOutput extends DoFn<Result, String> {
static Firestore db;
public static synchronized Firestore getDB() {
if (db == null) {
System.out.println("I'm being called");
// Initialize our Firestore instance
try {
GoogleCredentials credentials = GoogleCredentials.getApplicationDefault();
System.out.println("*************************");
System.out.println(credentials);
FirebaseOptions firebaseOptions =
new FirebaseOptions.Builder()
.setCredentials(credentials)
.setProjectId(GlobalVars.projectId)
.build();
FirebaseApp firebaseApp = FirebaseApp.initializeApp(firebaseOptions);
} catch (IOException e) {
e.printStackTrace();
}
db = FirestoreClient.getFirestore();
}
return db;
}
#ProcessElement
public void processElement(#Element Result result) {
DocumentReference docRef = getDB().collection("events")
.document("next2020")
.collection("transactions")
.document(result.job_id)
.collection("transactions")
.document();

Related

How to swtich the state of workitem with tfs Java sdk

I try to swtich the state of a workitem from "New" to "Active" like follows:
WorkItemCollection co = tpc.getWorkItemClient().query("select xxxxxx...");
WorkItem newWorkItem = co.getWorkItem(0);
newWorkItem.getFields().getField(CoreFieldReferenceNames.STATE).setValue("Active");
but there is error reported like this :
Exception in thread "main" java.lang.IllegalArgumentException: field id [10112] does not exist in this collection (wi=5789377,size=9)
I assume there's something wrong with this method or this method doesn't support Azure Devops Server 2019U1 since this package is last published in 2018.
My Test code:
public class HelloJava {
public static void main(String[] args) {
// TODO Auto-generated method stub
URI serverURI = null;
try {
serverURI = new URI("xxxxxxxx");
} catch (URISyntaxException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
Credentials credentials = new UsernamePasswordCredentials("xxx","xxx");
TFSTeamProjectCollection tpc = new TFSTeamProjectCollection(serverURI, credentials);
tpc.authenticate();
WorkItemClient myClient = tpc.getWorkItemClient();
WorkItemCollection myWorkitemCollection = myClient.query("SELECT [System.Id],[System.WorkItemType],[System.State] FROM workitems");
//Get workItem Task1
WorkItem myWorkItem = myWorkitemCollection.getWorkItem(0);
FieldCollection myFieldCollection = myWorkItem.getFields();
Field myField = myFieldCollection.getField(CoreFieldReferenceNames.STATE);
//Get workItem Task2
WorkItem myWorkItem2 = myWorkitemCollection.getWorkItem(1);
FieldCollection myFieldCollection2 = myWorkItem2.getFields();
Field myField2 = myFieldCollection2.getField(CoreFieldReferenceNames.STATE);
System.out.println(myField.getValue());
System.out.println(myField2.getValue());
myField2.setValue(myField.getValue()); //Where the error occurs.
System.out.println(myField.getValue());
System.out.println(myField2.getValue());
}
}
The Result:
The setValue() method will throw exception even when I'm trying to set Task1's state with Task2's state.
A strange thing is that it won't throw error if I give the current state as input... (If task1's state is To Do, it won't throw error for xx.setValue("To Do"). If I change the state to Doing via web portal, then the code throws error if I enter To Do next time!!!)
So I think you have to report this issue to the team of the sdk here to get a fix or share your feedback since maybe it just doesn't work for new Azure Devops Server 2019.

Verify if the deleteObject has actually deleted the object in AWS S3 Java sdk

I have the following method, which deletes a file from AWS S3 Bucket, however,
there is no exception thrown if the file doesn't exist
there is no success code or flag to see if the file has been deleted successfully
is there any workaround to deal with this situation.
#Override
public void deleteFile(String fileName) {
try {
this.client.deleteObject(builder ->
builder
.bucket(this.bucketName).key(fileName)
.build());
} catch (S3Exception ex) {
ex.printStackTrace();
}
}
If your request succeeded then your object is deleted. Note, that due to eventual consistency, the object is not guaranteed to disappear immediately. You need to check on the HTTP status code.
AmazonS3 as3 = new AmazonS3();
Status myStatus = as3.DeleteObject(<fill in paramters here>);
if (myStatus.Code >= 200 && myStatus.Code < 300)
{
// Success
}
else
{
// Delete Failed
// Handle specific Error Codes below
if (myStatus.Description == "AllAccessDisabled")
{
// Do something
}
if (myStatus.Description == "NoSuchKey")
{
// Do something
}
}
Also, there is an api available to check if the Object exists in S3
doesObjectExist
https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3.html#doesObjectExist-java.lang.String-java.lang.String-

How can I make tests wait for Vert.x Verticle deployment to be completed

I am implementing tests for my Vert.x application, but I am having issues in making Vert.x wait for the deploy of the Verticle in a graceful way.
This is my #BeforeClass Method:
#BeforeClass
public static void before(TestContext context)
{
vertx = Vertx.vertx();
DeploymentOptions options = new DeploymentOptions();
byte[] encoded;
JsonObject config;
try {
encoded = Files.readAllBytes(Paths.get("src/main/resources/config.json"));
config = new JsonObject(new String(encoded, Charset.defaultCharset()));
options.setConfig(config);
jdbc = JDBCClient.createShared(vertx, config , "PostgreSQL");
deployVerticle((result) -> loadTestData((result), jdbc), options);
while (true)
{
if (vertx.deploymentIDs().size() > 0)
break;
}
} catch
(IOException e)
{
e.printStackTrace();
}
}
Also, here is the implementation for the deployVerticle and loadTestData methods:
private static void deployVerticle(Handler<AsyncResult<Void>> next, DeploymentOptions options) {
vertx.deployVerticle(PostgreSQLClientVerticle.class.getName(), options, deployResult ->
{
if (deployResult.succeeded())
next.handle(Future.succeededFuture());
});
}
private static void loadTestData(AsyncResult<Void> previousOperation, JDBCClient jdbc)
{
if (previousOperation.succeeded())
{
jdbc.getConnection(connection -> {
if (connection.succeeded())
{
connection.result().query(deleteTestDataGeneration, queryResult ->
{
connection.result().close();
});
}
});
}
}
As you can see, right now I have a while (true) on the beforemethod to hold the process and make sure the verticle is actually deployed.
Otherwise, when the tests start running, the verticle is not yet fully deployed and I get a NullPointerException trying to reach the resources.
I've tried many different approaches like using CompositeFuture or using Future.compose method to make the "before tasks" sequential and make the program hold for completion.
I achieved in making those tasks sequential but failed on holding the process until they are completed.
One of the issues is, I think, the fact that the deployVerticle method returns the AsyncResult with succeeded == true after every step of the "deploy procedure" is done, instead of when the Verticle is totally up.
Meaning that the process gets a successful result before everything is actually up...but this is just a wild guess.
Bottom-line: I would like to find a way to wait for the Verticle to be totally deployed before proceeding to perform the tests, without having to do the while (true)loop that I currently have in there.
What you are missing is the Async async = context.async();. With that the unittest stays in the method until it is not set to complete. Then you are able to orchestrate your asychronous code to:
first deploy the verticle
then execute the loadtestGeneration
set the async to complete so that, the other unittest methods already can access to your verticle without nullpointerexception
I also made some cleanup, check it out:
BeforeClass
#BeforeClass
public static void before2(TestContext context){
Async async = context.async();
vertx = Vertx.vertx();
DeploymentOptions options = new DeploymentOptions();
byte[] encoded;
JsonObject config;
try {
encoded = Files.readAllBytes(Paths.get("src/main/resources/config.json"));
config = new JsonObject(new String(encoded, Charset.defaultCharset()));
options.setConfig(config);
jdbc = JDBCClient.createShared(vertx, config , "PostgreSQL");
deployVerticle2(options)
.compose(c -> loadTestData2(jdbc))
.setHandler(h -> {
if(h.succeeded()){
async.complete();
}else{
context.fail(h.cause());
}
});
} catch (IOException e){
context.fail(e);
}
}
DeployVerticle
private static Future<Void> deployVerticle2(DeploymentOptions options) {
Future<Void> future = Future.future();
vertx.deployVerticle(PostgreSQLClientVerticle.class.getName(), options, deployResult -> {
if (deployResult.failed()){
future.fail(deployResult.cause());
}else {
future.complete();
}
});
return future;
}
LoadTestData
private static Future<Void> loadTestData2(JDBCClient jdbc){
Future<Void> future = Future.future();
jdbc.getConnection(connection -> {
if (connection.succeeded()) {
connection.result().query(deleteTestDataGeneration, queryResult -> {
if(queryResult.failed()){
connection.result().close();
future.fail(queryResult.cause());
}else{
connection.result().close();
future.complete();
}
});
} else {
future.fail(connection.cause());
}
});
return future;
}

Firebase Admin SDk not working from backend Java

I am trying to push message in firebase real time database through firebase admin sdk java, but my problem is that the same code works
fine on my local system and I am able to push message successfully
in database but its not working in testing environment and testing
environment is AWS server with ubuntu installed in
it, on researching it on google I am not able to find solution for
this?
public static class User {
public String date_of_birth;
public String full_name;
public String nickname;
public User(String date_of_birth, String full_name) {
// ...
}
public User(String date_of_birth, String full_name, String nickname) {
// ...
}
}
Map<String, User> users = new HashMap<>();
users.put("alanisawesome", new User("June 23, 1912", "Alan Turing"));
users.put("gracehop", new User("December 9, 1906", "Grace Hopper"));
Code for making connection is.....
FileInputStream serviceAccount;
FirebaseOptions options;
try {
serviceAccount = new FileInputStream("/********.json");
options = new FirebaseOptions.Builder()
.setDatabaseUrl("https://myDatabaseUrl")
.setCredentials(GoogleCredentials.fromStream(serviceAccount))
.build();
FirebaseApp.initializeApp(options);
flagForConnectionWithFirebase=true;
LOGGER.info("Options are:: "+options);
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
final FirebaseDatabase database =
FirebaseDatabase.getInstance();
DatabaseReference ref = database.getReference(url);
String key=ref.push().getKey();
ref.child(key).setValueAsync(users);
After a long search I found that the server was not sync with centralised time zone. post doing that code run successfully on testing environment as well.
I found what seems to be quite a common problem that nobody cares to resolve and it is that there an dependency conflict in one of the Firebase dependencies
<groupId>com.vaadin.external.google</groupId>
<artifactId>android-json</artifactId>
It can be solved by excluding the dependency from Spring Boot Test and the pring-boot-configuration-processor. Now I am having some problems when running test, still cannot tell if they are related.

Issue with sapjco3 driver

I've written a Spring MVC (Spring framework 4.1.1) java 1.8 application that successfully connects to SAP using the sapjco3.jar driver, and I've accomplished this using the CustomDestinationDataProvider technique. I then use this drive to call RFCs in my SAP R/3 system. The java code is executed via api call from an AngularJS front end application.
Something that I've discovered occuring about 5% of the time that the call to SAP happens is the following error occurs:
NestedServletException: Handler processing failed; nested exception is
java.lang.Error: java.lang.IllegalStateException: DestinationDataProvider
already registered
Here's the contents of my CustomDestinationDataProvider.java file:
public class CustomDestinationDataProvider {
public class MyDestinationDataProvider implements DestinationDataProvider {
private DestinationDataEventListener eL;
private HashMap<String, Properties> secureDBStorage = new HashMap<String, Properties>();
public Properties getDestinationProperties(String destinationName) {
try {
Properties p = secureDBStorage.get(destinationName);
if(p!=null) {
if(p.isEmpty())
throw new DataProviderException(DataProviderException.Reason.INVALID_CONFIGURATION, "destination configuration is incorrect", null);
return p;
}
return null;
} catch(RuntimeException re) {
throw new DataProviderException(DataProviderException.Reason.INTERNAL_ERROR, re);
}
}
public void setDestinationDataEventListener(DestinationDataEventListener eventListener) {
this.eL = eventListener;
}
public boolean supportsEvents() {
return true;
}
public void changeProperties(String destName, Properties properties) {
synchronized(secureDBStorage) {
if(properties==null) {
if(secureDBStorage.remove(destName)!=null)
eL.deleted(destName);
} else {
secureDBStorage.put(destName, properties);
eL.updated(destName); // create or updated
}
}
}
}
public ArrayList<MaterialBean> executeAvailabilityCall(Properties connectProperties, String searchString) {
String destName = "ABAP_AS";
SAPDAO sapDAO = new SAPDAO();
ArrayList<MaterialBean> searchResults = new ArrayList<MaterialBean>();
MyDestinationDataProvider myProvider = new MyDestinationDataProvider();
JCoDestination dest;
try {
com.sap.conn.jco.ext.Environment.registerDestinationDataProvider(myProvider);
} catch(IllegalStateException providerAlreadyRegisteredException) {
}
myProvider.changeProperties(destName, connectProperties);
try {
dest = JCoDestinationManager.getDestination(destName);
searchResults = sapDAO.searchAvailability(dest, searchString);
} catch(JCoException e) {
e.printStackTrace();
} catch (Exception e) {
e.printStackTrace();
}
myProvider.changeProperties(destName, null);
try {
com.sap.conn.jco.ext.Environment.unregisterDestinationDataProvider(myProvider);
} catch(IllegalStateException providerAlreadyRegisteredException) {
throw new Error(providerAlreadyRegisteredException);
}
return searchResults;
} // end method executeAvailabilityCall()
} // end class CustomDestinationProvider()
My guess is that multiple api calls are occuring at the same time, and once the first query registers the destination data provider, the subsequent queries, which try to also register the destination data provider, fail because they are using the same value for 'destName' in the executeAvailabilityCall method.
Upon first glace, it seems to me like I should use a dynamic value for the destName variable instead of just using "ABAP_AS" for all queries. In other words, I should change the following line:
String destName = "ABAP_AS";
to something like this:
String destName = "ABAP_AS_" + LocalDateTime.now();
This would guarantee a unique value for the destName variable, thus a unique destination provider name.
Any thoughts on the wisdom of trying this? If this is not a good idea, what other solution would be worth exploring?
Yes, you should use multiple unique destination names for your various logon Properties configuration sets. Your class MyDestinationDataProvider is already implemented that way. But why putting a timestamp into the destination name? Why not simply using a destination name schema like "TargetSystem_<SID>_with_<username>"?
Regarding your exception, simply register MyDestinationDataProvider only once and do not permanently register and unregister it. This is not how JCo expects this to be implemented. Quote from the JCo JavaDoc at com.sap.conn.jco.ext.DestinationDataProvider:
Only one implementation of DestinationDataProvider can be registered.
For registering another implementation the infrastructure has first to
unregister the implementation that is currently registered. It is not
recommended to permanently exchange DestinationDataProvider
registrations. The one registered instance should globally manage all
destination configurations for the whole infrastructure environment.

Categories