We Keep Coding

Jsoup not connecting to webpage in Android Studio

I am working on a project right now where I use jsoup in a class with the function retrieveMedia in order to return an ArrayList filled with data from the webpage. I run it in a thread since you shouldn't be connecting to URLs from the main thread. I run it and join it. However, it doesn't work (I tested the same code in Eclipse separate from Android Studio and it worked fine). It seems that no matter what I do I can't get jsoup to connect to the webpage. Below is my class MediaRetriever.
public class MediaRetreiever {
public ArrayList<Media> retrieveMedia() {
ArrayList<Media> mediaOutput = new ArrayList<Media>(); //Store each scraped post
Thread downloadThread = new Thread(new Runnable() {
public void run() {
Document doc = null;
try {
doc = Jsoup.connect(<Website Im connecting to>).timeout(20000).get();
} catch (IOException e) {
System.out.println("Failed to connect to webpage.");
mediaOutput.add(new Media("Failed to connect", "oops", "", "oh well"));
return;
}
Elements mediaFeed = doc.getElementById("main").getElementsByClass("node");
for (Element e : mediaFeed) {
String title, author, imageUrl, content;
title=e.getElementsByClass("title").text().trim();
author=e.getElementsByClass("content").tagName("p").select("em").text().trim();
content=e.getElementsByClass("content").text().replace(author,"").trim();
Media media = new Media(title, author, "", content);
mediaOutput.add(media);
}
}
});
downloadThread.start();
try {
downloadThread.join();
} catch (InterruptedException e) {
e.printStackTrace();
}
return mediaOutput;
}
}
Running this class's method from another class and it doesn't ever connect. Any ideas?

Since you say that the problem persists only in Android, it looks like that you should add the user agent string to your request - first get the user agent string of a browser that displays correctly the site, and then add it to the request:
doc = Jsoup.connect(<Website Im connecting to>)
.userAgent("your-user-agent-string")
.timeout(20000).get();
And as a sidenote - if you are catching exception, don't print your own error message - print the original message, it may be very useful.

Write to Firestore from inside Google Cloud Dataflow

The core issue I have right now, is when I run the Dataflow pipeline deployed to Google Cloud Dataflow, I get the error:
java.lang.IllegalStateException: FirebaseApp with name [DEFAULT] doesn't exist.
If I run the same pipeline locally, it all works. So I SUSPECT either an authentication issue, or an environment issue.
Code bits:
The DEPLOY and REAL variables are to control whether or not to push to Cloud (or run locally) and whether or not to use my Pub/Sub source, or use moc'd data. Switching between moc'd and pub/sub data doesn't seem to have an effect on the Firestore situation at all. Only the deploy or not does.
The main() piece where I'm initializing the Firestore application:
public class BreakingDataTransactions {
// When true, this pulls from the specified Pub/Sub topic
static Boolean REAL = true;
// when set to true the job gets deployed to Cloud Dataflow
static Boolean DEPLOY = true;
public static void main(String[] args) {
// validate our env vars
if (GlobalVars.projectId == null ||
GlobalVars.pubsubTopic == null ||
GlobalVars.gcsBucket == null ||
GlobalVars.region == null) {
System.out.println("You have to set environment variables for project (BREAKING_PROJECT), pubsub topic (BREAKING_PUBSUB), region (BREAKING_REGION) and Cloud Storage bucket for staging (BREAKING_DATAFLOW_BUCKET) in order to deploy this pipeline.");
System.exit(1);
}
// Initialize our Firestore instance
try {
GoogleCredentials credentials = GoogleCredentials.getApplicationDefault();
System.out.println("*************************");
System.out.println(credentials);
FirebaseOptions firebaseOptions =
new FirebaseOptions.Builder()
.setCredentials(credentials)
.setProjectId(GlobalVars.projectId)
.build();
FirebaseApp firebaseApp = FirebaseApp.initializeApp(firebaseOptions);
} catch (IOException e) {
e.printStackTrace();
}
// Start dataflow pipeline
DataflowPipelineOptions options =
PipelineOptionsFactory.create().as(DataflowPipelineOptions.class);
options.setProject(GlobalVars.projectId);
if (DEPLOY) {
options.setRunner(DataflowRunner.class);
options.setTempLocation(GlobalVars.gcsBucket);
options.setRegion(GlobalVars.region);
}
Pipeline p = Pipeline.create(options);
And the piece where I'm processing things:
PCollection<Data> dataCollection =
jsonStrings
.apply(ParDo.of(JSONToPOJO.create(Data.class)))
.setCoder(AvroCoder.of(Data.class));
PCollection<Result> result =
dataCollection
.apply(Window.into(FixedWindows.of(Duration.standardSeconds(1))))
.apply(WithKeys.of(x -> x.operation + "-" + x.job_id))
.setCoder(KvCoder.of(StringUtf8Coder.of(), AvroCoder.of(Data.class)))
.apply(Combine.<String, Data, Result>perKey(new DataAnalysis()))
.apply(Reify.windowsInValue())
.apply(MapElements.into(TypeDescriptor.of(Result.class))
.<KV<String, ValueInSingleWindow<Result>>>via(
x -> {
Result r = new Result();
String key = x.getKey();
r.query_action = key.substring(0, key.indexOf("-"));
r.job_id = key.substring(key.indexOf("-") + 1);
r.average_latency = x.getValue().getValue().average_latency;
r.failure_percent = x.getValue().getValue().failure_percent;
r.timestamp = x.getValue().getTimestamp().getMillis();
return r;
}));
// this node will (hopefully) actually write out to Firestore
result.apply(ParDo.of(new FireStoreOutput()));
And finally, the FireStoreOutput class:
public static class FireStoreOutput extends DoFn<Result, String> {
Firestore db;
#ProcessElement
public void processElement(#Element Result result) {
db = FirestoreClient.getFirestore();
DocumentReference docRef = db.collection("events")
.document("next2020")
.collection("transactions")
.document(result.job_id)
.collection("transactions")
.document();
//System.out.println(docRef.getId());
// Add document data with id "alovelace" using a hashmap
Map<String, Object> data = new HashMap<>();
data.put("failure_percent", result.failure_percent);
data.put("average_latency", result.average_latency);
data.put("query_action", result.query_action);
data.put("timestamp", result.timestamp);
// asynchronously write data
ApiFuture<WriteResult> writeResult = docRef.set(data);
try {
writeResult.get();
} catch (InterruptedException e) {
e.printStackTrace();
} catch (ExecutionException e) {
e.printStackTrace();
}
;
}
}
The error occurs on the line: db = FirestoreClient.getFirestore();
I'm deploying the Dataflow job with the --serviceAccount flag specifying a service account that has permissions to do all the things.
So unless the GoogleCredentials credentials = GoogleCredentials.getApplicationDefault(); somehow doesn't work (but you see the print statement there, and it does correctly print out the credentials on build) that isn't it.
BUT, that only happens at build time...so I'm wondering if I have a persistence problem, where it initializes fine at build time, but when the job is actually running in the Cloud, it loses the initialization between the deployment and the processing. And if that's the case, how do I solve that problem?
Thanks!

Okay, I found a solution... The biggest issue was that my DAG's PCollection was split into two thread paths. I have two types of operations "read" and "write" so those results were each sending a PCollection to my FirestoreOut class, which is where I was attempting to initialize the Firestore app, resulting in the already initialized problem.
HOWEVER, making my db object a synchronized static object, and instituting a synchronized getDB() method where I initialize only if it's not set yet worked. Final updated relevant code for the FireStoreOut piece:
public static class FireStoreOutput extends DoFn<Result, String> {
static Firestore db;
public static synchronized Firestore getDB() {
if (db == null) {
System.out.println("I'm being called");
// Initialize our Firestore instance
try {
GoogleCredentials credentials = GoogleCredentials.getApplicationDefault();
System.out.println("*************************");
System.out.println(credentials);
FirebaseOptions firebaseOptions =
new FirebaseOptions.Builder()
.setCredentials(credentials)
.setProjectId(GlobalVars.projectId)
.build();
FirebaseApp firebaseApp = FirebaseApp.initializeApp(firebaseOptions);
} catch (IOException e) {
e.printStackTrace();
}
db = FirestoreClient.getFirestore();
}
return db;
}
#ProcessElement
public void processElement(#Element Result result) {
DocumentReference docRef = getDB().collection("events")
.document("next2020")
.collection("transactions")
.document(result.job_id)
.collection("transactions")
.document();

Verify if the deleteObject has actually deleted the object in AWS S3 Java sdk

I have the following method, which deletes a file from AWS S3 Bucket, however,
there is no exception thrown if the file doesn't exist
there is no success code or flag to see if the file has been deleted successfully
is there any workaround to deal with this situation.
#Override
public void deleteFile(String fileName) {
try {
this.client.deleteObject(builder ->
builder
.bucket(this.bucketName).key(fileName)
.build());
} catch (S3Exception ex) {
ex.printStackTrace();
}
}

If your request succeeded then your object is deleted. Note, that due to eventual consistency, the object is not guaranteed to disappear immediately. You need to check on the HTTP status code.
AmazonS3 as3 = new AmazonS3();
Status myStatus = as3.DeleteObject(<fill in paramters here>);
if (myStatus.Code >= 200 && myStatus.Code < 300)
{
// Success
}
else
{
// Delete Failed
// Handle specific Error Codes below
if (myStatus.Description == "AllAccessDisabled")
{
// Do something
}
if (myStatus.Description == "NoSuchKey")
{
// Do something
}
}
Also, there is an api available to check if the Object exists in S3
doesObjectExist
https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3.html#doesObjectExist-java.lang.String-java.lang.String-

Issues with converting java to c#

I'm attempting to convert the code located at How to use signalr in android Service from java to c# and have been making some progress. I'm now stuck at the final method. The java code is:
private void startSignalR() {
Platform.loadPlatformComponent(new AndroidPlatformComponent());
mInstance.setmHubConnection();
mInstance.setHubProxy();
ClientTransport clientTransport = new ServerSentEventsTransport(mInstance.mHubConnection.getLogger());
SignalRFuture<Void> signalRFuture = mInstance.mHubConnection.start(clientTransport);
try {
signalRFuture.get();
} catch (InterruptedException | ExecutionException e) {
Log.e("SimpleSignalR", e.toString());
return;
}
mInstance.sendMessage(MainActivity.unm,"Hello All!");
String CLIENT_METHOD_BROADAST_MESSAGE = "recievedMessage";
mInstance.mHubProxy.on(CLIENT_METHOD_BROADAST_MESSAGE,
new SubscriptionHandler2<String,LoginInfo>() {
#Override
public void run(final String msg,final LoginInfo loginInfo) {
final String finalMsg = loginInfo.FullName + " says " + loginInfo.Password;
Intent intent = new Intent();
intent.setAction(MY_ACTION);
intent.putExtra("DATAPASSED", finalMsg);
sendBroadcast(intent);
}
}
, String.class,LoginInfo.class);
}
Using a java to c# converter, this translated to:
private void startSignalR()
{
Platform.loadPlatformComponent(new AndroidPlatformComponent());
mInstance.setmHubConnection();
mInstance.setHubProxy();
ClientTransport clientTransport = new ServerSentEventsTransport(mInstance.mHubConnection.Logger);
SignalRFuture<Void> signalRFuture = mInstance.mHubConnection.Start(clientTransport);
try
{
signalRFuture.get();
}
catch (Exception e) when (e is InterruptedException || e is ExecutionException)
{
// Log.e("SimpleSignalR", e.ToString());
return;
}
mInstance.sendMessage("", "Hello All!");
string CLIENT_METHOD_BROADAST_MESSAGE = "recievedMessage";
//String CLIENT_METHOD_BROADAST_MESSAGE = "messageReceived";
mInstance.mHubProxy.on(CLIENT_METHOD_BROADAST_MESSAGE, new SubscriptionHandler2AnonymousInnerClass(this)
, typeof(string), typeof(LoginInfo));
}
private class SubscriptionHandler2AnonymousInnerClass : SubscriptionHandler2<string, LoginInfo>
{
private readonly SignalRSrv outerInstance;
public SubscriptionHandler2AnonymousInnerClass(SignalRSrv outerInstance)
{
this.outerInstance = outerInstance;
}
//JAVA TO C# CONVERTER WARNING: 'final' parameters are not available in .NET:
//ORIGINAL LINE: #Override public void run(final String msg,final LoginInfo loginInfo)
public override void run(string msg, LoginInfo loginInfo)
{
//JAVA TO C# CONVERTER WARNING: The original Java variable was marked 'final':
//ORIGINAL LINE: final String finalMsg = loginInfo.FullName + " says " + loginInfo.Password;
string finalMsg = loginInfo.FullName + " says " + loginInfo.Password;
Intent intent = new Intent();
intent.Action = MY_ACTION;
intent.PutExtra("DATAPASSED", finalMsg);
sendBroadcast(intent);
}
}
This, of course, generated several errors in Visual Studio 2017.
First, the line Platform.loadPlatformComponent(new AndroidPlatformComponent()); generated the error Platform is inaccessible due to its protection level. Platform in Xamarin for Visual Studio 2017 is indeed protected and is a internal class in System and I cannot change this, so I'm at a loss as how to proceed with it. The same line generates the error The type or namespace name 'AndroidPlatformComponent' could not be found, these errors a numerous and not unexpected I just can't find an equivalent to AndroidPlatformComponent in Visual Studio 2017 so I'm at a loss as how to solve this one.
Next, on this line ClientTransport clientTransport = new ServerSentEventsTransport(mInstance.mHubConnection.Logger); generates the error The type or namespace name 'ClientTransport' could not be found, I was also unable to find an equivalent to this and again I'm at a loss as to proceed. Also on this line, .Logger is not defined for the hub connection, apparently it's .getLogger() in java, I was unable to find an equivalent for this one as well.
Next the line SignalRFuture<Void> signalRFuture = mInstance.mHubConnection.Start(clientTransport);' generates the error 1The type or namespace name 'SignalRFuture<>' could not be found, this seemes to be specific to SignalR, again, I am unable to find an equivalent.
The next one has me totally stumped, the line private class SubscriptionHandler2AnonymousInnerClass : SubscriptionHandler2<string, LoginInfo> generates the error The type or namespace name 'SubscriptionHandler2<,>' could not be found. I've looked everywhere online and read up on AnonymousInnerClass, but it was not help with this.
I'm hoping that the users here are more familiar with SignalR and the differences between c# functionality and java functionality. I'm not at all familiar with java nor am I familiar with SignalR and foreground services.

As it turns out, the last method in the java code I was converting was wiring up an event to pass the message received from the hub to the activity. In c# / Visual Studio (2017), that's done very differently which is why I didn't understand/recognize what was going on. So I created a handler in C# and execute a popup message for the message. This in itself may pose problems, but at least I know what's going on. This is the code I wrote to start SignalR from within the service and WireUp the handler:
private void startSignalR()
{
// Company, Department, and section are private variables
// their values are pulled from the intent bundle
// in the OnBind method, that code is:
// Bundle bundlee = intent.GetBundleExtra("TheBundle");
// MyUser = bundlee.GetParcelable("MyUser") as User;
// This information is put into the bundle when the user is logged in.
// I then pass that information to the SignalR client
// when I set the hub connection and placed on the querystring to the hub.
mInstance.setmHubConnection(username, firstname,lastname,company,department,section);
mInstance.setHubProxy();
try
{
// Connect the client to the hup
mInstance.mHubConnection.Start();
// Set the event handler
mInstance.WireUp();
}
catch (System.Exception e) when (e is InterruptedException || e is ExecutionException)
{
ShowMessage("Error: + e.Message)
}
}
This is the WireUp code, this is a method in the client code:
public void WireUp()
{
// set the event handler
mHubProxy.On("broadcastMessage", (string platform, string message) =>
{
if (OnMessageReceived != null)
OnMessageReceived(this, string.Format("{0}: {1}", platform, message));
});
}
As I had anticipated, the popup message won't appear when the app is in the background, so I'm researching a workaround

webservice client implementaion in axis2-1.6.2 in java

I have implemented webservice client in axis2-1.6.2 in java and I get response when I call first time and for subsequent second time I get below error
java.lang.NullPointerException
at org.apache.axis2.client.OperationClient.prepareMessageContext(OperationClient.java:293)
at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:180)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:165)
at org.apache.axis2.ccws.CustomerCareServiceStub.subscriberRetrieveLite(CustomerCareServiceStub.java:2380)
at Prepost.SubscriberRetrieveBalance.subscriberRetrieveLite(SubscriberRetrieveBalance.java:111)
at Prepost.CheckUser.doGet(CheckUser.java:149)
here is the API implementation class constructor which sets unique parameter which is same for all requests
public SubscriberRetrieveBalance(String url, String strCON_TimeOut, String strSO_TimeOut) {
try {
this.url = url;
stub = new CustomerCareServiceStub(url);
ServiceClient sClient = stub._getServiceClient();
Options options = sClient.getOptions();
options.setProperty(HTTPConstants.REUSE_HTTP_CLIENT, Constants.VALUE_TRUE);
options.setProperty(AddressingConstants.WS_ADDRESSING_VERSION, AddressingConstants.Submission.WSA_NAMESPACE);
//options.setTimeOutInMilliSeconds(2000);
TransportInDescription transportIn = new TransportInDescription("HTTP");
options.setTransportIn(transportIn);
options.setProperty(HTTPConstants.SO_TIMEOUT, Integer.parseInt(strSO_TimeOut));
options.setProperty(HTTPConstants.CONNECTION_TIMEOUT, Integer.parseInt(strCON_TimeOut));
sClient.setOptions(options);
} catch (Exception e) {
if (e.getMessage().equals("Can not find the external id")) {
System.out.println("Exception ::" + e.getMessage());
}
}
}
and it is called in a servlet and for performance issue I make object of this class for different-2 states(urls) and saved these object to hashmap when first request comes for respective states then make new object and use that object for subsequent requests for that state
SubscriberRetrieveBalance objBal = null;
BalanceBean bal = new BalanceBean();
if (mapObj.isEmpty() || (mapObj.get(strIP) == null)) {
objBal = new SubscriberRetrieveBalance(url, strCON_TimeOut, strSO_TimeOut);
mapObj.put(strIP, objBal);
} else {
objBal = mapObj.get(strIP);
}
bal = objBal.subscriberRetrieveLite(strMsisdn, userId, token, strCircleId, strCircleName, strSessionId, strDlgId);
first time it gives response and then gives nullpointer exception and above error for all requests that belongs to that state
This code is working fine with axis2-1.5
Is there any change in axis2-1.6.2 version that every time it needs new object of API implemented class
Please suggest.