We Keep Coding

How to set Safari download location - Selenium WebDriver

Trying to set safari (driver) download directory to specific location.
Now it will just download files to default "Downloads" folder.
String currentDirectory = System.getProperty("user.dir");
String downloadFilePath = currentDirectory+"/download/";
Already tried:
dc.setCapability("safari.download.dir", downloadFilePath);
dc.setCapability("browser.download.dir", downloadFilePath);
dc.setCapability("safari.options.dataDir", downloadFilePath); // ("safari.options.dataDir" // this part won't work)
safariOptions.setCapability("safari.options.dataDir", downloadFilePath);
safariPrefs.put("download.deafult_directory", downloadFilePath); // this one I am using for chromedriver (chromePrefs)

Well, you can try to use an HTTP connection to download the file! That way, you can for sure have it saved to the directory specified in your variable.
Most languages have APIs (or libraries) for performing HTTP requests. For example, to accomplish this in Java, you could use URL.openConnection()
String link = linkElement.getAttribute("href");
URL url = new URL(link);
HttpURLConnection httpURLConnection = (HttpURLConnection) url.openConnection();
httpURLConnection.setRequestMethod("GET");
Then you can use HttpURLConnection.getInputStream() to write the file contents to your preferred location.
try (InputStream in = httpURLConnection.getInputStream()) {
Files.copy(in, new File("/path/to/file.ext").toPath(),
StandardCopyOption.REPLACE_EXISTING);
}
ONLY IN CASE YOU ARE USING COOKIES:
You can add them to your HTTP connection. If in this case, you must use a password saved on your cookies, this can be very useful.
Set<Cookie> cookies = webDriver.manager().getCookies();
String cookieString = "";
for (Cookie cookie : cookies) {
cookieString += cookie.getName() + "=" + cookie.getValue() + ";";
}
httpURLConnection.addRequestProperty("Cookie", cookieString);

Creating a URL Using Java - What's the Best Practice?

I have an application that is calling a rest service. I need to pass it a URL and right now I'm creating the URL by concatenating a string.
I'm doing it this way:
String urlBase = "http:/api/controller/";
String apiMethod = "buy";
String url = urlBase + apiMethod;
The above is fake obviously, but the point is I'm using simple string concats.
Is this the best practice? I'm relatively new to Java. Should I be building a URL object instead?
Thanks

if you have a base path which needs some additional string to be added to it you have 2 options:
First is, using String.format():
String baseUrl = "http:/api/controller/%s"; // note the %s at the end
String apiMethod = "buy";
String url = String.format(baseUrl, apiMethod);
Or using String.replace():
String baseUrl = "http:/api/controller/{apiMethod}";
String apiMethod = "buy";
String url = baseUrl.replace("\\{apiMethod}", apiMethod);
The nice thing about both answers is, that the string that needs to be inserted, doesn't have to be at the end.

If you are using jersey-client. The following would be the best practice to access the subresources without making the code ugly
Resource: /someApp
Sub-Resource: /someApp/getData
Client client = ClientBuilder.newClient();
WebTarget webTarget = client.target("https://localhost:7777/someApp/").path("getData");
Response response = webTarget.request().header("key", "value").get();

If you are using plain Java, it's better to use dedicated class for URL building which throws exception if provided data are invalid in semantic way.
It has various constructors, you can read about it here.
Example
URL url = new URL(
"http",
"stackoverflow.com",
"/questions/50989746/creating-a-url-using-java-whats-the-best-practive"
);
System.out.println(url);

How can I get the first URL result on Google Video Search? (tag selector)

I want to get the first URL result google video search programmatically using JSoup. I have a problem with Google video encoding or Html tags.(Probably HTML tag: .g>.r>a)
public static String getYoutubeURLByName(String search) throws UnsupportedEncodingException, MalformedURLException, MalformedURLException, MalformedURLException, MalformedURLException, IOException {
String google = "https://www.google.com/videohp?hl=";
String charset = "UTF-8";
String userAgent = "Mozilla/5.0";
Elements links = Jsoup.connect(google + URLEncoder.encode(search, charset)).userAgent(userAgent).get().select(".g>.r>a");
String url = links.get(0).absUrl("href");
url = URLDecoder.decode(url.substring(url.indexOf('=') + 1, url.indexOf('&')), "UTF-8");
return url;
}
I get the following error. I think that Jsoup.connect can't add Element on array list because of something is wrong with encoding url or html tag. (Probably html tag, my poor html :( )
at java.util.ArrayList.rangeCheck(ArrayList.java:657)
at java.util.ArrayList.get(ArrayList.java:433)
at getYoutubeURLByName(.java:100)
Not possible duplicate of: How can you search Google Programmatically Java API
Important Edit:
And string google should be:
https://www.google.com/search?tbm=vid&hl=en-TR&source=hp&biw=&bih=&q=

It seems that your selector is not correct, because elements are not direct childs, try using:
Elements links = Jsoup.connect(google + URLEncoder.encode(search, charset)).userAgent(userAgent).get().select(".g .r a");

Access GoogleCloudStorage using GoogleCloudEndpoints

I'm working on this project in which I'm using a Google-App-Engine backend connected to an Android app via Google-Cloud-Endpoints. For Google-Cloud-Datastore access I'm using Objectify and everything works fine.
Now I decided to add the functionality to upload images to Google-Cloud-Storage but I couldn't find a clear explanation on how to do this using the Google-Cloud-Endpoints setup.
I found the following explanation how to use Google-Cloud-Storage with Google-App-Engine:
https://cloud.google.com/appengine/docs/java/googlecloudstorageclient/app-engine-cloud-storage-sample
but instead of adding it to the Endpoints Api the article writes an additional servlet.
Furthermore I found this example of upload/download for Android:
github.com /thorrism/GoogleCloudExample
Sadly this is using the Google Cloud Storage API for direct access to the Google-Cloud-Storage and you need to add a P12-file to the asset folder, which seems unsecure.
My Google-App-Engine code looks like that:
#Api(
name = "example",
version = "v1",
scopes = { Constants.EMAIL_SCOPE },
clientIds = { Constants.WEB_CLIENT_ID, Constants.ANDROID_CLIENT_ID, Constants.API_EXPLORER_CLIENT_ID },
audiences = {Constants.ANDROID_AUDIENCE},
description = "API for the Example Backend application."
)
public class ExampleApi{
#ApiMethod(name = "doSomething", path = "dosomething", httpMethod = HttpMethod.POST)
public String doSomething(#Named("text") String text){
TestEntity test = new TestEntity(text);
ofy().save().entity(test).now();
return test;
}
After I uploaded it I generated the Endpoints Client Library and imported it into my android project.
Then I'm calling Endpoints from Android like explained here:
https://cloud.google.com/appengine/docs/java/endpoints/calling-from-android#creating_the_service_object
public static com.appspot.******.example.Example buildServiceHandler(Context context, String email) {
GoogleAccountCredential credential = GoogleAccountCredential.usingAudience(
context, AppConstants.AUDIENCE);
credential.setSelectedAccountName(email);
com.appspot.******.example.Example.Builder builder = new com.appspot.******.example.Example.Builder(
AppConstants.HTTP_TRANSPORT,
AppConstants.JSON_FACTORY, null);
builder.setApplicationName("example-server");
return builder.build();
}
sApiServiceHandler = buildServiceHandlerWithAuth(context,email);
And each Api-Method I call like this:
com.appspot.******.example.Example.DoSomething doSomething = sApiServiceHandler.doSomething(someString);
doSomething.execute();
All of this works fine, but only for storing/receiving Datastore Entities. How would I go about uploading/downloading files to Google Cloud Storage using the Google Cloud Endpoints setup?
Is it somehow possible to send a POST with my image data via Endpoints to the UploadServlet using the already build ServiceHandler ?
Is it possible to call a servlet from an Endpoints Method? How am I supposed to send the Post to the Servlet and how would I go about the authentication?
Any help or advice would be greatly appreciated!

There are different ways to do this, but the most recommended way is to use Signed URLs, so that your Android app can upload the file securely to Google Cloud Storage directly, without going through your Endpoints backend. The basic process is:
1) Create an Endpoints method that creates a new signed URL and returns it to the Android client. Signing the URL on the server still requires a P12 key but is stored on App Engine, not on the client, so is secure. Try to use a short expiration for the URL, for example no more than 5 minutes.
2) Have the Android client upload the file directly to the signed URL, as you would doing a normal HTTP PUT to the Cloud Storage XML API to upload a file (resumable uploads with the JSON API are also supported, but not covered here).
Your Endpoints method might look like this:
#ApiMethod(name = "getUploadUrl", path = "getuploadurl", httpMethod = HttpMethod.GET)
public MyApiResponse getUploadUrl(#Named("fileName") String fileName
#Named("contentType" String contentType)
{
String stringToSign
= "PUT\n" + contentType
+ "\n" + EXPIRATION_TIMESTAMP_EPOCH_SECONDS + "\n"
+ YOUR_GCS_BUCKET + "/" + fileName;
// Load P12 key
FileInputStream fileInputStream = new FileInputStream(PATH_TO_P12_KEY);
KeyStore keyStore = KeyStore.getInstance("PKCS12");
keyStore.load(fileInputStream, password);
PrivateKey key = keyStore.getKey(privatekey", YOUR_P12_KEY_PASSWORD);
// Get signature
Signature signer = Signature.getInstance("SHA256withRSA");
signer.initSign(key);
signer.update(stringToSign.getBytes("UTF-8"));
byte[] rawSignature = signer.sign();
String signature = new String(Base64.encodeBase64(rawSignature, false), "UTF-8");
// Construct signed url
String url
= "http://storage.googleapis.com/" + YOUR_GCS_BUCKET + fileName
+ "?GoogleAccessId=" + P12_KEY_SERVICE_ACCOUNT_CLIENT_ID
+ "&Expires=" + EXPIRATION_TIMESTAMP_EPOCH_SECONDS
+ "&Signature=" + URLEncoder.encode(signature, "UTF-8");
// Endpoints doesn't let you return 'String' directly
MyApiResponse response = new MyApiResponse();
response.setString(url);
return response;
}
On the Android side, you might use the method like this:
// Get the upload URL from the API
getUploadUrl = sApiServiceHandler.getUploadUrl(fileName, contentType);
MyApiResponse response = getUploadUrl.execute();
String uploadUrl = response.getString();
// Open connection to GCS
URL url = new URL(uploadUrl);
HttpURLConnection httpConnection = (HttpURLConnection) url.openConnection();
httpConnection.setDoOutput(true);
httpConnection.setRequestMethod("PUT");
httpConnection.setRequestProperty("Content-Type", contentType);
// Write file data
OutputStreamWriter out = new OutputStreamWriter(httpConnection.getOutputStream());
out.write(fileData);
out.flush();
// Get response, check status code etc.
InputStreamReader in = new InputStreamReader(httpConnection.getInputStream());
// ...
(Disclaimer: I'm just typing code freely into a text editor but not actually testing it, but it should be enough to give you a general idea.)

check for validity of URL in java. so as not to crash on 404 error

Essentially, like a bulletproof tank, i want my program to absord 404 errors and keep on rolling, crushing the interwebs and leaving corpses dead and bludied in its wake, or, w/e.
I keep getting this error:
Exception in thread "main" org.jsoup.HttpStatusException: HTTP error fetching URL. Status=404, URL=https://en.wikipedia.org/wiki/Hudson+Township+%28disambiguation%29
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:537)
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:493)
at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:205)
at org.jsoup.helper.HttpConnection.get(HttpConnection.java:194)
at Q.Wikipedia_Disambig_Fetcher.all_possibilities(Wikipedia_Disambig_Fetcher.java:29)
at Q.Wikidata_Q_Reader.getQ(Wikidata_Q_Reader.java:54)
at Q.Wikipedia_Disambig_Fetcher.all_possibilities(Wikipedia_Disambig_Fetcher.java:38)
at Q.Wikidata_Q_Reader.getQ(Wikidata_Q_Reader.java:54)
at Q.Runner.main(Runner.java:35)
But I can't understand why because I am checking to see if I have a valid URL before I navigate to it. What about my checking procedure is incorrect?
I tried to examine the other stack overflow questions on this subject but they're not very authoritative, plus I implemented the many of the solutions from this one and this one, so far nothing has worked.
I'm using the apache commons URL validator, this is the code I've been using most recently:
//get it's normal wiki disambig page
String URL_check = "https://en.wikipedia.org/wiki/" + associated_alias;
UrlValidator urlValidator = new UrlValidator();
if ( urlValidator.isValid( URL_check ) )
{
Document docx = Jsoup.connect( URL_check ).get();
//this can handle the less structured ones.
and
//check the validity of the URL
String URL_czech = "https://www.wikidata.org/wiki/Special:ItemByTitle?site=en&page=" + associated_alias + "&submit=Search";
UrlValidator urlValidator = new UrlValidator();
if ( urlValidator.isValid( URL_czech ) )
{
URL wikidata_page = new URL( URL_czech );
URLConnection wiki_connection = wikidata_page.openConnection();
BufferedReader wiki_data_pagecontent = new BufferedReader(
new InputStreamReader(
wiki_connection.getInputStream()));

The URLConnection throws an error when the status code of the webpage your downloading returns anything other than 2xx (such as 200 or 201 ect...). Instead of passing Jsoup a URL or String to parse your document consider passing it an input stream of data which contains the webpage.
Using the HttpURLConnection class we can try to download the webpage using getInputStream() and place that in a try/catch block and if it fails attempt to download it via getErrorStream().
Consider this bit of code which will download your wiki page even if it returns 404
String URL_czech = "https://en.wikipedia.org/wiki/Hudson+Township+%28disambiguation%29";
URL wikidata_page = new URL(URL_czech);
HttpURLConnection wiki_connection = (HttpURLConnection)wikidata_page.openConnection();
InputStream wikiInputStream = null;
try {
// try to connect and use the input stream
wiki_connection.connect();
wikiInputStream = wiki_connection.getInputStream();
} catch(IOException e) {
// failed, try using the error stream
wikiInputStream = wiki_connection.getErrorStream();
}
// parse the input stream using Jsoup
Jsoup.parse(wikiInputStream, null, wikidata_page.getProtocol()+"://"+wikidata_page.getHost()+"/");

The Status=404 error means there's no page at that location. Just because a URL is valid doesn't mean there's anything there. A validator can't tell you that. The only way you can determine that is by fetching it, and seeing if you get an error, as you're doing.