We Keep Coding

Getting unable to create a new remote session, using Appium, browserstack, proxies and okhttp, Java, Eclipse

I'm very new to automation and I'm trying to write a code in Java to setup remote connection using proxy and okhttp. But, it keeps throwing error:
org.openqa.selenium.SessionNotCreatedException: Unable to create a new remote session. Original error: Too many tunnel connections attempted: 21
Getting the error on the last line of the code snippet, i.e. on the return statement.
Request your help in finding the issue in my code. Below is my code snippet:
Function is being called as below:
General.driver = connectViaProxy(caps);
Implementation of connectViaProxy:
public static AndroidDriver<AndroidElement> connectViaProxy(DesiredCapabilities caps) throws FileNotFoundException, IOException {
String proxyHost = Config.getValue("proxy.host");
int proxyPort = Config.getValueint("proxy.port");
String proxyUserDomain = Config.getValue("proxy.user-domain");
String proxyUser = Config.getValue("proxy.user");
String proxyPassword=EncryptionUtil.passwordDecoder(Config.getValue("proxy.encrypted-password").getBytes());
URL url;
try {
url = new URL("https://"+Config.getValue("BrowserStack.userName")+":"+Config.getValue("BrowserStack.accessKey")+"#hub-cloud.browserstack.com/wd/hub");
} catch (MalformedURLException e) {
throw new RuntimeException(e.getMessage(), e);
}
Authenticator proxyAuthenticator = new Authenticator()
{
#Override
public Request authenticate(Route route, Response response) throws IOException {
String credential = Credentials.basic(proxyUserDomain + "\\" + proxyUser, proxyPassword);
return response.request().newBuilder()
.header("Proxy-Authorization", credential)
.build();
}
};
okhttp3.OkHttpClient.Builder client = new okhttp3.OkHttpClient.Builder()
.connectTimeout(60, TimeUnit.SECONDS)
.writeTimeout(60, TimeUnit.SECONDS)
.readTimeout(60, TimeUnit.SECONDS)
.proxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress(proxyHost, proxyPort)))
.proxyAuthenticator(proxyAuthenticator);
Factory factory = new MyHttpClientFactory(new org.openqa.selenium.remote.internal.OkHttpClient(client.build(), url));
HttpCommandExecutor executor = new AppiumCommandExecutor(MobileCommand.commandRepository, url, factory);
return new AndroidDriver<AndroidElement>(executor, caps);
}

I believe the proxy in your network is not passing the first request made by your test script to establish connection to BrowserStack.
Try to whitelist *.browserstack.com in your proxy for ports 80 and 443.
If that does not work, use local testing and follow the steps from the below documentation so that the request to BrowserStack is by-passed by the proxy.
https://www.browserstack.com/docs/automate/selenium/test-behind-proxy/configure-settings

Java HttpClient 4.x Runnng into somewhat random errors

I am trying to do GET on thousands of URL as quickly as I can. Using the Apache HTTPClient v4.x I can do this but usually end up with around 3-5% of the requests ending up with either host look up failed (<10% of the errors) and the rest either timing out (<10%) or network read errors.
So basically my loop iterates over the URLs and submit the working threads to the Executor service. Below are the code snippets for the important pieces:
Executor
--------
public static ExecutorService pool = Executors.newFixedThreadPool(400);
AppConfig.monitors.forEach((key, monitor) -> {
results.add(AppConfig.pool.submit(new WebRequest(monitor)));
}
});
Notes:
I have tried all ranges between 50-1000 for thread count.
I submit it to a Callable thread which returns a future and I iterater over the results in a subsequent loop.
CLIENT Code
-----------
cm = new PoolingHttpClientConnectionManager();
cm.setDefaultMaxPerRoute(2);
cm.setMaxTotal(1000);
RequestConfig rc = RequestConfig.custom()
.setSocketTimeout(4000)
.setConnectTimeout(4000)
.setCookieSpec(CookieSpecs.IGNORE_COOKIES)
.setMaxRedirects(1)
.setRedirectsEnabled(true)
.setCircularRedirectsAllowed(false)
.build();
sslContext = SSLContextBuilder
.create()
.loadTrustMaterial(TrustSelfSignedStrategy.INSTANCE)
.build();
sslcsf = new SSLConnectionSocketFactory(sslContext, new NoopHostnameVerifier());
client = HttpClients.custom()
.setConnectionManager(cm)
.setDefaultHeaders(headers)
.setDefaultRequestConfig(rc)
.setSSLSocketFactory(sslcsf)
.disableAutomaticRetries()
.setRedirectStrategy(DefaultRedirectStrategy.INSTANCE)
Notes:
All URLs are final and there are neither regular or circular redirects.
Also note that I have run these URLs through the HTTPClient one at a time and they work without issue. So in general timeouts should not occur.
All of the domain are accessible.
REQUEST Code
------------
public class WebRequest implements Callable<Monitor> {
#Override
public Monitor call() throws Exception {
HttpRequestBase request;
HttpContext context = HttpClientContext.create();
request = new HttpGet(monitor.getUrl());
try (CloseableHttpResponse response = WebClient.client.execute(request, context);) {
request.releaseConnection();
}
}
}
Please let me know if you need additional information.

Springboot SLL API consume

I am trying to consume an API from a 3rd party server. The 3rd party sent me an SSL certificated named certificate.p12 which is the cert file which I use to do the handshake. I have created a custom RestTemplate with SSL as follows:
#Configuration
public class CustomRestTemplate {
private static final String PASSWORD = "fake_password";
private static final String RESOURCE_PATH = "keystore/certificate.p12";
private static final String KEY_TYPE = "PKCS12";
#Bean
public RestTemplate restTemplate(RestTemplateBuilder builder) throws Exception {
char[] password = PASSWORD.toCharArray();
SSLContext sslContext = SSLContextBuilder
.create()
// .loadKeyMaterial(keyStore(RESOURCE_PATH, password), password)
.loadKeyMaterial(keyStore(getClass().getClassLoader().getResource(RESOURCE_PATH).getFile(), password), password)
.loadTrustMaterial(null, new TrustSelfSignedStrategy())
.build();
HttpClient client = HttpClients
.custom()
.setSSLContext(sslContext)
.build();
return builder
.requestFactory(() -> new HttpComponentsClientHttpRequestFactory(client))
.build();
}
private KeyStore keyStore(String file, char[] password) throws Exception {
KeyStore keyStore = KeyStore.getInstance(KEY_TYPE);
File key = ResourceUtils.getFile(file);
try (InputStream in = new FileInputStream(key)) {
keyStore.load(in, password);
}
return keyStore;
}
}
I then call the endpoint using the following code:
#Component
#Service
public class TransactionService implements TransactionInterface {
#Autowired
private CustomRestTemplate restTemplate = new CustomRestTemplate();
private static final String BASE_URL = "https://41.x.x.x:xxxx/";
#Override
public List<Transaction> getUnsentTransactions(int connectionId) throws Exception {
HttpEntity<?> httpEntity = new HttpEntity<>(null, new HttpHeaders());
ResponseEntity<Transaction[]> resp = restTemplate
.restTemplate(new RestTemplateBuilder())
.exchange(BASE_URL + "path/end_point/" + connectionId, HttpMethod.GET, httpEntity, Transaction[].class);
return Arrays.asList(resp.getBody());
}
}
I get an the following stacktrace when trying to consume the api:
org.springframework.web.client.ResourceAccessException: I/O error on GET request for \"https://41.x.x.x:xxxx/path/endpoint/parameters\": Certificate for <41.x.x.x> doesn't match any of the subject alternative names: [some_name_here, some_name_here];
I do not have much experience with TLS or SSL certificates. I am really stuck at the moment and hoping I can get some help here. The 3rd party provided me with a testing site where I can test the endpoints and after importing the certificate.p12 file into my browser I can reach the endpoints using their testing site but my Springboot application still does not reach the endpoint.
Do I need to copy the cert into a specific folder? This does not seem like the case because I get a FileNotFoundException if I change the path or filename and I get a password incorrect error if I enter the wrong password for the certificate.p12 file. I tried using Postman to test it but Postman returns the same stacktrace as my web application.
Looking at the information above, am I missing something? Is the keystore not being created during runtime? Do I need to bind the certificate to the JVM or my outgoing request?
Any help would be appreciated.
Thanks

It looks like you are trying to connect to a server which doesn't have a valid name in the certificate. For example, if you are connecting to "stackoverflow.com", the certificate needs that domain in the "subject" or the "subject alternative names" field.
Even a testing site should have a valid certificate, but if that's not possible (as it's a third party site and you can't change it yourself), you can disable the verification using this question
Of course, this should only be done for testing.

How to use connection pooling with HttpsUrlConnectionMessageSender

I am trying to make a Soap request using Spring WS. The request needs to send across client certificate to the server. I have figured out the configuration that makes the correct request to the server. Code presented below
#Bean
public HttpsUrlConnectionMessageSender messageSender() throws Exception {
HttpsUrlConnectionMessageSender httpsUrlConnectionMessageSender = new HttpsUrlConnectionMessageSender();
httpsUrlConnectionMessageSender.setTrustManagers(<TrustManager>);
httpsUrlConnectionMessageSender.setKeyManagers(<KeyManager>);
httpsUrlConnectionMessageSender.setHostnameVerifier(new HostnameVerifier() {
public boolean verify(String hostname, SSLSession session) {
return true;
}
});
return httpsUrlConnectionMessageSender;
}
#Bean
#Scope
public WebServiceTemplate webServiceTemplate() throws Exception {
WebServiceTemplate webServiceTemplate = new WebServiceTemplate(<messageFactory>);
webServiceTemplate.setMarshaller(<jaxb2Marshaller>);
webServiceTemplate.setUnmarshaller(<jaxb2Marshaller>);
webServiceTemplate.setDefaultUri(<URL>);
webServiceTemplate.setMessageSender(());
return webServiceTemplate;
}
Now I was wondering how does HttpsUrlConnectionMessageSender manage connections? Does it create new connection for every request or it has internal connection pooling. How do we use external connection pooling here?

The client has default connection pool of 20 and default connection pool per request 2.
But you can overwrite it like this:
public HttpComponentsMessageSender httpComponentsMessageSender(Integer readTimeout, Integer connectionTimeOut,
Integer throttlingLimit) {
HttpComponentsMessageSender sender = new HttpComponentsMessageSender();
sender.setReadTimeout(readTimeout);
sender.setConnectionTimeout(connectionTimeOut);
PoolingHttpClientConnectionManager connManager = new PoolingHttpClientConnectionManager();
CloseableHttpClient poolingHttpSoapClient = HttpClients.custom().setConnectionManager(connManager)
.addInterceptorFirst(new RemoveSoapHeadersInterceptor()).setMaxConnPerRoute(throttlingLimit)
.setMaxConnTotal(throttlingLimit + 1).build();
sender.setHttpClient(poolingHttpSoapClient);
return sender;
}

OkHTTP (v3.0.0-RC1) Post Request with Parameters

UPDATE
I'm using okHttp library, version 3.0.0 RC-1. I have a url and I have to send a post request with parameters. For example: https://myurl.com/login?username=123456&password=12345
I have code like this:
public String okHttpRequest(){
try {
final TrustManager[] trustAllCerts = new TrustManager[]{new X509TrustManager() {
#Override
public void checkClientTrusted(X509Certificate[] chain, String authType) throws CertificateException {
}
#Override
public void checkServerTrusted(X509Certificate[] chain, String authType) throws CertificateException {
}
#Override
public X509Certificate[] getAcceptedIssuers() {
return null;
}
}
};
final SSLContext sslContext = SSLContext.getInstance("SSL");
sslContext.init(null,trustAllCerts, new java.security.SecureRandom());
final javax.net.ssl.SSLSocketFactory sslSocketFactory = sslContext.getSocketFactory();
OkHttpClient.Builder builder = new OkHttpClient.Builder();
builder.sslSocketFactory(sslSocketFactory);
builder.hostnameVerifier(new HostnameVerifier() {
#Override
public boolean verify(String hostname, SSLSession session) {
return true;
}
}).build();
final OkHttpClient client = new OkHttpClient();
HttpUrl url = HttpUrl.parse("https://myUrl.com/login").newBuilder()
.addQueryParameter("username", "123456")
.addQueryParameter("password", "123456")
.build();
Request request = new Request.Builder()
.url(url)
.build();
client.newCall(request).enqueue(new Callback() {
#Override
public void onFailure(Request request, IOException e) {
Log.d("Request:", "Is NOT sent " + request.toString() + " METHOD: " + request.method());
e.printStackTrace();
}
#Override
public void onResponse(Response response) throws IOException {
Log.d("Request:", "Is sent " + response.toString());
}
});
}catch(Exception e){
e.printStackTrace();
}
return "okHttp is Working!!! ";
}
Whenever I try, It fails, so onFailure method is executing. What is the problem? Am I adding request params incorrectly? Please help...

Yes, you are adding the query parameters incorrectly. Here's how it should be done:
final OkHttpClient client = new OkHttpClient();
HttpUrl url = HttpUrl.parse("https://myUrl.com/login").newBuilder()
.addQueryParameter("password", "123456")
.addQueryParameter("username", "123456")
.build();
Request request = new Request.Builder()
.url(url)
.build();
(...)
The problem is that you are submitting your data in the body of the request, as if it were a HTML form submit, and not as a query parameter, as you intended. Using HttpUrl allows you to add query parameters to your URL.
Worth noting that you can also simply do this:
Request request = new Request.Builder()
.url("https://myurl.com/login?username=123456&password=12345")
.build();
So:
Use HttpUrl and it's builder to create your URL and add parameters.
Use FormBody to create the content of your request (as if it were a html form you're submitting), if you need it.
Also, make sure you have internet permission in your app, make sure you have an active internet connection, etc, but it seems you already have.
Note: this is for OkHttp 2.x (since that's what I use), but it should be the same or similar for 3.x.
Let me know if it works for you.