I'm very new to automation and I'm trying to write a code in Java to setup remote connection using proxy and okhttp. But, it keeps throwing error:
org.openqa.selenium.SessionNotCreatedException: Unable to create a new remote session. Original error: Too many tunnel connections attempted: 21
Getting the error on the last line of the code snippet, i.e. on the return statement.
Request your help in finding the issue in my code. Below is my code snippet:
Function is being called as below:
General.driver = connectViaProxy(caps);
Implementation of connectViaProxy:
public static AndroidDriver<AndroidElement> connectViaProxy(DesiredCapabilities caps) throws FileNotFoundException, IOException {
String proxyHost = Config.getValue("proxy.host");
int proxyPort = Config.getValueint("proxy.port");
String proxyUserDomain = Config.getValue("proxy.user-domain");
String proxyUser = Config.getValue("proxy.user");
String proxyPassword=EncryptionUtil.passwordDecoder(Config.getValue("proxy.encrypted-password").getBytes());
URL url;
try {
url = new URL("https://"+Config.getValue("BrowserStack.userName")+":"+Config.getValue("BrowserStack.accessKey")+"#hub-cloud.browserstack.com/wd/hub");
} catch (MalformedURLException e) {
throw new RuntimeException(e.getMessage(), e);
}
Authenticator proxyAuthenticator = new Authenticator()
{
#Override
public Request authenticate(Route route, Response response) throws IOException {
String credential = Credentials.basic(proxyUserDomain + "\\" + proxyUser, proxyPassword);
return response.request().newBuilder()
.header("Proxy-Authorization", credential)
.build();
}
};
okhttp3.OkHttpClient.Builder client = new okhttp3.OkHttpClient.Builder()
.connectTimeout(60, TimeUnit.SECONDS)
.writeTimeout(60, TimeUnit.SECONDS)
.readTimeout(60, TimeUnit.SECONDS)
.proxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress(proxyHost, proxyPort)))
.proxyAuthenticator(proxyAuthenticator);
Factory factory = new MyHttpClientFactory(new org.openqa.selenium.remote.internal.OkHttpClient(client.build(), url));
HttpCommandExecutor executor = new AppiumCommandExecutor(MobileCommand.commandRepository, url, factory);
return new AndroidDriver<AndroidElement>(executor, caps);
}
I believe the proxy in your network is not passing the first request made by your test script to establish connection to BrowserStack.
Try to whitelist *.browserstack.com in your proxy for ports 80 and 443.
If that does not work, use local testing and follow the steps from the below documentation so that the request to BrowserStack is by-passed by the proxy.
https://www.browserstack.com/docs/automate/selenium/test-behind-proxy/configure-settings
I am using okhttp3 and trying to see how do I pass userId & pswd to authenticate with proxy server that accepts only HTTPS protocol. I already saw exmaple over SO & on other sites(link below) but they don't have HTTPS protocol.
https://botproxy.net/docs/how-to/okhttpclient-proxy-authentication-how-to/
Can anyone please tell me how to use HTTPS protocol to call proxy server?
It is not officially supported but there is a workaround.
https://github.com/square/okhttp/issues/6561
Authenticator proxyAuthenticator = new Authenticator() {
#Override public Request authenticate(Route route, Response response) throws IOException {
String credential = Credentials.basic(username, password);
return response.request().newBuilder().header("Proxy-Authorization", credential).build();
}
};
OkHttpClient client = new OkHttpClient.Builder()
.proxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress(proxyHost, proxyPort)))
.proxyAuthenticator(proxyAuthenticator);
.socketFactory(new DelegatingSocketFactory(SSLSocketFactory.getDefault()))
.build();
I'm trying to use JDK 11 HttpClient to make requests through a corporate proxy which requires authentication by login and password. According to JDK's intro, I'm building an instance of client by means of:
HttpClient httpClient = HttpClient.newBuilder()
.version(HTTP_1_1)
.proxy(ProxySelector.of(new InetSocketAddress("proxy.mycompany.com", 3128)))
.authenticator(authenticator)
.build();
, where authenticator is:
Authenticator authenticator = new Authenticator() {
#Override
protected PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication("login", "password".toCharArray());
}
};
And then I execute the request itself:
HttpRequest outRequest = HttpRequest.newBuilder()
.version(HTTP_1_1)
.GET()
.uri(URI.create("https://httpwg.org/asset/http.svg")) // no matter which URI to request
.build();
HttpResponse<String> inResponse = httpClient.send(outRequest, BodyHandlers.ofString());
But instead of valid response from the target server (https://httpwg.org) I receive HTTP 407 (Proxy Authentication Required), i.e. HttpClient does not use the provided authenticator.
I've tried various solutions mentioned here and here but none of them helped.
What is the correct way to make it work?
You have to set the "Proxy-Authorization" header on the request.
HttpClient httpClient = HttpClient.newBuilder()
.version(HTTP_1_1)
.proxy(ProxySelector.of(new InetSocketAddress("proxy.mycompany.com", 3128)))
.build();
String encoded = new String(Base64.getEncoder().encode("login:password".getBytes()));
HttpRequest outRequest = HttpRequest.newBuilder()
.version(HTTP_1_1)
.GET()
.uri(URI.create("https://httpwg.org/asset/http.svg")) // no matter which URI to request
.setHeader("Proxy-Authorization", "Basic " + encoded)
.build();
By default, basic authentication with the proxy is disabled when tunneling through an authenticating proxy since java 8u111.
You can re-enable it by specifying -Djdk.http.auth.tunneling.disabledSchemes="" on the java command line.
See the jdk 8u111 release notes
Use
System.setProperty("jdk.http.auth.proxying.disabledSchemes", "");
System.setProperty("jdk.http.auth.tunneling.disabledSchemes", "");
Construct your http client with proxy selector and authenticator
HttpClient client = HttpClient.newBuilder()
.authenticator(yourCustomAuthenticator)
.proxy(yourCustomProxySelector)
.build();
In your Authenticator override
#Override
protected PasswordAuthentication getPasswordAuthentication() {
if (getRequestorType().equals(RequestorType.PROXY)) {
return getPasswordAuthentication(getRequestingHost());
} else {
return null;
}
}
protected PasswordAuthentication getPasswordAuthentication(String proxyHost) {
// your logic to find username and password for proxyHost
return new PasswordAuthentication(proxyUsername, proxyPassword.toCharArray());
// or return null
}
I am using the OkHttp library to make requests from my application to facebook api, however I need to work on a proxy network, instantiating OkHttpClient() and calling OkHttpClient.newCall(request).execute() I get a timeout message because my proxy stop the request.
After a little research I found the following solution:
int proxyPort = 8080;
String proxyHost = "proxyHost";
final String username = "username";
final String password = "password";
Authenticator proxyAuthenticator = new Authenticator() {
#Override public Request authenticate(Route route, Response response) throws IOException {
String credential = Credentials.basic(username, password);
return response.request().newBuilder()
.header("Proxy-Authorization", credential)
.build();
}
};
OkHttpClient client = new OkHttpClient.Builder()
.connectTimeout(60, TimeUnit.SECONDS)
.writeTimeout(60, TimeUnit.SECONDS)
.readTimeout(60, TimeUnit.SECONDS)
.proxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress(proxyHost, proxyPort)))
.proxyAuthenticator(proxyAuthenticator)
.build();
This works great, however I would not want to leave the proxy information in the code or in the application.
Is there any way to configure the proxy as environment variable or in some external file where OkHttp would be able to complete the requests?
I would use system environment variables for storing this sensitive configuration. If you do not have property file, system variable would be good option.
You can update you authenticate method to this:
Authenticator proxyAuthenticator = new Authenticator() {
#Override public Request authenticate(Route route, Response response) throws
IOException {
String username = System.getenv("PROXY_USERNAME");
String password = System.getenv("PROXY_PASSWORD");
if (username == null || username.isEmpty() || password == null || password.isEmpty() )
throw new IllegalStateException("Proxy information is not defined in system variables");
String credential = Credentials.basic(username, password);
return response.request().newBuilder()
.header("Proxy-Authorization", credential)
.build();
}
};
and remove
final String username = "username";
final String password = "password";
class fields.
Now when you run your application you can define the variables on the computer itself or better to pass them as parameters to you java application. For instance:
java -jar -DPROXY_USERNAME=userName -DPROXY_PASSWORD=password yourJar.jar
Is there an easier way to setup the http client for preemptive basic authentication than what described here?
In previous version (3.x) it used to be a simple method call (eg, httpClient.getParams().setAuthenticationPreemptive(true)).
The main thing I want to avoid is adding the BasicHttpContext to each method I execute.
If you are looking to force HttpClient 4 to authenticate with a single request, the following will work:
String username = ...
String password = ...
UsernamePasswordCredentials creds = new UsernamePasswordCredentials(username, password);
HttpRequest request = ...
request.addHeader(new BasicScheme().authenticate(creds, request));
It's difficult to do this without passing a context through every time, but you can probably do it by using a request interceptor. Here is some code that we use (found from their JIRA, iirc):
// Pre-emptive authentication to speed things up
BasicHttpContext localContext = new BasicHttpContext();
BasicScheme basicAuth = new BasicScheme();
localContext.setAttribute("preemptive-auth", basicAuth);
httpClient.addRequestInterceptor(new PreemptiveAuthInterceptor(), 0);
(...)
static class PreemptiveAuthInterceptor implements HttpRequestInterceptor {
public void process(final HttpRequest request, final HttpContext context) throws HttpException, IOException {
AuthState authState = (AuthState) context.getAttribute(ClientContext.TARGET_AUTH_STATE);
// If no auth scheme avaialble yet, try to initialize it
// preemptively
if (authState.getAuthScheme() == null) {
AuthScheme authScheme = (AuthScheme) context.getAttribute("preemptive-auth");
CredentialsProvider credsProvider = (CredentialsProvider) context.getAttribute(ClientContext.CREDS_PROVIDER);
HttpHost targetHost = (HttpHost) context.getAttribute(ExecutionContext.HTTP_TARGET_HOST);
if (authScheme != null) {
Credentials creds = credsProvider.getCredentials(new AuthScope(targetHost.getHostName(), targetHost.getPort()));
if (creds == null) {
throw new HttpException("No credentials for preemptive authentication");
}
authState.setAuthScheme(authScheme);
authState.setCredentials(creds);
}
}
}
}
This is the same solution as Mat's Mannion's, but you don't have to put localContext to each request. It's simpler, but it adds authentication to ALL requests. Useful, if you don't have control over individual requests, as in my case when using Apache Solr, which uses HttpClient internally.
import org.apache.http.HttpException;
import org.apache.http.HttpHost;
import org.apache.http.HttpRequest;
import org.apache.http.HttpRequestInterceptor;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.AuthState;
import org.apache.http.auth.Credentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.impl.auth.BasicScheme;
import org.apache.http.protocol.HttpContext;
import org.apache.http.protocol.HttpCoreContext;
httpClient.addRequestInterceptor(new PreemptiveAuthInterceptor(), 0);
(...)
static class PreemptiveAuthInterceptor implements HttpRequestInterceptor {
public void process(final HttpRequest request, final HttpContext context) throws HttpException, IOException {
AuthState authState = (AuthState) context.getAttribute(HttpClientContext.TARGET_AUTH_STATE);
// If no auth scheme available yet, try to initialize it
// preemptively
if (authState.getAuthScheme() == null) {
CredentialsProvider credsProvider = (CredentialsProvider) context.getAttribute(HttpClientContext.CREDS_PROVIDER);
HttpHost targetHost = (HttpHost) context.getAttribute(HttpCoreContext.HTTP_TARGET_HOST);
Credentials creds = credsProvider.getCredentials(new AuthScope(targetHost.getHostName(), targetHost.getPort()));
if (creds == null) {
throw new HttpException("No credentials for preemptive authentication");
}
authState.update(new BasicScheme(), creds);
}
}
}
Of course, you have to set the credentials provider:
httpClient.getCredentialsProvider().setCredentials(
new AuthScope(url.getHost(), url.getPort()),
new UsernamePasswordCredentials(username, password))
The AuthScope must not contain realm, as it is not known in advance.
A lot of the answers above use deprecated code. I am using Apache SOLRJ version 5.0.0.
My code consists of
private HttpSolrClient solrClient;
private void initialiseSOLRClient() {
URL solrURL = null;
try {
solrURL = new URL(urlString);
} catch (MalformedURLException e) {
LOG.error("Cannot parse the SOLR URL!!" + urlString);
throw new SystemException("Cannot parse the SOLR URL!! " + urlString, e);
}
String host = solrURL.getHost();
int port = solrURL.getPort();
AuthScope authScope = new AuthScope(host, port);
BasicTextEncryptor textEncryptor = new BasicTextEncryptor();
textEncryptor.setPassword("red bananas in the spring");
String decryptPass = textEncryptor.decrypt(pass);
UsernamePasswordCredentials creds = new UsernamePasswordCredentials(userName, decryptPass);
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(
authScope,
creds);
HttpClientBuilder builder = HttpClientBuilder.create();
builder.addInterceptorFirst(new PreemptiveAuthInterceptor());
builder.setDefaultCredentialsProvider(credsProvider);
CloseableHttpClient httpClient = builder.build();
solrClient = new HttpSolrClient(urlString, httpClient);
}
The PreemptiveAuthInterceptor is now as follows:-
static class PreemptiveAuthInterceptor implements HttpRequestInterceptor {
public void process(final HttpRequest request, final HttpContext context) throws HttpException, IOException {
AuthState authState = (AuthState) context.getAttribute(HttpClientContext.TARGET_AUTH_STATE);
// If no auth scheme available yet, try to initialize it
// preemptively
if (authState.getAuthScheme() == null) {
CredentialsProvider credsProvider = (CredentialsProvider)
context.getAttribute(HttpClientContext.CREDS_PROVIDER);
HttpHost targetHost = (HttpHost) context.getAttribute(HttpCoreContext.HTTP_TARGET_HOST);
AuthScope authScope = new AuthScope(targetHost.getHostName(), targetHost.getPort());
Credentials creds = credsProvider.getCredentials(authScope);
if(creds == null){
}
authState.update(new BasicScheme(), creds);
}
}
}
A little late to the party but I came accross the thread trying to solve this for proxy pre-authorization of a post request. To add to Adam's response, I found the following worked for me:
HttpPost httppost = new HttpPost(url);
UsernamePasswordCredentials creds = new UsernamePasswordCredentials(username, password);
Header bs = new BasicScheme().authenticate(creds, httppost);
httppost.addHeader("Proxy-Authorization", bs.getValue());
Thought that might be helpful for anyone else who runs into this.
I think the best way may be to just do it manually. I added the following function
Classic Java:
import javax.xml.bind.DatatypeConverter;
...
private static void addAuthHeader(HttpRequestBase http, String username, String password) throws UnsupportedEncodingException {
String encoded = DatatypeConverter.printBase64Binary((username + ":" + password).getBytes("UTF-8"));
http.addHeader("AUTHORIZATION", "Basic " + encoded);
}
HTTPRequestBase can be an instance of HttpGet or HttpPost
Android:
import android.util.Base64;
...
private static void addAuthHeader(HttpRequestBase http, String username, String password) throws UnsupportedEncodingException {
String encoded = Base64.encodeToString((username + ":" + password).getBytes("UTF-8"), Base64.NO_WRAP);
http.addHeader("AUTHORIZATION", "Basic " + encoded);
}
I'm using this code, based on my reading of the HTTPClient 4.5 docs:
HttpClientContext ctx = HttpClientContext.create()
ctx.setCredentialsProvider(new BasicCredentialsProvider())
ctx.setAuthCache(new BasicAuthCache())
UsernamePasswordCredentials creds = new UsernamePasswordCredentials(user, pass)
AuthScope authScope = new AuthScope(host, port)
ctx.getCredentialsProvider.setCredentials(authScope, credentials)
// This part makes authentication preemptive:
HttpHost targetHost = new HttpHost(host, port, scheme)
ctx.getAuthCache.put(targetHost, new BasicScheme())
...and make sure you always pass that context to HTTPClient.execute().
I don't quite get your closing comment. It's the HttpClient that has all of that machinery for doing preemptive auth, and you only have to do that once (when you construct and configure your HttpClient). Once you've done that, you construct your method instances the same way as always. You don't "add the BasicHttpContext" to the method.
Your best bet, I'd think, is to have your own object that sets up all of the junk required for preemptive auth, and has a simple method or methods for executing requests on given HTTPMethod objects.
in android,Mat Mannion's answer can't resolve https,still send two requests,you can do like below,the trick is append authHeader with user-agent：
public static DefaultHttpClient createProxyHttpClient() {
try {
final DefaultHttpClient client = createPlaintHttpClient();
client.setRoutePlanner(new HttpRoutePlanner() {
#Override
public HttpRoute determineRoute(HttpHost target, HttpRequest request, HttpContext context) throws HttpException {
boolean isSecure = "https".equalsIgnoreCase(target.getSchemeName());
if (needProxy) {
Header header = isSecure ? ProxyUtils.createHttpsAuthHeader() : ProxyUtils.createAuthHeader();
if (isSecure) {
client.getParams().setParameter(CoreProtocolPNames.USER_AGENT, com.netease.cloudmusic.utils.HttpRequest.USER_AGENT + "\r\n" + header.getName() + ":" + header.getValue());
} else {
client.getParams().setParameter(CoreProtocolPNames.USER_AGENT, com.netease.cloudmusic.utils.HttpRequest.USER_AGENT);
if (request instanceof RequestWrapper) {
request = ((RequestWrapper) request).getOriginal();
}
request.setHeader(header);
}
String host = isSecure ? ProxyUtils.SECURE_HOST : ProxyUtils.HOST;
int port = isSecure ? ProxyUtils.SECURE_PORT : ProxyUtils.PORT;
return new HttpRoute(target, null, new HttpHost(host, port), isSecure);
} else {
client.getParams().setParameter(CoreProtocolPNames.USER_AGENT, com.netease.cloudmusic.utils.HttpRequest.USER_AGENT);
return new HttpRoute(target, null, isSecure);
}
}
});
return client;
} catch (Exception e) {
e.printStackTrace();
return new DefaultHttpClient();
}
}
public static DefaultHttpClient createPlaintHttpClient() {
try {
KeyStore trustStore = KeyStore.getInstance(KeyStore.getDefaultType());
trustStore.load(null, null);
PlainSSLSocketFactory socketFactory = new PlainSSLSocketFactory(trustStore);
socketFactory.setHostnameVerifier(SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER);
BasicHttpParams params = new BasicHttpParams();
HttpConnectionParams.setConnectionTimeout(params, 30000);
HttpConnectionParams.setSoTimeout(params, 30000);
HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1);
HttpProtocolParams.setContentCharset(params, HTTP.UTF_8);
SchemeRegistry registry = new SchemeRegistry();
registry.register(new Scheme("http", PlainSocketFactory.getSocketFactory(), 80));
registry.register(new Scheme("https", socketFactory, 443));
ThreadSafeClientConnManager ccm = new ThreadSafeClientConnManager(params, registry);
HttpClientParams.setCookiePolicy(params, CookiePolicy.BROWSER_COMPATIBILITY);
final DefaultHttpClient client = new DefaultHttpClient(ccm, params);
client.setRoutePlanner(new HttpRoutePlanner() {
#Override
public HttpRoute determineRoute(HttpHost target, HttpRequest arg1, HttpContext arg2) throws HttpException {
client.getParams().setParameter(CoreProtocolPNames.USER_AGENT, com.netease.cloudmusic.utils.HttpRequest.USER_AGENT);
return new HttpRoute(target, null, "https".equalsIgnoreCase(target.getSchemeName()));
}
});
return client;
} catch (Exception e) {
e.printStackTrace();
return new DefaultHttpClient();
}
}
SolrConfig:
#Configuration
public class SolrConfig {
#Value("${solr.http.url}")
private String solrUrl;
#Value("${solr.http.username}")
private String solrUser;
#Value("${solr.http.password}")
private String solrPassword;
#Value("${solr.http.pool.maxTotal}")
private int poolMaxTotal;
#Value("${solr.http.pool.maxPerRoute}")
private int pollMaxPerRoute;
#Bean
public SolrClient solrClient() {
PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
connectionManager.setMaxTotal(poolMaxTotal);
connectionManager.setDefaultMaxPerRoute(pollMaxPerRoute);
CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(solrUser, solrPassword));
CloseableHttpClient httpClient = HttpClientBuilder.create()
.addInterceptorFirst(new PreemptiveAuthInterceptor())
.setConnectionManager(connectionManager)
.setDefaultCredentialsProvider(credentialsProvider)
.build();
return new HttpSolrClient.Builder(solrUrl).withHttpClient(httpClient).build();
}
}
PreemptiveAuthInterceptor:
public class PreemptiveAuthInterceptor implements HttpRequestInterceptor {
public void process(final HttpRequest request, final HttpContext context)
throws HttpException {
AuthState authState = (AuthState) context
.getAttribute(HttpClientContext.TARGET_AUTH_STATE);
// If no auth scheme available yet, try to initialize it
// preemptively
if (authState.getAuthScheme() == null) {
CredentialsProvider credentialsProvider = (CredentialsProvider) context
.getAttribute(HttpClientContext.CREDS_PROVIDER);
HttpHost targetHost = (HttpHost) context
.getAttribute(HttpCoreContext.HTTP_TARGET_HOST);
Credentials credentials = credentialsProvider.getCredentials(new AuthScope(
targetHost.getHostName(), targetHost.getPort()));
if (credentials == null) {
throw new HttpException(
"No credentials for preemptive authentication");
}
authState.update(new BasicScheme(), credentials);
}
}
}