We Keep Coding

javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative names matching IP address

I have a java application calling external api which is hosted on address like https://10.20.30.40:1234/test/myurl
this have a domain base certifcate with CN like *.myappdomain.au
We have done registration on our linux server of the certificate.
I have even tried loading the certificate with following code but it is of no use and we are getting same error
private static SSLSocketFactory createSSLSocketFactory(String certificatePath) throws IOException, CertificateException, KeyStoreException, NoSuchAlgorithmException, KeyManagementException {
File crtFile = new File(certificatePath);
Certificate certificate = CertificateFactory.getInstance("X.509").generateCertificate(new FileInputStream(crtFile));
KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType());
keyStore.load(null, null);
keyStore.setCertificateEntry("server", certificate);
TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
trustManagerFactory.init(keyStore);
SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(null, trustManagerFactory.getTrustManagers(), null);
return sslContext.getSocketFactory();
}
One thing i tried which work is adding entry in host like
10.20.30.40 myappdomain.au
and then using url like
https://myappdomain.au:1234/test/myurl
then application works
Any idea what more i need to do

Well, if the API is hosted on an IP address, the SSL certificate has to define that IP address as Subject Alternative Name. This however won't work for services like Let's Encrypt.
I guess the main cause of the issue is that you're trying to access the API by its IP address rather than its FQDN. Changing the URL of the API to the appropriate DNS name for the IP address in your source code should yield in everything working, as long as the DNS name resolves to something related to the domain the wildcard certificate was issued for (e.g. api.myappdomain.au).

try to run this code before connect:
public static void trustAllCerts() {
TrustManager[] trustAllCerts = new TrustManager[]{
new X509TrustManager() {
public java.security.cert.X509Certificate[] getAcceptedIssuers() {
return null;
}
public void checkClientTrusted(
java.security.cert.X509Certificate[] certs, String authType) {
}
public void checkServerTrusted(
java.security.cert.X509Certificate[] certs, String authType) {
}
}
};
try {
SSLContext sc = SSLContext.getInstance("SSL");
sc.init(null, trustAllCerts, new java.security.SecureRandom());
HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
} catch (Exception e) {
e.printStackTrace();
}
}

Impossible to start multiple HTTPS connections from Spring application

I have a Spring Boot application that tries to open a javax.net.ssl.HttpsURLConnection to a server but the response received is: java.io.IOException: Server returned HTTP response code: 403 for URL: https://serverIP:8443/path
When the keyStore, trustStore and their passwords are set as system properties the request works correctly and the expected JSON response is received:
System.setProperty("javax.net.ssl.keyStore", "src/main/resources/myKeyStore.p12");
System.setProperty("javax.net.ssl.trustStore", "src/main/resources/myTrustStore.truststore");
System.setProperty("javax.net.ssl.keyStorePassword", "myPassword");
System.setProperty("javax.net.ssl.trustStorePassword", "myPassword");
But the 403 response code is received when trying to set the information in SSLContext, instead of setting the system properties, by using this method that returns an SSLContext object:
public static SSLContext getSslContext(String trustStoreFile, String keystoreFile, String password)
throws GeneralSecurityException, IOException {
final KeyStore keystore = KeyStore.getInstance("pkcs12"); // also tried with JKS
try (final InputStream inKeystore = new FileInputStream(keystoreFile)) {
keystore.load(inKeystore, password.toCharArray());
}
try (final InputStream inTruststore = new FileInputStream(trustStoreFile)) {
keystore.load(inTruststore, password.toCharArray());
}
final KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance("PKIX"); // also tried with .getDefaultAlgorithm()
keyManagerFactory.init(keystore, password.toCharArray());
final TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
trustManagerFactory.init(keystore);
X509TrustManager x509Tm = null;
for (final TrustManager trustManager : trustManagerFactory.getTrustManagers()) {
if (trustManager instanceof X509TrustManager) {
x509Tm = (X509TrustManager) trustManager;
break;
}
}
final X509TrustManager finalTm = x509Tm;
final X509ExtendedTrustManager customTm = new X509ExtendedTrustManager() {
#Override
public java.security.cert.X509Certificate[] getAcceptedIssuers() {
return finalTm.getAcceptedIssuers();
}
#Override
public void checkClientTrusted(java.security.cert.X509Certificate[] certs, String authType) {
}
#Override
public void checkServerTrusted(java.security.cert.X509Certificate[] certs, String authType) {
}
#Override
public void checkClientTrusted(java.security.cert.X509Certificate[] xcs, String string, Socket socket) throws CertificateException {
}
#Override
public void checkServerTrusted(java.security.cert.X509Certificate[] xcs, String string, Socket socket) throws CertificateException {
}
#Override
public void checkClientTrusted(java.security.cert.X509Certificate[] xcs, String string, SSLEngine ssle) throws CertificateException {
}
#Override
public void checkServerTrusted(java.security.cert.X509Certificate[] xcs, String string, SSLEngine ssle) throws CertificateException {
}
};
final SSLContext sslContext = SSLContext.getInstance("TLS"); // also tried with SSL
sslContext.init(
keyManagerFactory.getKeyManagers(),
new TrustManager[]{customTm},
new SecureRandom());
final HostnameVerifier allHostsValid = new HostnameVerifier() {
#Override
public boolean verify(String hostname, SSLSession session) {
return true;
}
};
HttpsURLConnection.setDefaultHostnameVerifier(allHostsValid);
return sslContext;
}
OBS: The trustStore and keyStore have the same password, that's why the method has only one password parameter and used for both key and trust manager factories.
The way the getSslContext method is called and used is:
final SSLContext sslContext = SSLContextHelper.getSslContext("src/main/resources/myTrustStore.truststore",
"src/main/resources/myKeyStore.p12",
"myPassword");
final SSLSocketFactory sslSocketFactory = sslContext.getSocketFactory();
final URL url = new URL("https://serverIP:8443/path");
final HttpsURLConnection urlConnection = (HttpsURLConnection) url.openConnection();
urlConnection.setSSLSocketFactory(sslSocketFactory);
// tried adding some headers to the request
urlConnection.addRequestProperty("Content-Type", "application/json");
urlConnection.addRequestProperty("Accept", "application/json");
urlConnection.addRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0");
urlConnection.connect();
final InputStream inputstream = urlConnection.getInputStream();
The error is thrown at the last line when trying to get the inputStream of the URL connection.
Also, I tried using the following classes from org.apache.http: SSLConnectionSocketFactory, HttpClient, HttpGet, HttpResponse but response code is still 403.
I can only think that there is something missing from the SSL configuration because the system properties work. Any suggestions on what I miss setting in the SSLContext/SSLSocketFactory or how can I solve/better debug the problem are welcome! Thanks!

I managed to open the HTTPS connections only by using Spring's RestTemplate (org.springframework.web.client.RestTemplate) that uses the org.apache.http.client.HttpClient.
The method for getting the RestTemplate that has in its SSLContext the keyStore, trustStore and their passwords is the following:
public RestTemplate getRestTemplate(final String keyStoreFile, final String trustStoreFile,
final String password) throws Exception {
final SSLContext sslContext = SSLContextBuilder.create()
.loadKeyMaterial(ResourceUtils.getFile(keyStoreFile), password.toCharArray(), password.toCharArray())
.loadTrustMaterial(ResourceUtils.getFile(trustStoreFile), password.toCharArray())
.build();
final HttpClient client = HttpClients.custom()
.setSSLContext(sslContext)
.build();
final HttpComponentsClientHttpRequestFactory httpComponentsClientHttpRequestFactory = new HttpComponentsClientHttpRequestFactory();
httpComponentsClientHttpRequestFactory.setHttpClient(client);
return new RestTemplate(httpComponentsClientHttpRequestFactory);
}
The way that the RestTemplate is used for the HTTPS call is:
final String keyStoreFile = "src/main/resources/myKeyStore.p12";
final String trustStoreFile = "src/main/resources/myTrustStore.truststore";
final String password = "myPassword"; // same password for keyStore and trustStore
final String response = getRestTemplate(keyStoreFile, trustStoreFile, password).getForObject("https://serverIP:8443/path", String.class);
LOGGER.info("Response received: " + response);
Hope this helps anyone, had a lot of struggle with the HTTPS connections :)

java.lang.IllegalStateException: Unable to extract the trust manager on AndroidPlatform, sslSocketFactory

In my application, I have integrated two libraries i.e. chat and video calling...The problem is when I open video library first, then both chat and video calling libraries are working fine but when I open the chat library first, and then open the video, it leads to an exception...I think it is a problem with sockets with default trustmangers..
Sample code I am using in video calling library to create sslcontext
trustManagers = new TrustManager[]{new X509TrustManager() {
#Override
public void checkClientTrusted(X509Certificate[] chain, String authType)
throws CertificateException {
}
#Override
public void checkServerTrusted(X509Certificate[] chain, String authType)
throws CertificateException {
}
#Override
public X509Certificate[] getAcceptedIssuers() {
return new X509Certificate[0];
}
}
};
try {
sslContext = SSLContext.getInstance("TLS");
sslContext.init(null, trustManagers, null);
} catch (NoSuchAlgorithmException | KeyManagementException e) {
e.printStackTrace();
}
The exception I got this:
java.lang.IllegalStateException: Unable to extract the trust manager on AndroidPlatform, sslSocketFactory is class org.conscrypt.OpenSSLSocketFactoryImpl...Please help any one.

In my case it happened on emulator API 16 when I used https://stackoverflow.com/a/50264472/2914140 in this way:
val sslcontext: SSLContext = SSLContext.getInstance("TLSv1.2")
sslcontext.init(null, null, null)
val tlsSocketFactory = TLSSocketFactory()
val okHttpClient = OkHttpClient().newBuilder()
.sslSocketFactory(tlsSocketFactory)
.build()
When I add a trustManager parameter to sslSocketFactory method, the exception disappears. But a problem with SSL connection doesn't disappear.

How do I initialize a TrustManagerFactory with multiple sources of trust?

My application has a personal keystore containing trusted self-signed certificates for use in the local network - say mykeystore.jks. I wish to be able to connect to public sites(say google.com) as well as ones in my local network using self-signed certificates which have been provisioned locally.
The problem here is that, when I connect to https://google.com, path building fails, because setting my own keystore overrides the default keystore containing root CAs bundled with the JRE, reporting the exception
sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
However, if I import a CA certificate into my own keystore(mykeystore.jks) it works fine. Is there a way to support both?
I have my own TrustManger for this purpose,
public class CustomX509TrustManager implements X509TrustManager {
X509TrustManager defaultTrustManager;
public MyX509TrustManager(KeyStore keystore) {
TrustManagerFactory trustMgrFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
trustMgrFactory.init(keystore);
TrustManager trustManagers[] = trustMgrFactory.getTrustManagers();
for (int i = 0; i < trustManagers.length; i++) {
if (trustManagers[i] instanceof X509TrustManager) {
defaultTrustManager = (X509TrustManager) trustManagers[i];
return;
}
}
public void checkServerTrusted(X509Certificate[] chain, String authType)
throws CertificateException {
try {
defaultTrustManager.checkServerTrusted(chain, authType);
} catch (CertificateException ce) {
/* Handle untrusted certificates */
}
}
}
I then initialize the SSLContext,
TrustManager[] trustManagers =
new TrustManager[] { new CustomX509TrustManager(keystore) };
SSLContext customSSLContext =
SSLContext.getInstance("TLS");
customSSLContext.init(null, trustManagers, null);
and set the socket factory,
HttpsURLConnection.setDefaultSSLSocketFactory(customSSLContext.getSocketFactory());
The main program,
URL targetServer = new URL(url);
HttpsURLConnection conn = (HttpsURLConnection) targetServer.openConnection();
If I don't set my own trust managers, it connects to https://google.com just fine. How do I get a "default trust manager" which points to the default key store?

In trustMgrFactory.init(keystore); you're configuring defaultTrustManager with your own personal keystore, not the system default keystore.
Based on reading the source code for sun.security.ssl.TrustManagerFactoryImpl, it looks like trustMgrFactory.init((KeyStore) null); would do exactly what you need (load the system default keystore), and based on quick testing, it seems to work for me.

The answer here is how I came to understand how to do this. If you just want to accept the system CA certs plus a custom keystore of certs I simplified it into a single class with some convenience methods. Full code available here:
https://gist.github.com/HughJeffner/6eac419b18c6001aeadb
KeyStore keystore; // Get your own keystore here
SSLContext sslContext = SSLContext.getInstance("TLS");
TrustManager[] tm = CompositeX509TrustManager.getTrustManagers(keystore);
sslContext.init(null, tm, null);

I've run into the same issue with Commons HttpClient. Working solution for my case was to create delegation chain for PKIX TrustManagers in following way:
public class TrustManagerDelegate implements X509TrustManager {
private final X509TrustManager mainTrustManager;
private final X509TrustManager trustManager;
private final TrustStrategy trustStrategy;
public TrustManagerDelegate(X509TrustManager mainTrustManager, X509TrustManager trustManager, TrustStrategy trustStrategy) {
this.mainTrustManager = mainTrustManager;
this.trustManager = trustManager;
this.trustStrategy = trustStrategy;
}
#Override
public void checkClientTrusted(
final X509Certificate[] chain, final String authType) throws CertificateException {
this.trustManager.checkClientTrusted(chain, authType);
}
#Override
public void checkServerTrusted(
final X509Certificate[] chain, final String authType) throws CertificateException {
if (!this.trustStrategy.isTrusted(chain, authType)) {
try {
mainTrustManager.checkServerTrusted(chain, authType);
} catch (CertificateException ex) {
this.trustManager.checkServerTrusted(chain, authType);
}
}
}
#Override
public X509Certificate[] getAcceptedIssuers() {
return this.trustManager.getAcceptedIssuers();
}
}
And initialize HttpClient in following way (yes it's ugly):
final SSLContext sslContext;
try {
sslContext = SSLContext.getInstance("TLS");
final TrustManagerFactory javaDefaultTrustManager = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
javaDefaultTrustManager.init((KeyStore)null);
final TrustManagerFactory customCaTrustManager = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
customCaTrustManager.init(getKeyStore());
sslContext.init(
null,
new TrustManager[]{
new TrustManagerDelegate(
(X509TrustManager)customCaTrustManager.getTrustManagers()[0],
(X509TrustManager)javaDefaultTrustManager.getTrustManagers()[0],
new TrustSelfSignedStrategy()
)
},
secureRandom
);
} catch (final NoSuchAlgorithmException ex) {
throw new SSLInitializationException(ex.getMessage(), ex);
} catch (final KeyManagementException ex) {
throw new SSLInitializationException(ex.getMessage(), ex);
}
SSLConnectionSocketFactory sslSocketFactory = new SSLConnectionSocketFactory(sslContext);
PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager(
RegistryBuilder.<ConnectionSocketFactory>create()
.register("http", PlainConnectionSocketFactory.getSocketFactory())
.register("https", sslSocketFactory)
.build()
);
//maximum parallel requests is 500
cm.setMaxTotal(500);
cm.setDefaultMaxPerRoute(500);
CredentialsProvider cp = new BasicCredentialsProvider();
cp.setCredentials(
new AuthScope(apiSettings.getIdcApiUrl(), 443),
new UsernamePasswordCredentials(apiSettings.getAgencyId(), apiSettings.getAgencyPassword())
);
client = HttpClients.custom()
.setConnectionManager(cm)
.build();
In your case with simple HttpsURLConnection you may get by with simplified version of delegating class:
public class TrustManagerDelegate implements X509TrustManager {
private final X509TrustManager mainTrustManager;
private final X509TrustManager trustManager;
public TrustManagerDelegate(X509TrustManager mainTrustManager, X509TrustManager trustManager) {
this.mainTrustManager = mainTrustManager;
this.trustManager = trustManager;
}
#Override
public void checkClientTrusted(
final X509Certificate[] chain, final String authType) throws CertificateException {
this.trustManager.checkClientTrusted(chain, authType);
}
#Override
public void checkServerTrusted(
final X509Certificate[] chain, final String authType) throws CertificateException {
try {
mainTrustManager.checkServerTrusted(chain, authType);
} catch (CertificateException ex) {
this.trustManager.checkServerTrusted(chain, authType);
}
}
#Override
public X509Certificate[] getAcceptedIssuers() {
return this.trustManager.getAcceptedIssuers();
}
}
A detailed description of the solution is here: https://blog.novoj.net/posts/2016-02-29-how-to-make-apache-httpclient-trust-lets-encrypt-certificate-authority/

For Android developers, this can be much easier. In summary, you can add a xml res file to config your custom certs.
Step 1: open your manifest xml add an attribute.
<manifest ... >
<application android:networkSecurityConfig="#xml/network_security_config"
... >
...
</application>
</manifest>
Step 2: Add network_security_config.xml to res/xml, config certs as you want.
<?xml version="1.0" encoding="utf-8"?>
<network-security-config>
<base-config>
<trust-anchors>
<certificates src="#raw/extracas"/>
<certificates src="system"/>
</trust-anchors>
</base-config>
</network-security-config>
Note: this xml can support many other usage, and this solution only works on api24+.
Official reference: here

import com.google.common.collect.ImmutableList;
import com.google.common.collect.Iterables;
import java.security.KeyStore;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import java.util.Arrays;
import java.util.List;
import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManager;
import javax.net.ssl.TrustManagerFactory;
import javax.net.ssl.X509TrustManager;
/**
* Represents an ordered list of {#link X509TrustManager}s with additive trust. If any one of the composed managers
* trusts a certificate chain, then it is trusted by the composite manager.
*
* This is necessary because of the fine-print on {#link SSLContext#init}: Only the first instance of a particular key
* and/or trust manager implementation type in the array is used. (For example, only the first
* javax.net.ssl.X509KeyManager in the array will be used.)
*
* #author codyaray
* #since 4/22/2013
* #see <a href="http://stackoverflow.com/questions/1793979/registering-multiple-keystores-in-jvm">
* http://stackoverflow.com/questions/1793979/registering-multiple-keystores-in-jvm
* </a>
*/
#SuppressWarnings("unused")
public class CompositeX509TrustManager implements X509TrustManager {
private final List<X509TrustManager> trustManagers;
public CompositeX509TrustManager(List<X509TrustManager> trustManagers) {
this.trustManagers = ImmutableList.copyOf(trustManagers);
}
public CompositeX509TrustManager(KeyStore keystore) {
this.trustManagers = ImmutableList.of(getDefaultTrustManager(), getTrustManager(keystore));
}
#Override
public void checkClientTrusted(X509Certificate[] chain, String authType) throws CertificateException {
for (X509TrustManager trustManager : trustManagers) {
try {
trustManager.checkClientTrusted(chain, authType);
return; // someone trusts them. success!
} catch (CertificateException e) {
// maybe someone else will trust them
}
}
throw new CertificateException("None of the TrustManagers trust this certificate chain");
}
#Override
public void checkServerTrusted(X509Certificate[] chain, String authType) throws CertificateException {
for (X509TrustManager trustManager : trustManagers) {
try {
trustManager.checkServerTrusted(chain, authType);
return; // someone trusts them. success!
} catch (CertificateException e) {
// maybe someone else will trust them
}
}
throw new CertificateException("None of the TrustManagers trust this certificate chain");
}
#Override
public X509Certificate[] getAcceptedIssuers() {
ImmutableList.Builder<X509Certificate> certificates = ImmutableList.builder();
for (X509TrustManager trustManager : trustManagers) {
for (X509Certificate cert : trustManager.getAcceptedIssuers()) {
certificates.add(cert);
}
}
return Iterables.toArray(certificates.build(), X509Certificate.class);
}
public static TrustManager[] getTrustManagers(KeyStore keyStore) {
return new TrustManager[] { new CompositeX509TrustManager(keyStore) };
}
public static X509TrustManager getDefaultTrustManager() {
return getTrustManager(null);
}
public static X509TrustManager getTrustManager(KeyStore keystore) {
return getTrustManager(TrustManagerFactory.getDefaultAlgorithm(), keystore);
}
public static X509TrustManager getTrustManager(String algorithm, KeyStore keystore) {
TrustManagerFactory factory;
try {
factory = TrustManagerFactory.getInstance(algorithm);
factory.init(keystore);
return Iterables.getFirst(Iterables.filter(
Arrays.asList(factory.getTrustManagers()), X509TrustManager.class), null);
} catch (NoSuchAlgorithmException | KeyStoreException e) {
e.printStackTrace();
}
return null;
}
}

Although this question is 6 years old, I want to share my solution for this challenge. It uses the same code snippet under the covers from Cody A. Ray which Hugh Jeffner also shared.
SSLFactory sslFactory = SSLFactory.builder()
.withDefaultTrustMaterial() // --> uses the JDK trusted certificates
.withTrustMaterial("/path/to/mykeystore.jks", "password".toCharArray())
.build();
HttpsURLConnection.setDefaultSSLSocketFactory(sslFactory.getSslSocketFactory());
During the ssl handshake process it will first check if the server certificate is present in the jdk trusted certificates, if not it will continue by also checking your custom keystore and if it doesn't find a match it will fail. You can even further chain it with more custom keystores, or pem files, or list of certificates etc. See here for other configurations: other possible configurations
This library is maintained by me and you can find it here: https://github.com/Hakky54/sslcontext-kickstart

SSLContext and PKCS12 keystore - works on JVM but not on Android

Android version:
final SSLContext context = SSLContext.getInstance("TLS");
final KeyStore keystore = KeyStore.getInstance("PKCS12");
keystore.load(ctx.getAssets().open("ca_cli.pkcs12"), "password".toCharArray());
final KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
keyManagerFactory.init(keystore, "password".toCharArray());
context.init(keyManagerFactory.getKeyManagers(), new TrustManager[] { new X509TrustManager() {
#Override
public X509Certificate[] getAcceptedIssuers() { return new X509Certificate[] {}; }
#Override
public void checkClientTrusted(final X509Certificate[] arg0, final String arg1) throws CertificateException {
// TODO Auto-generated method stub
}
#Override
public void checkServerTrusted(final X509Certificate[] arg0, final String arg1) throws CertificateException {
// TODO Auto-generated method stub
}
} }, new SecureRandom());
JVM version:
final SSLContext context = SSLContext.getInstance("TLS");
final KeyStore keystore = KeyStore.getInstance("pkcs12");
keystore.load(new FileInputStream(new File("ca-cli.pkcs12")), "password".toCharArray());
final KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
keyManagerFactory.init(keystore, "password".toCharArray());
context.init(keyManagerFactory.getKeyManagers(), new TrustManager[] {
new X509TrustManager() {
#Override
public X509Certificate[] getAcceptedIssuers() { return new X509Certificate[]{}; }
#Override
public void checkClientTrusted(final X509Certificate[] arg0, final String arg1) throws CertificateException {
// TODO Auto-generated method stub
}
#Override
public void checkServerTrusted(final X509Certificate[] arg0, final String arg1) throws CertificateException {
// TODO Auto-generated method stub
}
}
}, new SecureRandom());
The code is pretty same but JVM version works fine and Android version produces:
09-16 12:24:17.024: E/AuthByPasswordLoader(14580): Got unexpected error
09-16 12:24:17.024: E/AuthByPasswordLoader(14580):
javax.net.ssl.SSLHandshakeException:
javax.net.ssl.SSLProtocolException: SSL handshake terminated:
ssl=0x656b0148: Failure in SSL library, usually a protocol error
09-16 12:24:17.024: E/AuthByPasswordLoader(14580): error:14094410:SSL
routines:SSL3_READ_BYTES:sslv3 alert handshake failure
(external/openssl/ssl/s3_pkt.c:1290 0x40086500:0x00000003)
I've tried to convert PKCS12 -> BKS but it doesn't helped...

So, in case someone face the same issue - it turned out that this is a bug, which was introduced in Android 3.0.
Quoting Kenny Root (Android Security Discussion thread):
Thanks, it looks like a bug that was introduced in Android 3.0. It will require that client key types have the same CA type. You are seeing this because your client certificate is RSA and the CA is EC.
You can work around this bug for your situation by wrapping the KeyManager and intercepting calls to chooseClientAlias to add "RSA_EC" to the keyTypes.
If you're interested in the particular fix that will be in a future release: https://android-review.googlesource.com/66581