We Keep Coding

SunMSCAPI not recognizing private keys for some keystore entries

I apologize that this is so long. If you are familiar with doing client-auth in Java you can probably skim/skip to the bottom. I took me a long time to find all the relevant bits of information from disparate sources. Maybe this will help someone else.
I'm working on a proof-of-concept to get client-side authentication working from a Java client using the Windows keystore. I have a servlet that I have created that can request or require client certificates. It simply returns an HTML response containing the certificate information including subject DN and the serial number. I've worked through a number of experiments with browsers (mainly Chrome and IE) to verify it is working. I've gotten successful client authentication working with both certs I've generated with SSL and also using certs issued by my companies internal CA.
My next step is to have a Java client that works with the MSCAPI keystore in Java. The reason I went this route is that the certificates we want to use are issued by our internal CA and are automatically added to the Personal keystore in Windows and are marked as non-exportable. I know there are ways to get the private key out of the keystore with some free tools if you are an admin on your workstation. This isn't a viable option in this case for various reasons. Here's the code I ended up with:
private static SSLSocketFactory getFactory() throws NoSuchAlgorithmException, KeyStoreException, CertificateException, IOException, UnrecoverableKeyException, KeyManagementException
{
KeyStore roots = KeyStore.getInstance("Windows-ROOT", new SunMSCAPI());
KeyStore personal = KeyStore.getInstance("Windows-MY", new SunMSCAPI());
roots.load(null,null);
personal.load(null,null);
TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
tmf.init(roots);
KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
kmf.init(personal, "".toCharArray());
SSLContext context = SSLContext.getInstance("TLS");
context.init(kmf.getKeyManagers(), tmf.getTrustManagers(), new SecureRandom());
return context.getSocketFactory();
}
This works but when I connect, the Java client is authenticating with the client-certs I had previously generated. I don't see an obvious way to force the selection of a specific key so I figured it was just picking the first key it came across that the host trusts. I then removed these certs that I had created from the windows keystore and it would no longer authenticate. I triple-checked that the server-side was still works with the browsers and it does. I then readded these certs to the personal keystore and removed the trust of my pseudo-CA from the server-side. Still works in the browser and not in the client.
I added -Djavax.net.debug=ssl:handshake to my VM parameters and started looking through the output. One thing I see is are lines "found key for : [alias]" for each of the certs that I have added but not for the ones added through the 'self-enrollment' feature of certmgr.msc. Initially I thought it was because they were exportable but I removed them and added one back as non-exportable and java can still use it. I'm at a loss as to why SunMSCAPI won't use these certs. Both the ones created by me and internal CA are sha256RSA. Both are enabled for client authentication. When I add the following code, I see that isKeyEntry() is false for all the key-pairs I want to use:
Enumeration<String> aliases = personal.aliases();
while (aliases.hasMoreElements())
{
String alias = aliases.nextElement();
System.out.println(alias + " " + personal.isKeyEntry(alias));
}
I found someone else with a similar issue here but no definitive answer.

Convert X509Certificate to PEM or ASN.1/DER

I have a chain of certificates (X509Certificate []), but I have only one certificate in the chain. I need to get the complete chain.
I have tried the openssl command, but that is not useful here. Can someone please tell me how to:
Convert this X509Certificate to PEM or ASN.1/DER that I can save in my file storage?
Get the complete chain using this certificate?
Edit:
So, code-wise what I'm trying to achieve is something like:
protected static String convertToPem(X509Certificate cert) {
Base64 encoder = new Base64(64);
String cert_begin = "-----BEGIN CERTIFICATE-----\n";
String end_cert = "-----END CERTIFICATE-----";
byte[] derCert = cert.getEncoded();
String pemCertPre = new String(Base64.encodeBase64(derCert));
String pemCert = cert_begin + pemCertPre + end_cert;
return pemCert;
}
But, this is not working. Basically, I'm looking for a method that takes a X509Certificate object and then converts it to a .pem etc, that is saved on the device.

Convert this X509Certificate object to .cer/ .per/ .der that I can save in my file storage?
See, for example, the answer at OpenSSL's rsautl cannot load public key created with PEM_write_RSAPublicKey. It tells you how to convert keys to/from PEM and ASN.1/DER format, and includes a treatment of Traditional Format (a.k.a. SubjectPublicKeyInfo).
If you are not doing it programmatically, then you should search for the answer. There are plenty of off-topic question on how to use the openssl command to convert between ASN.1/DER and PEM. Or ask on Super User, where they specialize in commands and their use.
Get the complete chain using this certificate?
This is a well known problem in PKI called the Which Directory problem. The solution is to have the server or service provide the missing intermediate CA certificates. If you can't validate a web server or service's identity because you are missing intermediate CA certificates, then the server is misconfigured.
Once you have the intermediate CA certificates, you still have to root trust somewhere. You can use the self-signed CA, or one of the intermediates signed by the self-signed CA.
This answer is helpful in troubleshooting a misconfugred server using OpenSSL's s_client: SSL site and browser warning.
Related: if there was a global directory of certificates like the ITU envisioned in X.500, then you would not have the second problem. A relying party or user agent would just fetch the certificate it needed from the directory.
But we lack a central directory, so relying parties and user agents often use the CA Zoo (a.k.a., the local Trust Store or cacerts.pem). This has its own set of problems, like the wrong CA certifying a site or service.
One of the off-shoots is the CA Cartel, where browser are in partnership with the CAs at the CA/Browser Forum. Browser have requirements for inclusion, but they often can't punish a misbehaving CA like Trustwave.
And the browsers have managed to box themselves into a position where the Internet of Things (IoT) will not work because of the browser's reliance/requirements on server certificates signed by a CA.

JAVA Use two keystore same application

I'm trying to develop an application in java that basically gets a large number of data from REST-JSONs APIs and insert that on a DB. My problem is that two of the URLs I was working with, thrown errors about SSL certificates. To solve that, I created a jks file and inserted their certificates. The problem is, once I put that on my code, jvm only uses the certificates on that file, all others are rejected. So I want to make my application accepts jsons from either "cacerts.jks" and "custom.jks". Please don't tell me to add the certificates to the "teste.jks" (native from jvm), because the app won't run on my computer, I need the certificates on a sepparated file.
One more observation, I'm working with about 90 different URLs and I'm using the following line to indicate jvm, which keystore to use:
System.setProperty("javax.net.ssl.trustStore",System.getProperty("user.dir")+"/libs/teste.jks");
//or (for native one):
System.setProperty("javax.net.ssl.trustStore", "cacerts.jks");

Implement a subclass of javax.net.ssl.TrustManager or one of its subclasses like javax.net.ssl.X509TrustManager (MyTrustManager in the example below) and tweak the SSLContext to use it. Your trust manager implementation can do whatever you want with regard to validating/trusting certificates.
SSLContext context = SSLContext.getInstance("TLS");
MyTrustManager tm = new MyTrustManager();
SecureRandom sr = SecureRandom.getInstance("SHA1PRNG", "SUN");
context.init(null, new TrustManager[] { tm }, sr);
SSLContext.setDefault(context);
HttpsURLConnection.setDefaultSSLSocketFactory(context.getSocketFactory());

Create two TrustManagers, one that uses the default truststore and one that uses your own, and initialize the SSLContext with both:
sslContext.init(null, new TrustManager[]{tm1, tm2}, null);
See the JSSE Reference Guide for further details.

If there's more than one certificate in Jetty's key store, how does it choose?

There is some code in our system for automatically generating self-signed certificates into a key store which is then used by Jetty. If a key for a given host already exists then nothing happens but if it doesn't exist, we generate a new key, like this:
public void generateKey(String commonName) {
X500Name x500Name = new X500Name("CN=" + commonName);
CertAndKeyGen keyPair = new CertAndKeyGen("DSA", "SHA1withDSA");
keyPair.generate(1024);
PrivateKey privateKey = keyPair.getPrivateKey();
X509Certificate certificate = keyPair.getSelfCertificate(x500Name, 20*365*24*60*60);
Certificate[] chain = { certificate };
keyStore.setEntry(commonName, privateKey, "secret".toCharArray(), chain);
}
This all works fine as long as there is only one key and certificate in the key store. Once you have multiple keys, weird things happen when you try to connect:
java.io.IOException: HTTPS hostname wrong: should be <127.0.0.1>
This was quite a mystifying error but I finally managed to track it down by writing a unit test which connects to the server and asserts that the CN on the certificate matches the hostname. What I found was quite interesting - Jetty seems to arbitrarily choose which certificate to present to the client, but in a consistent fashion.
For instance:
If "CN=localhost" and "CN=cheese.mydomain" are in the key store, it always chose "CN=cheese.mydomain".
If "CN=127.0.0.1" and "CN=cheese.mydomain" are in the key store, it always chose "CN=cheese.mydomain".
If "CN=192.168.222.100" (cheese.mydomain) and "CN=cheese.mydomain" are in the key store, it always chose "CN=192.168.222.100".
I wrote some code which loops through the certificates in the store to print them out and found that it isn't consistently choosing the first certificate or anything trivial like that.
So exactly what criteria does it use? Initially I thought that localhost was special but then the third example baffled me completely.
I take it that this is somehow decided by the KeyManagerFactory, which is SunX509 in my case.

This is indeed ultimately decided by the KeyManager (generally obtained from a KeyManagerFactory).
A keystore can have a number of certificates stored under different aliases. If no alias is explicitly configured via certAlias in the Jetty configuration, the SunX509 implementation will pick the first aliases it finds for which there is a private key and a key of the right type for the chosen cipher suite (typically RSA, but probably DSA in your case here). There's a bit more to it to the choice logic, if you look at the Sun provider implementation, but you shouldn't really rely on the order in general, just the alias name.
You can of course give Jetty your own SSLContext with your own X509KeyManager to choose the alias. You would have to implement:
chooseServerAlias(String keyType, Principal[] issuers, Socket socket)
Unfortunately, apart from keyType and issuers, all you get to make the decision is the socket itself. At best, the useful information you get there are the local IP address and the remote one.
Unless your server is listening to multiple IP addresses on the same port, you will always get the same local IP address. (Here, obviously, you have at least two: 127.0.0.1 and 192.168.222.100, but I suspect you're not really interested in localhost except for your own tests.) You would need Server Name Indication (SNI) support on the server side to be able to make a decision based on the requested host names (by clients that support it). Unfortunately, SNI was only introduced in Java 7, but only on the client side.
Another problem you will face here is that Java clients will complain about IP addresses in the Subject DN's CN. Some browsers would tolerate this, but this is not compliant with the HTTPS specification (RFC 2818). IP addresses must be Subject Alternative Name entries of IP-address type.

Java SSL connect, add server cert to keystore programmatically

I am connecting an SSL client to my SSL server.
When the client fails to verify a certificate due to the root not existing in the client's key store, I need the option to add that certificate to the local key store in code and continue.
There are examples for always accepting all certificates, but I want the user to verify the cert and add it to local key store without leaving the application.
SSLSocketFactory sslsocketfactory = (SSLSocketFactory) SSLSocketFactory.getDefault();
SSLSocket sslsocket = (SSLSocket) sslsocketfactory.createSocket("localhost", 23467);
try{
sslsocket.startHandshake();
} catch (IOException e) {
//here I want to get the peer's certificate, conditionally add to local key store, then reauthenticate successfully
}
There is a whole lot of stuff about custom SocketFactory, TrustManager, SSLContext, etc and I don't really understand how they all fit together or which would be the shortest path to my goal.

You could implement this using a X509TrustManager.
Obtain an SSLContext with
SSLContext ctx = SSLContext.getInstance("TLS");
Then initialize it with your custom X509TrustManager by using SSLContext#init. The SecureRandom and the KeyManager[] may be null. The latter is only useful if you perform client authentication, if in your scenario only the server needs to authenticate you don't need to set it.
From this SSLContext, get your SSLSocketFactory using SSLContext#getSocketFactory and proceed as planned.
As concerns your X509TrustManager implementation, it could look like this:
TrustManager tm = new X509TrustManager() {
public void checkClientTrusted(X509Certificate[] chain,
String authType)
throws CertificateException {
//do nothing, you're the client
}
public X509Certificate[] getAcceptedIssuers() {
//also only relevant for servers
}
public void checkServerTrusted(X509Certificate[] chain,
String authType)
throws CertificateException {
/* chain[chain.length -1] is the candidate for the
* root certificate.
* Look it up to see whether it's in your list.
* If not, ask the user for permission to add it.
* If not granted, reject.
* Validate the chain using CertPathValidator and
* your list of trusted roots.
*/
}
};
Edit:
Ryan was right, I forgot to explain how to add the new root to the existing ones. Let's assume your current KeyStore of trusted roots was derived from cacerts (the 'Java default trust store' that comes with your JDK, located under jre/lib/security). I assume you loaded that key store (it's in JKS format) with KeyStore#load(InputStream, char[]).
KeyStore ks = KeyStore.getInstance("JKS");
FileInputStream in = new FileInputStream("<path to cacerts"");
ks.load(in, "changeit".toCharArray);
The default password to cacerts is "changeit" if you haven't, well, changed it.
Then you may add addtional trusted roots using KeyStore#setEntry. You can omit the ProtectionParameter (i.e. null), the KeyStore.Entry would be a TrustedCertificateEntry that takes the new root as parameter to its constructor.
KeyStore.Entry newEntry = new KeyStore.TrustedCertificateEntry(newRoot);
ks.setEntry("someAlias", newEntry, null);
If you'd like to persist the altered trust store at some point, you may achieve this with KeyStore#store(OutputStream, char[].

In the JSSE API (the part that takes care of SSL/TLS), checking whether a certificate is trusted doesn't necessarily involve a KeyStore. This will be the case in the vast majority of cases, but assuming there will always be one is incorrect. This is done via a TrustManager.
This TrustManager is likely to use the default trust store KeyStore or the one specified via the javax.net.ssl.trustStore system property, but that's not necessarily the case, and this file isn't necessarily $JAVA_HOME/jre/lib/security/cacerts (all this depends on the JRE security settings).
In the general case, you can't get hold of the KeyStore that's used by the trust manager you're using from the application, more so if the default trust store is used without any system property settings.
Even if you were able to find out what that KeyStore is, you would still be facing two possible problems (at least):
You might not be able to write to that file (which is not necessarily a problem if you're not saving the changes permanently).
This KeyStore might not even be file-based (e.g. it could be the Keychain on OSX) and you might not have write access to it either.
What I would suggest is to write a wrapper around the default TrustManager (more specifically, X509TrustManager) which performs the checks against the default trust manager and, if this initial check fails, performs a callback to the user-interface to check whether to add it to a "local" trust store.
This can be done as shown in this example (with a brief unit test), if you want to use something like jSSLutils.
Preparing your SSLContext would be done like this:
KeyStore keyStore = // ... Create and/or load a keystore from a file
// if you want it to persist, null otherwise.
// In ServerCallbackWrappingTrustManager.CheckServerTrustedCallback
CheckServerTrustedCallback callback = new CheckServerTrustedCallback {
public boolean checkServerTrusted(X509Certificate[] chain,
String authType) {
return true; // only if the user wants to accept it.
}
}
// Without arguments, uses the default key managers and trust managers.
PKIXSSLContextFactory sslContextFactory = new PKIXSSLContextFactory();
sslContextFactory
.setTrustManagerWrapper(new ServerCallbackWrappingTrustManager.Wrapper(
callback, keyStore));
SSLContext sslContext = sslContextFactory.buildSSLContext();
SSLSocketFactory sslSocketFactory = sslContext.getSslSocketFactory();
// ...
// Use keyStore.store(...) if you want to save the resulting keystore for later use.
(Of course, you don't need to use this library and its SSLContextFactory, but implement your own X509TrustManager, "wrapping" or not the default one, as you prefer.)
Another factor you have to take into account is the user interaction with this callback. By the time the user has made the decision to click on accept or reject (for example), the handshake may have timed out, so you may have to make a second attempt to connect when the user has accepted the certificate.
Another point to take into account in the design of the callback is that the trust manager doesn't know which socket is being used (unlike its X509KeyManager counterpart), so there should be as little ambiguity as to what user action caused that pop-up (or however you want to implement the callback). If multiple connections are made, you wouldn't want to validate the wrong one.
It seems possible to solve this by using a distinct callback, SSLContext and SSLSocketFactory per SSLSocket that's supposed to make a new connection, some way of tying up the SSLSocket and the callback to the action taken by the user to trigger that connection attempt in the first place.