We Keep Coding

AWS Kafka (MSK) - How to generate Keystore and truststore and use the same in my Spring Cloud Stream application?

Is there any info as to how can I use the AWS MSK details in my Spring Cloud Stream application ?
I believe we need to generate a keystore and truststore and then incorporate the same in our application ? I went through the "Client Authentication" page of the AWS MSK and found that to be very confusing.
Can anyone help me with steps on this ? I am just trying to deploy this application which uses the AWS MSK (3 brokers).
Thank you.

Short answer:
Your kafka client will need this in the configuration:
# security settings
security.protocol=SSL
ssl.truststore.location=/tmp/kafka.client.truststore.jks
ssl.truststore.password=
ssl.endpoint.identification.algorithm=
That is if you use the same JVM truststore from the tutorial, and no password. The ssl.endpoint.identification.algorithm turns off the host name verification.
Long answer:
I wondered the same thing after going through the tutorial, wondering why the JVM truststore magically works when connecting to MSK. The explanation is this:
If you take a peek at what certificates this truststore imported
keytool --list -v -keystore /tmp/kafka.client.truststore.jks | grep Owner
One of them is Starfield Services Root Certificate Authority, when Amazon purchased the company, the CA became one of Amazon's (see all of them here https://www.amazontrust.com/repository/). Since JVM truststore trusts this CA, it also trusts anything signed by the CA, and the MSK cluster is one of them.
If you would prefer to generate your own truststore, download one of the Amazon's certificate and import
keytool -keystore kafka.client.truststore.jks -alias CARoot -importcert -file {downloaded-cert} -storepass {your-password}
Thanks,
Yanan

I was looking for the same thing so posting as an answer might help someone else. Here is official documentation that can help.
On Ubuntu machine follow working for me, also you do not need to import/create just used the existing cacerts
cp /home/ubuntu/.sdkman/candidates/java/8.0.232.hs-adpt/jre/lib/security/cacerts /tmp/kafka.client.truststore.jks
create a text file named client.properties with the following contents.
security.protocol=SSL
ssl.truststore.location=/tmp/kafka.client.truststore.jks
and we are good to consume message
./kafka-console-consumer.sh --bootstrap-server mykafka.kafka.us-west-2.amazonaws.com:9094 --consumer.config /home/ubuntu/client.properties --topic demo --from-beginning

Debian SSL certificates ERROR: cannot verify xxx certificate / javax.net.ssl.SSLHandshakeException

I have a strange problem (it's 100% server configuration problem,) for example I want to download something from Dropbox:
Resolving dl.dropboxusercontent.com... 23.23.160.146, 50.17.227.107,
54.221.248.69, ... Connecting to dl.dropboxusercontent.com|23.23.160.146|:443... connected. ERROR:
cannot verify dl.dropboxusercontent.com’s certificate, issued by
“/C=US/ST=CA/O=SonicWALL Inc./CN=SonicWALL Firewall DPI-SSL”:
Self-signed certificate encountered. To connect to
dl.dropboxusercontent.com insecurely, use ‘--no-check-certificate’.
Yes, I know that I can use --non-check-certificate but when I want to use SSL connection in Java app I have something like this:
javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to
find valid certification path to requested target
This app works great in other servers or in local machines, any ideas what is wrong here?

/C=US/ST=CA/O=SonicWALL Inc./CN=SonicWALL Firewall DPI-SSL
Your traffic is visibly intercepted by a deep packet inspection firewall that acts as a MITM proxy to monitor your traffic.
This can generally be thought of as "legitimate" MITM attacker. (However legitimate this may be depends on a number of legal and ethical aspects.) Your local network administrator should be able to tell you a little bit about this. If this is part of a company network, this company is monitoring your traffic, including the contents of your HTTPS connection (so it's no longer secure from end-to-end). It the firewall does its job properly, it should still secure the connection from the firewall to the server (It's probably hard to know whether it check certificates properly.)
In general, such a firewall or proxy acts as its own Certification Authority, effectively forging each certificate as requested.
Most clients on the corporate network would trust certificates it issues (like the one you're facing) because system administrators would also install the CA certificate as a trusted certificate into each machine within that network. You probably have it the OS trusted root certificates.
However, it is likely that network administrators did not install this CA certificate into your JRE installation (which uses its own set of trust anchors by default).
Try to export that CA certificate (see the name above) from a reference machine and import it into the truststore you're using (either your JRE's default truststore: cacerts or a new truststore you build and pass to your application via the javax.net.trustStore properties).
Typically, you can use something like this (assuming you've exported the Firewall's CA as "firewallca.pem"):
keytool -import -file firewallca.pem -alias firewallca -keystore truststore.jks
If the truststore.jks file doesn't exist, it will be created. Otherwise, you can take a copy of the cacerts file in the lib/security directory of your JRE. You can also do this directly on the cacerts file (using -keystore /path/to/truststore.jks, provided you have write access to it).
If you choose not to do it on the default truststore (that cacerts file) but use a local truststore like truststore.jks, you'll can use it by using this system property when running your application: -Djavax.net.trustStore=/path/to/truststore.jks
For other ways of configuring your truststore, check this answer.
Another way to fix this is to tell Java to use your OS's truststore. I'll assume you're using Windows here. Using these system properties should work:
-Djavax.net.trustStore=NONE -Djavax.net.trustStoreType=WINDOWS-ROOT
(Try with WINDOWS-MY instead of WINDOWS-ROOT if this doesn't work.)
The WINDOWS-MY/WINDOWS-ROOT is a bit buggy in that it will miss some of the certificates in the Windows store: it uses the certificate "friendly name" (non unique) as the keystore alias (unique), so a certificate with a given friendly name will hide the others with the same name. (This effectively reduces the number of CA certificates that are trusted unfortunately.)
Since you're in an environment where presumably all the traffic goes through your DPI firewall, you'll probably only ever have to use one CA certificate at most. As long is it doesn't share its "friendly name" in the Windows list with any other cert, you should be fine.

You need to add the server's SSL certificate in your client's Java keystore. Take a look at this SO post:
Resolving javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed Error?
TIP: This is because your client's JVM does not 'trust' the server's SSL cert. So you need to add the cert in your keystore.

how to enable trust for chain certs in weblogic or java

I have an Oracle service Bus domain running on weblogic 10.3.6 with 2 managed servers in cluster.
We have a proxy service deployed on this domain which goes to an external Business service for validating address. This Business Service is listening on SSL port and SSL cert has EntrustCACert>IntermidateCert1>IntermidateCert2>ServerCert
Issue: Getting below error during connectivity testing -
General runtime error: [Security:090548]The certificate chain received from ws2.site1.com - 197.109.80.xxx contained a V3 CA certificate which was missing the basic constraints.
Solutions i have tried:
1) added JAVA_OPTIONS="${JAVA_OPTIONS} -Dweblogic.security.SSL.enforceConstraints=off " in the startup script and it resolve the issue. But i was told this is not the prefered way and use trust keystore to implement.
2)to implement trust, I copied all the 4 certs in following reverse order EntrustCACert>IntermidateCert1>IntermidateCert2>ServerCert in one file named DSperian.pem and imported in Jrockit cacerts file( /apps/Oracle/jrockit-jdk1.6.0_31-R28.2.3-4.1.0/jre/lib/security/cacerts ) using below command but getting the same above error.
keytool -import -alias DSperian -trustcacerts -file DSperian.pem -keystore cacerts
Question:
Please let me know if im doing the right way to import cert to create trust. so my OSB domain will blindly trust the Business service ( web-service) and ignore
the "basic constraints" error. Do i need to utilize weblogic specific trust keystore file but this OSB weblogic is running on non-ssl port ?
are there any other options available ? Asking Business-service to update their cert to include "basic constraints" is not an option.

There is actually a problem with your certificate.
Security certificates have a set of constraints that allow them to perform certain functions (or restrict them to certain functions).
Have a look at each of the certificates and ensure that they have constraints assigned to them that are suitable for the task.
To get more details on the error enable SSL debugging in your web logic server add the following to your server startup script
-Dssl.debug=true -Dweblogic.StdoutDebugEnabled=true
More information in Configure SSL in WebLogic server
Use the following command to validate your keystone
java utils.ValidateCertChain -jks my key mykeystore
Then, whoever provided the root certificate needs to fix up your issues.

Why do I get a handshake failure (Java SSL)

I'm connecting to a web service over HTTPS. I've done all that I think is required to make it work, but in the end I get a handshake failure.
I found out that as a new user I can't post more than 2 links due to "spam protection" - thanx a lot stackoverflow...anyway here's a link to a pastebin post with all the links spelled out...so when I write "link#1" here it's a reference to these links: http://pastebin.com/y4zGNRC7
I verified the same behavior using HttpClient (GET on the service URL) and actually calling the web service via a CXF proxy
I'm setting both the keystore and truststore - I tried both the "in code" way ( link#1 ) and setting the system properties - i.e. System.setProperty("javax.net.ssl.keyStore", "mykeystore.jks");
SSL debug is on ( javax.net.debug=all )
SSL debug blurts out the contents of both keystore and truststore (i.e. looks like java "knows about them") - link#2
seems like there's some client-server communication going on, but then it crashes for some reason link#3
I successfully connected to the server using the client and CA certificates both in a browser (Chrome) and using openssl s_client
wireshark shows less client-server talk from java ( link#4 ) then for example from Chrome ( link#5 )
Another strange thing is, that I seem to be getting the same behavior when I set the keystore and when I don't (the only difference is that when I do the keystore contents get printed in the console, but that's it).
I tried googling the problem and I saw numerous similar posts here on stackoverflow, but nothing helped.
I tried changing the protocol version ("TLSv1", "SSLv3", even the weird v2 Hello).
Any help would be appreciated - maybe there's some fundamental thing I might have overlooked...I'm getting desperate here...
Thanx
PS I'm running java 1.6 update 30 on Fedora Core 15 (64bit)

The problem was that even though the keystore and truststore was set, java decided not to send the client certificate to the server. The reason for this was the fact, that the server requested a certificate signed by the RootCA authority, but the client certificate is signed by a SubCA authority (which is issued by the RootCA).
Originally the keystore only contained the client cert and the truststore the SubCA cert.
I then tried to add the SubCA cert to the keystore too, but java just ignored it.
So this solves the hanshake failure mystery, but not my problem.
I created a separate question for that...sigh :-(
why doesn't java send the client certificate during SSL handshake?

I think the trust store not containing the CA is the most likely issue. You can use the Java keytool to import the certificate for the site into the cacerts file doing something like:
keytool -keystore pathtocacerts -import -trustcacerts -v -alias aliasName -file root.crt
The default cacerts keystore password is changeit. The cacerts file is usually under jre/lib/security directory.

You don't provide enough information, but I'm guessing your client truststore is not properly configured. The truststore contains the trusted certificates that are used to sign other certs, and must include the root certificate(s) for the server and client cert chains. The client keystore contains the client SSL certificate and private key.

Firefox "ssl_error_no_cypher_overlap" error

My co-workers and I are having a problem using Firefox 3.0.6 to access a Java 1.6.0___11 web application we're developing. Everything works fine anywhere from 1-30 minutes into the session...but eventually, the connection fails and the following error appears:
Secure Connection Failed
An error occurred during a connection to 10.x.x.x.
Cannot communicate securely with peer: no common encryption algorithm(s).
(Error code: ssl_error_no_cypher_overlap)
IE works fine. Firefox throws the error in both Windows and Fedora, so the problem doesn't appear to be tied to an OS. The Java EE application runs on a Tomcat 6.0.16 server. All pages are encrypted using TLS 1.0 through an Apache 2.2.8 HTTP server with mod_nss.
Our Apache server is configured to reject SSL 3.0 connections. One hypothesis we have is that Firefox might be trying to establish a SSL 3.0 connection...but why?
Based some Googling, we tried the following things, but without success:
using Firefox 2.x (some people reported instances where 2.x worked but 3.x didn't):
enabling SSL2
disabling SSL3
disabling OCSP (Tool > Options > Advanced > Encryption > Validation)
ensuring that the anti-virus/firewall of the client computer isn't blocking or scanning port 443 (https port)
Any ideas?

I had the same issue while renewing the certificate for our server at www.tpsynergy.com . After importing the new server certificate and restarting the tomcat, the error we were getting was ERR_SSL_VERSION_OR_CIPHER_MISMATCH. After lot of research, I used this link https://www.sslshopper.com/certificate-key-matcher.html to compare the csr (certificate signing request to the actual certificate). They both did not match. So I created a new csr and obtained a new certificate and installed the same. It worked.
So the full steps for the process are
From the same server where the certificate will be installed, create CSR
keytool -keysize 2048 -genkey -alias tomcat -keyalg RSA -keystore tpsynergy.keystore
(change the domain name as needed)
While creating this, it will ask for first name and last name. Do not give your name, but use the domain name. For example I gave it as www.tpsynergy.com
2.keytool -certreq -keyalg RSA -alias tomcat -file csr.csr -keystore tpsynergy.keystore
This will create a csr.csr file in the same folder. copy the contents of this to the godaddy site and create the new certificate.
The downloaded certificate zip file will have three files
gd_bundle-g2-g1.crt
gdig2.crt
youractualcert.crt
You will need to download the root cert gdroot-g2.crt from godaddy repository.
Copy all these files to the same directory from where you created the CSR file and where the keystore file is located.
Now run the below commands one by one to import the certs into the keystore
keytool -import -trustcacerts -alias root -file gd_bundle-g2-g1.crt -keystore tpsynergy.keystore
keytool -import -trustcacerts -alias root2 -file gdroot-g2.crt -keystore tpsynergy.keystore
keytool -import -trustcacerts -alias intermediate -file gdig2.crt -keystore tpsynergy.keystore
keytool -import -trustcacerts -alias tomcat -file yourdomainfile.crt -keystore tpsynergy.keystore
Ensure that server.xml file in conf folder has this entry
Restart the tomcat

Given what you've tried and the error messages, I'd say this was more to do with the exact cipher algorithm used rather than the TLS/SSL version. Are you using a non-Sun JRE by any chance, or a different vendor's security implementation? Try a different JRE/OS to test your server if you can. Failing that you might just be able to see what's going on with Wireshark (with a filter of 'tcp.port == 443').

If you review the process of SSL negotiation at Wikipedia, you will know that at the beginning ClientHello and ServerHello messages are sent between the browser and the server.
Only if the cyphers provided in ClientHello have overlapping items on the server, ServerHello message will contain a cypher that both sides support. Otherwise, SSL connection will not be initiated as there is no common cypher.
To resolve the problem, you need to install cyphers (usually at OS level), instead of trying hard on the browser (usually the browser relies on the OS). I am familiar with Windows and IE, but I know little about Linux and Firefox, so I can only point out what's wrong but cannot deliver you a solution.

I had similar issues browsing to secure sites (https://) when using Burp (or at least an issue that would bring you to this page when searching Google):
ssl_error_no_cypher_overlap in Firefox
ERR_SSL_VERSION_OR_CIPHER_MISMATCH in Chrome
It turned out to be an issue with using Java 8. When I switched to Java 7, the problem stopped.

Under advanced settings of firefox you should be able to set the encryption. By default SSL3.0 and TLS1.0 should be checked, so if firefox is trying to create ssl 3.0 connectons try unchecking the ssl 3.0s setting.
if that doesn't work, try searching the about:config page for "ssl2"
My Firefox has settings with ssl2 set to false by default...

The first thing I would check is the config for mod_nss. It is the odd one out, for it is yours and there is none in the world like it :-) Whereas if there was some huge bug in Firefox or mod_nss itself, I guess you'd have found out about it by now in your google quest. The fact that you've fiddled with the config (e.g. disabling SSL3, and various other random tweaks), is also suspicious.
I'd back track to a very vanilla mod_nss config and see if that works. Then change things systematically towards your current config until you can reproduce the problem. By the sound of it the source of the error is somewhere in the cipher spec config of mod_nss and the related protocol negotiation stuff. So maybe you inadvertently changed something there when trying to turn off SSLv3 (incidentally, why disable SSL3? Normally people disable V2?).
One other thing to check is that you're on the latest mod_nss and it's not a known bug in that. The fact that it manages to start the session and then fails later is interesting - it suggests that maybe it is trying to renegotiate the session and failing to negotiate ciphers at that point. So it might be the symmetric ciphers. Or it could simply be an implementation bug in your version of mod_nss that somehow garbles the protocol.
One other idea, and this is a wild guess, is the browser is trying to resume a session which was negotiated with SSLv3 before you disabled it, and something breaks when trying to resume that session when V3 is turned off, or maybe mod_nss just doesn't implement it right.
The java/tomcat stuff seems like a red herring as unless I've misunderstood your description, none of that is involved in the SSL handshake/protocol.

I had the same problem with a really old local router and was not able to open its WebGUI because of self-signed certificates. The solution was to install an old Firefox Portable version. I tested the following versions:
Firefox 33.1.1 (worked)
Firefox 45.0.2 (worked)
Firefox 56.0.2 (failed)
This is strange because it should be only a problem since version 59, but as long it works, it's ok for me.

If you get the no cipher overlap error on firefox, and you have left it at default settings, you are using what must be a very insecure site trying to use a very weak "export grade" cipher. Use of these ciphers is discouraged these days and I personally would stop using a site trying to use such a weak cipher.

I've had the same problem; to solve was enough to enable all the SSL schemas in "about:config". I was finding them by filtering with ssl. First I anabled all options for afret disabling the unnecessary ones.

"Error code: ssl_error_no_cypher_overlap" error message after login, when Welcome screen expected--using Firefox browser
Solution
Enable support for 40-bit RSA encryption in the Firefox Browser:
1: enter 'about:config' in Browser Address bar
2: find/select "security.ssl3.rsa_rc4_40_md5"
3: set boolean to TRUE

What worked for me is I:
Went to about:config.
Typed "security" in the search box.
Set all of the returned entries to their defaults.
Typed "ssl" in the search box.
Set all of the returned results to their defaults.
Enabled ssl2.
Disabled ssl3.
Restarted Firefox.
Note about restarting Firefox: When I do start it very soon after closing it,
it often has a file access problem, which requires me to delete places.sqlite
and places.sqlite-journal in C:\WINDOWS\Application Data\Mozilla\Firefox\Profiles\n18091xv.default. This causes me to lose my history, plus bookmarks have to be
restored from a backup each time this happens. I wait from five to ten minutes or
more to avoid this hassle.
Running Firefox v3.5.1 on WinMe

"Error code: ssl_error_no_cypher_overlap" error message after login, when Welcome screen expected--using Firefox browser
Solution
1: enter 'about:config' in Browser Address bar
2: find/select "security.ssl3.rsa_rc4_40_md5"
3: set boolean to TRUE