i have set up a https connection between the server and the client, where the client is a java program and the server is a servlet. I have used the following code to print the certificate details from the server.
URL url = new URL("https://localhost:8443/cert");
HttpsURLConnection connection = (HttpsURLConnection) url.openConnection();
connection.setSSLSocketFactory(sslsocketfactory);
connection.setDoOutput(true);
if(connection!=null){
Certificate[] certs = connection.getServerCertificates();// #1
System.out.println("Cert Type : " + certs[0].getType());
System.out.println("Cert Hash Code : " + certs[0].hashCode());
System.out.println("Cert Public Key Algorithm : " + certs[0].getPublicKey().getAlgorithm());
System.out.println("Cert Public Key Format : " + certs[0].getPublicKey().getFormat());
System.out.println("\n");
}
But I am getting the following exception.
java.lang.IllegalStateException: connection not yet open
I thought handshake should take place as soon as theurl.openconnection() method in called.
What is the problem here?
The Exception is thrown line number '#1'(see comments in the code above)
you're trying to connect to a self signed certificate host. Follow the instructions for Option 2 of the answer from this answer, which should allow you connect.
I replaced the connection.setDoOutput() with a connection.connect() and the code worked correctly for me.
Do not use this mechanism for anything other than testing - you should be using a validly signed certificate
The handshake of an SSLSocket "[...] can be initiated in one of three ways:"
calling startHandshake which explicitly begins handshakes, or
any attempt to read or write application data on this socket causes an implicit handshake, or
a call to getSession tries to set up a session if there is no currently valid session, and an implicit handshake is done.
It seems that the only one applicable via HttpsURLConnection is to try to read/write something (as you can't get hold of the underlying SSLSocket instance directly to start the handshake or get the session).
If you just call connection.getInputStream(); before trying to get the certificate, this should initiate the handshake, and you'll get the certificates afterwards.
Note that you will only reach that point if the certificate you want to see is trusted by your SSLContext/SSLSocketFactory. To do so, you can build a keystore (which you will use as a truststore) that contains that certificate, as described in this answer. You can use if from your own SSLContext or use it globally via the javax.net.ssl.trustStore* system properties (to be set before any SSL usage. Avoid examples of TrustManagers that don't check anything (there are a few here on SO): it will just disable the certificate verification altogether, making the connection potentially vulnerable to MITM attacks.
Related
How to get the details of an SSLHandshakeException? I try to get the certificates to see the cause of the exception but it throw only a NullPointerException.
URL url = new URL( "https://localhost:9443/" );
HttpsURLConnection connection = (HttpsURLConnection)url.openConnection();
try {
connection.getInputStream();
} catch( SSLHandshakeException e ) {
for( Certificate certificate : connection.getServerCertificates() ) {
System.err.println( certificate );
}
}
but it throw only a NullPointerException.
Exception in thread "main" java.lang.NullPointerException
at java.base/sun.net.www.protocol.https.HttpsClient.getServerCertificates(HttpsClient.java:695)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getServerCertificates(AbstractDelegateHttpsURLConnection.java:272)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getServerCertificates(HttpsURLConnectionImpl.java:212)
at TestSSL.main(TestSSL.java:18)
From HttpsURLConnection you cannot get any SSL/TLS-level data if the handshake fails.
If you use SSLSocket (or SSLEngine but that's much more work) I think you can get peer certificates before the handshake ends but after the peer's Certificate message has been processed and the cert chain validated. Of course in that situation any handshake failure is not caused by the certificates, so looking at the certificates is a waste of time if what you want is to 'see the cause of the exception'.
In fact in general many handshake failures (and exceptions) are not caused by certificates at all, and when they are are often that cause cannot be seen by looking at the certificates.
In general to see the cause of this exception, like many, you should look at the exception. In particular, e.getMessage() usually tells you the cause (at least the cause experienced by JSSE, which may differ from the original or 'ultimate' cause). In addition e.getCause() if nonnull can contain useful details (though often among greater clutter). And this works at both URLConnection and JSSE levels (and also the j11+ HttpClient).
Added: you can see the server's certificates at commandline with keytool -printcert -sslserver $host[:$port] -- but that's not programming and is offtopic. (Note even though the operation printcert is grammatically singular, it actually does the whole chain.)
I need to use Java to connect to a WebSocket server that is using a self-signed certificate. I'm trying to use the Jetty library and am pretty new at Java but I am finding it very difficult to figure out what needs to be done. I can connect using NodeJS very simply:
const WebSocket = require('ws');
const ws = new WebSocket('wss://192.168.100.220:9000/', ['ws-valence'], {
rejectUnauthorized: false,
});
However, modifying the example I found on the Jetty docs doesn't get me very far.
I implemented a basic client that works well with an echo test server, like in the example linked above. Then I went on to configure it with my own protocol and IP Address:
private static void connectToBasestation() {
// String destUri = "ws://echo.websocket.org";
String basestationUri = "wss://192.168.100.220:9000/";
SslContextFactory ssl = new SslContextFactory(); // ssl config
ssl.setTrustAll(true); // trust all certificates
WebSocketClient client = new WebSocketClient(ssl); // give ssl config to client
BasestationSocket socket = new BasestationSocket();
ArrayList<String> protocols = new ArrayList<String>();
protocols.add("ws-valence");
try
{
client.start();
URI bsUri = new URI(basestationUri);
ClientUpgradeRequest request = new ClientUpgradeRequest();
request.setSubProtocols(protocols);
client.connect(socket, bsUri, request);
System.out.printf("Connecting to : %s%n", bsUri);
// wait for closed socket connection.
socket.awaitClose(5,TimeUnit.SECONDS);
}
catch (Throwable t)
{
t.printStackTrace();
}
finally
{
try
{
client.stop();
}
catch (Exception e)
{
e.printStackTrace();
}
}
}
However, I'm getting an UpgradeException with 0 null as the values and my onConnect method is never getting called. I'm guessing this is a security issue, but I can't be certain since the server is an old machine -- a bit of a black box. But I'm thinking maybe something is wrong with my approach? Can anyone lend any advice here?
Edit 1: Included trustful SSL factory as suggested. It did not change anything, including the stack trace from below.
Edit 3: There is a similar question listed above, but this is different since 1) I'm getting a different error code and 2) Adding a trustful SSL factory does not solve the issue.
Edit 2: Here is the stack trace I am getting from my OnError below:
Caused by: javax.net.ssl.SSLException: Received fatal alert: handshake_failure
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634)
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800)
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.fill(SslConnection.java:681)
at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.process(HttpReceiverOverHTTP.java:128)
at org.eclipse.jetty.client.http.HttpReceiverOverHTTP.receive(HttpReceiverOverHTTP.java:73)
at org.eclipse.jetty.client.http.HttpChannelOverHTTP.receive(HttpChannelOverHTTP.java:133)
at org.eclipse.jetty.client.http.HttpConnectionOverHTTP.onFillable(HttpConnectionOverHTTP.java:155)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:291)
at org.eclipse.jetty.io.ssl.SslConnection$3.succeeded(SslConnection.java:151)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
... 3 more
org.eclipse.jetty.websocket.api.UpgradeException: 0 null
at org.eclipse.jetty.websocket.client.WebSocketUpgradeRequest.onComplete(WebSocketUpgradeRequest.java:522)
at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:216)
at org.eclipse.jetty.client.ResponseNotifier.notifyComplete(ResponseNotifier.java:208)
at org.eclipse.jetty.client.HttpReceiver.terminateResponse(HttpReceiver.java:470)
at org.eclipse.jetty.client.HttpReceiver.abort(HttpReceiver.java:552)
at org.eclipse.jetty.client.HttpChannel.abortResponse(HttpChannel.java:156)
at org.eclipse.jetty.client.HttpSender.terminateRequest(HttpSender.java:381)
at org.eclipse.jetty.client.HttpSender.abort(HttpSender.java:566)
at org.eclipse.jetty.client.HttpSender.anyToFailure(HttpSender.java:350)
at org.eclipse.jetty.client.HttpSender$CommitCallback.failed(HttpSender.java:717)
at org.eclipse.jetty.client.http.HttpSenderOverHTTP$HeadersCallback.failed(HttpSenderOverHTTP.java:310)
at org.eclipse.jetty.io.WriteFlusher$PendingState.fail(WriteFlusher.java:263)
at org.eclipse.jetty.io.WriteFlusher.onFail(WriteFlusher.java:516)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint$FailWrite.run(SslConnection.java:1251)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)
A TLS/SSL handshake error is rather generic.
You don't know what part of the TLS/SSL handshake the issue occurred in.
You can use -Djavax.net.debug=all command line option on Java to see the raw details of the TLS/SSL handshake, and this might be a good place to start troubleshooting your issues with.
Some options ...
For certificate name issues
If you connect to a server and the provided certificate does not
match the hostname you used in the URI to connect, this is a violation of
the endpoint identification algorithm present in Java itself.
Example Scenario:
You connect to wss://192.168.1.0:8443/chat
The certificate reports itself as chatserver.acme.com
This is a violation, as the hostname in the URI 192.168.1.0 does not match the certificate chatserver.acme.com
This is especially common when testing with wss://localhost or wss://127.0.0.1
You can tell Java to not perform the Endpoint Identification check like this ...
SslContextFactory.Client ssl = new SslContextFactory.Client(); // ssl config
ssl.setEndpointIdentificationAlgorithm(null); // disable endpoint identification algorithm.
WebSocketClient client = new WebSocketClient(ssl); // give ssl config to client
⚠️ WARNING: This is not recommended, and can easily allow for man-in-the-middle attacks!
For a certificate trust issues
Try enabling trust for all certificates.
Enable SSL/TLS for WebSocket Client
Trust All Certificates on the SSL/TLS Configuration
Example (assuming Jetty 9.4.19.v20190610 or newer):
SslContextFactory.Client ssl = new SslContextFactory.Client(); // ssl config
ssl.setTrustAll(true); // trust all certificates
WebSocketClient client = new WebSocketClient(ssl); // give ssl config to client
⚠️ WARNING: This is not recommended, and can easily allow for man-in-the-middle attacks!
For certificate algorithm issues
The algorithm used to create the certificate will limit the available Cipher Suites made available during the TLS/SSL handshake.
For example, If the server only had a DSA certificate (known vulnerable), then none of the RSA or ECDSA certificates would be available.
The number of bits used to create the certificate is also relevant, as if the server certificate had too few, then Java itself will reject it.
If you are in control of the server certificate, make sure you have generated a certificate that contains both a RSA and ECDSA certificate, with at least 2048 bits for RSA (or more), and 256 bits for ECDSA.
For a cipher suite issues
Try an empty Cipher Suite exclusion list on the Jetty side.
⚠️ WARNING: This lets you use KNOWN vulnerable Cipher Suites!
Enable SSL/TLS for WebSocket Client
Blank out the Cipher Suite Exclusion List
Example (assuming Jetty 9.4.19.v20190610 or newer):
SslContextFactory.Client ssl = new SslContextFactory.Client(); // ssl config
ssl.setExcludeCipherSuites(); // blank out the default excluded cipher suites
WebSocketClient client = new WebSocketClient(ssl); // give ssl config to client
⚠️ WARNING: This is not recommended, and any modern computer (even cell phones) can easily read your encrypted traffic
I am attempting to connect to an HTTPS endpoint in Java. Every method I have tried (more details below) ends up generating this stack trace:
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1138)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:753)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
I have tried:
Connecting with the javax SOAP libs and a new URL("https://...")
Connecting with new URL("https://...").openConnection()
Creating an SSL connection by hand:
Security.addProvider(new com.sun.net.ssl.internal.ssl.Provider());
SSLSocketFactory factory = (SSLSocketFactory) SSLSocketFactory.getDefault();
SSLSocket socket = (SSLSocket) factory.createSocket("...", 443);
Writer out = new OutputStreamWriter(socket.getOutputStream());
// https requires the full URL in the GET line
//
out.write("GET / HTTP/1.0\r\n");
out.write("\r\n");
out.flush();
// read response
BufferedReader in = new BufferedReader(
new InputStreamReader(socket.getInputStream()));
int c;
while ((c = in.read()) != -1) {
System.out.write(c);
}
out.close();
in.close();
socket.close();
A few more details:
Every method I have tried has worked against other SSL servers, it's this particular server (I am not at liberty to discuss what server, it's a business partner)
I can connect to this server both with a web browser, and with a faked up SOAP request with curl; This is something Java-specific.
So, it seems pretty clear that there is some disagreement between Java and the HTTPS server over how the handshake should go down, which probably means the server has some strange SSL configuration. However, I don't have direct access to the server, and the people who do are halfway around the world, so communication is a little strained due to very different timezones.
If my assumptions there are correct, what possible SSL problems could there be? What might cause something like this? Where can I ask the people in control of the server to look for issues? When I do the request with curl, I get back these server configuration headers:
Server: Apache/2.2.9 (Debian) mod_jk/1.2.26 PHP/5.2.6-1+lenny10 with Suhosin-Patch mod_ssl/2.2.9 OpenSSL/0.9.8g mod_perl/2.0.4 Perl/v5.10.0
X-Powered-By: PHP/5.2.6-1+lenny10
X-SOAP-Server: NuSOAP/0.7.3 (1.114)
It is an SSL version problem. The server only supports SSLv3, and Java will start at v2, and attempt to negotiate upwards, but not all servers support that type of negotiation.
Forcing java to use SSLv3 only is the only solution I'm aware of.
Edit, there are two ways to do this that I'm aware of:
If you are creating the socket by hand, you can set the enabled protocols
socket.setEnabledProtocols(new String[] { "SSLv3" });
If you are using a higher level library, you probably need to set all SSL requests to use v3 only, which is accomplished with the "https.protocols" system property:
java -Dhttps.protocols=SSLv3
Maybe also try setting the HTTP version to 1.1 instead of 1.0, as there's some real advantages to the newer standard.
I have Java SSL socket and c client with OpenSSL (java clients works ok with this Java server). Handshake fails and i'm getting Java exception:
javax.net.ssl.SSLHandshakeException: no cipher suites in common
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1904)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:279)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:269)
at sun.security.ssl.ServerHandshaker.chooseCipherSuite(ServerHandshaker.java:901)
at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:629)
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:167)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:901)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:837)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1023)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1332)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:889)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:154)
at java.io.BufferedReader.readLine(BufferedReader.java:317)
at java.io.BufferedReader.readLine(BufferedReader.java:382)
at EchoServer.main(EchoServer.java:36)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
Here is how server SSL socket is created:
public class EchoServer {
public static void main(String[] arstring) {
try {
final KeyStore keyStore = KeyStore.getInstance("JKS");
final InputStream is = new FileInputStream("/Path/mySrvKeystore.jks");
keyStore.load(is, "123456".toCharArray());
final KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory .getDefaultAlgorithm());
kmf.init(keyStore, "123456".toCharArray());
final TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory .getDefaultAlgorithm());
tmf.init(keyStore);
SSLContext sc = SSLContext.getInstance("TLSv1.2");
sc.init(kmf.getKeyManagers(), tmf.getTrustManagers(), new java.security.SecureRandom());
SSLServerSocketFactory sslserversocketfactory = (SSLServerSocketFactory) SSLServerSocketFactory.getDefault();
SSLServerSocket sslserversocket = (SSLServerSocket) sslserversocketfactory.createServerSocket(9997);
SSLSocket sslsocket = (SSLSocket) sslserversocket.accept();
sslsocket.setEnabledCipherSuites(sc.getServerSocketFactory().getSupportedCipherSuites());
InputStream inputstream = sslsocket.getInputStream();
InputStreamReader inputstreamreader = new InputStreamReader(inputstream);
BufferedReader bufferedreader = new BufferedReader(inputstreamreader);
String string = null;
while ((string = bufferedreader.readLine()) != null) {
System.out.println(string);
System.out.flush();
}
} catch (Exception exception) {
exception.printStackTrace();
}
}
}
C client:
BIO *certbio = NULL;
BIO *outbio = NULL;
SSL_METHOD *ssl_method;
SSL_CTX *ssl_ctx;
SSL *ssl;
int sd;
// These function calls initialize openssl for correct work.
OpenSSL_add_all_algorithms();
ERR_load_BIO_strings();
ERR_load_crypto_strings();
SSL_load_error_strings();
// Create the Input/Output BIO's.
certbio = BIO_new(BIO_s_file());
outbio = BIO_new_fp(stdout, BIO_NOCLOSE);
// initialize SSL library and register algorithms
if(SSL_library_init() < 0)
BIO_printf(outbio, "Could not initialize the OpenSSL library !\n");
ssl_method = (SSL_METHOD*)TLSv1_2_method();
// Try to create a new SSL context
if ( (ssl_ctx = SSL_CTX_new(ssl_method)) == NULL)
BIO_printf(outbio, "Unable to create a new SSL context structure.\n");
// flags
SSL_CTX_set_options(ssl_ctx, SSL_OP_NO_SSLv2 | SSL_OP_NO_SSLv3 | SSL_OP_NO_COMPRESSION | SSL_OP_CIPHER_SERVER_PREFERENCE);
// Create new SSL connection state object
ssl = SSL_new(ssl_ctx);
// Make the underlying TCP socket connection
struct sockaddr_in address;
memset(&address, 0, sizeof(address));
address.sin_family = AF_INET;
address.sin_port = htons(port);
const char *dest_url = this->host.c_str();
address.sin_addr.s_addr = inet_addr(dest_url);
address.sin_port = htons(port);
sd = socket(AF_INET, SOCK_STREAM, 0);
int connect_result = ::connect(sd, (struct sockaddr*)&address, sizeof(address));
if (connect_result != 0) {
BIO_printf(outbio, "Failed to connect over TCP with error %i\n", connect_result);
throw IOException("Connection refused");
} else {
BIO_printf(outbio, "Successfully made the TCP connection to: %s:%i\n", dest_url, port);
}
// Attach the SSL session to the socket descriptor
SSL_set_fd(ssl, sd);
// Try to SSL-connect here, returns 1 for success
int ssl_connect_result = SSL_connect(ssl);
if (ssl_connect_result != 1)
BIO_printf(outbio, "Error: Could not build a SSL session to: %s:%i with error %i\n", dest_url, port, ssl_connect_result);
else
BIO_printf(outbio, "Successfully enabled SSL/TLS session to: %s\n", dest_url);
Here is output on client side:
Error: Could not build a SSL session to: 127.0.0.1:9997 with error -1
Update 1
int ssl_connect_result = SSL_connect(ssl);
if (ssl_connect_result != 1) {
int error_code = SSL_get_error(ssl, ssl_connect_result); // =1
BIO_printf(outbio, "Error: Could not build a SSL session to: %s:%i with error %i (%i)\n", dest_url, port, ssl_connect_result, error_code);
} else {
BIO_printf(outbio, "Successfully enabled SSL/TLS session to: %s\n", dest_url);
}
And the output is:
Error: Could not build a SSL session to: 127.0.0.1:9997 with error -1 (1)
Update 2
I forgot to note that I'm using self-signed certificate, generated by keytool from JDK.
Update 3
I've noted i missed some lines and I've added:
OpenSSL_add_all_ciphers();
OpenSSL_add_all_digests();
but still no luck - getting the same error -1.
Update 4
Here is Java client which is accepted by the server code above:
SSLSocketFactory sslsocketfactory = (SSLSocketFactory) SSLSocketFactory.getDefault();
SSLSocket sslsocket = (SSLSocket) sslsocketfactory.createSocket(ip, port);
sslsocket.setEnabledCipherSuites(sslsocketfactory.getSupportedCipherSuites());
InputStream inputstream = System.in;
InputStreamReader inputstreamreader = new InputStreamReader(inputstream);
BufferedReader bufferedreader = new BufferedReader(inputstreamreader);
OutputStream outputstream = sslsocket.getOutputStream();
OutputStreamWriter outputstreamwriter = new OutputStreamWriter(outputstream);
String string = null;
outputstreamwriter.write("hello");
outputstreamwriter.flush();
while ((string = bufferedreader.readLine()) != null) {
outputstreamwriter.write(string);
outputstreamwriter.flush();
}
sslsocket.close();
I've checked that I can't see plain data in packets intercepted in the network so it does perform some data encryption.
I don't believe that Java server accepts a Java client either, unless the Java client similarly does .setEnabledCipherSuites (all-supported) -- if so it is using an anonymous (unauthenticated) ciphersuite that is not secure against active attack, and although many people are still stuck in the passive-only threat model of about 1980, today active attacks are common. That is why JSSE's default cipherlist excludes anonymous ciphers -- unless you override it. And why OpenSSL's default cipherlist also excludes them -- which you didn't override.
(add) To explain in smaller words, anonymous ciphersuites ARE encrypted (with a few exceptions not relevant here because they are never preferred) but NOT authenticated. The word "unauthenticated" means "not authenticated"; it does not mean "not encrypted". The word "unencrypted" is used to mean "not encrypted". "Not encrypted" means something that just looks at the channel, like Wireshark, can see the plaintext. "Not authenticated" means an attacker who intercepts (possibly diverts) your traffic can cause you to establish your "secure" session with the attacker in the middle, and they can decrypt your data, copy and/or change it as they wish, re-encrypt it, and send it on, and you will think it is correct and private when it isn't. Google or search here (I think mostly security.SE and superuser) for things like "man in the middle attack", "ARP spoof", "MAC spoof", "DNS poisoning", "BGP attack", etc.
The immediate problem is you aren't using the keystore. You create an SSLContext with key and trust managers from it, but then you create the socket from SSLServerSocketFactory.getDefault() which doesn't use the context. Use sc.getServerSocketFactory() instead.
(add) why? Every SSLSocket (and SSLServerSocket) is linked to an SSLContext which among other things controls the privatekey(s) and certificate(s) or chain(s) used and certificates trusted. (SSL/TLS connections normally authenticate the server only, so in practice the server only needs a key-and-chain, and the client only needs the root cert, but Java uses the same keystore file format for both and it's easy to just code both.) Since your code has set the particular SSLContext sc to contain a suitable key-and-cert, sc.getServerSocketFactory() creates a factory which creates an SSLServerSocket which in turn creates an SSLSocket (for each connection if more than one) which uses that key-and-(as long as the client's supported cipher list allows it, and here it does).
(add) SSLServerSocketFactory.getDefault() creates a factory, and thus sockets, using the default SSL context, which by default contains NO key-and-chain, although you can change this with system properties as described in the documentation cleverly hidden at http://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html . As a result it cannot negotiate a ciphersuite that is authenticated. Since both Java and OpenSSL by default disable unauthenticated ciphersuites, this is why you get "no cipher suites in common" unless you .setEnabled to include the unauthenticated and insecure ciphersuites in Java, and still get it for OpenSSL client since you didn't do anything to enable unauthenticated and insecure ciphersuites there.
(add) If you look carefully at your Wireshark trace, you will see in the ServerHello that the selected ciphersuite uses DH_anon or ECDH_anon key exchange -- "anon" is an abbreviation for "anonymous" which means "not authenticated" as explained above -- and there is no Certificate message from the server, and (less obvious unless you know it) the ServerKeyExchange data is not signed. Also I predict if you have your Java client check sslsocket.getSession().getCipherSuite() and/or sslsocket.getSession().getPeerCertificates() after the handshake is done, which since you don't do it explicitly will be on the first socket-level I/O which will be the outputstreamwriter.flush(), you will see the anonymous ciphersuite, and no peer cert (it throws PeerNotAuthenticated).
Other points:
(1) In general whenever you get SSL_ERROR from SSL_get_error(), or any error return from lower level routines like EVP_* and BIO_*, you should use the ERR_* routines to get details of the error and log/display them; see https://www.openssl.org/docs/faq.html#PROG6 and https://www.openssl.org/docs/manmaster/crypto/ERR_print_errors.html et amici. Especially since you HAVE loaded the error strings, thus avoiding https://www.openssl.org/docs/faq.html#PROG7 . In this case however you already know enough from the server side, so client side details aren't needed.
(2) You don't need _add_all_ciphers and _add_all_digests, they are included in _add_all_algorithms.
(3) OP_NO_SSLv2 and 3 have no effect on TLSv1_2_method and OP_SERVER_CIPHER_PREFERENCE has no effect on a client. (They do no harm, they are just useless and possibly confusing.)
(4) Once you get past the cipher negotiation, the OpenSSL client will need the root cert for the server; since you intend to use a self-signed cert (once you fix the server to use the keystore at all) that cert is its own root. In 1.0.2 (not earlier) you could also use a non-root trust anchor, but not by default and you don't have one anyway. I assume certbio was intended for this, but you never open it on an actual file or do anything else with it, and anyway the SSL library cannot use a BIO for its truststore. You have three choices:
put the cert(s) in a file, or a directory using special hash names, and pass the file and/or directory name(s) to SSL_CTX_load_verify_locations. If you only want one root (your own) using the CAfile option is easier.
put or add the cert(s) to the default file or hashed directory determined by your OpenSSL library compilation and call SSL_CTX_set_default_verify_paths; this is commonly something like /etc/pki or /var/ssl. If you want to use the same cert(s) for multiple programs or for commandline openssl this shared location is usually easier.
use the BIO and/or other means to (open and) read the cert(s) into memory, build your own X509_STORE containing them, and put that in your SSL_CTX. This is more complicated, so I won't expand on it unless you want to.
(5) Your dest_url is (at least in this case?) an address string, not a URL; those are different though related things and thinking they are the same will cause you lots more problems. For most programs it is better to handle a host name string with classic gethostbyname and fall back to inet_addr, or better the "new" (since 1990s) getaddrinfo which can handle both name and address strings and both IPv4 and v6 (also new since 1990s but finally gaining traction). At the very least you should check for inet_addr returning INADDR_NONE indicating it failed.
SSLContext sc = SSLContext.getInstance("TLSv1.2");
Java used to do two thing with code similar to this:
enable SSLv3
disable TLS 1.1 and 1.2
... Even though you called out TLS. Effectively, all you could get was a SSLContext with SSLv3 and TLS 1.0 (unless you were willing to do more work). It may not be the case anymore, but it would explain the error you are seeing (especially if your are using Java 7 or 8).
You need to do more work in Java to get "TLS 1.0 or above" and "just TLS 1.2". For that, see Which Cipher Suites to enable for SSL Socket?. It shows you how to enable/disable both protocols and cipher suites.
SSL_CTX_set_options(ssl_ctx, SSL_OP_NO_SSLv2 | SSL_OP_NO_SSLv3 | SSL_OP_NO_COMPRESSION | SSL_OP_CIPHER_SERVER_PREFERENCE);
You should also set a cipher suite list since OpenSSL includes broken/weak/wounded ciphers by default. Something like:
const char PREFERRED_CIPHERS[] = "HIGH:!aNULL:!kRSA:!PSK:!SRP:!MD5:!RC4";
long res = SSL_CTX_set_cipher_list(ssl_ctx, PREFERRED_CIPHERS);
ASSERT(res == 1);
I think your final step is to ensure the self-signed certificate is trusted.
For a Java client, see How to properly import a selfsigned certificate into Java keystore that is available to all Java applications by default?. For an OpenSSL client, just load it up using SSL_CTX_load_verify_locations, see SSL_CTX_load_verify_locations Fails with SSL_ERROR_NONE.
Note well: OpenSSL prior to 1.1.0 did not perform hostname validation. You will have to do that yourself. If you need to perform hostname validation, then lift the code from cURL's url.c. I seem to recall looking at Daniel's code, and its very solid. You can also lift the code from Google's CT project. I never audited it, so I don't know what its like.
There are two other checks you have to make; they are discussed at SSL/TLS Client on the OpenSSL wiki.
There is some code in our system for automatically generating self-signed certificates into a key store which is then used by Jetty. If a key for a given host already exists then nothing happens but if it doesn't exist, we generate a new key, like this:
public void generateKey(String commonName) {
X500Name x500Name = new X500Name("CN=" + commonName);
CertAndKeyGen keyPair = new CertAndKeyGen("DSA", "SHA1withDSA");
keyPair.generate(1024);
PrivateKey privateKey = keyPair.getPrivateKey();
X509Certificate certificate = keyPair.getSelfCertificate(x500Name, 20*365*24*60*60);
Certificate[] chain = { certificate };
keyStore.setEntry(commonName, privateKey, "secret".toCharArray(), chain);
}
This all works fine as long as there is only one key and certificate in the key store. Once you have multiple keys, weird things happen when you try to connect:
java.io.IOException: HTTPS hostname wrong: should be <127.0.0.1>
This was quite a mystifying error but I finally managed to track it down by writing a unit test which connects to the server and asserts that the CN on the certificate matches the hostname. What I found was quite interesting - Jetty seems to arbitrarily choose which certificate to present to the client, but in a consistent fashion.
For instance:
If "CN=localhost" and "CN=cheese.mydomain" are in the key store, it always chose "CN=cheese.mydomain".
If "CN=127.0.0.1" and "CN=cheese.mydomain" are in the key store, it always chose "CN=cheese.mydomain".
If "CN=192.168.222.100" (cheese.mydomain) and "CN=cheese.mydomain" are in the key store, it always chose "CN=192.168.222.100".
I wrote some code which loops through the certificates in the store to print them out and found that it isn't consistently choosing the first certificate or anything trivial like that.
So exactly what criteria does it use? Initially I thought that localhost was special but then the third example baffled me completely.
I take it that this is somehow decided by the KeyManagerFactory, which is SunX509 in my case.
This is indeed ultimately decided by the KeyManager (generally obtained from a KeyManagerFactory).
A keystore can have a number of certificates stored under different aliases. If no alias is explicitly configured via certAlias in the Jetty configuration, the SunX509 implementation will pick the first aliases it finds for which there is a private key and a key of the right type for the chosen cipher suite (typically RSA, but probably DSA in your case here). There's a bit more to it to the choice logic, if you look at the Sun provider implementation, but you shouldn't really rely on the order in general, just the alias name.
You can of course give Jetty your own SSLContext with your own X509KeyManager to choose the alias. You would have to implement:
chooseServerAlias(String keyType, Principal[] issuers, Socket socket)
Unfortunately, apart from keyType and issuers, all you get to make the decision is the socket itself. At best, the useful information you get there are the local IP address and the remote one.
Unless your server is listening to multiple IP addresses on the same port, you will always get the same local IP address. (Here, obviously, you have at least two: 127.0.0.1 and 192.168.222.100, but I suspect you're not really interested in localhost except for your own tests.) You would need Server Name Indication (SNI) support on the server side to be able to make a decision based on the requested host names (by clients that support it). Unfortunately, SNI was only introduced in Java 7, but only on the client side.
Another problem you will face here is that Java clients will complain about IP addresses in the Subject DN's CN. Some browsers would tolerate this, but this is not compliant with the HTTPS specification (RFC 2818). IP addresses must be Subject Alternative Name entries of IP-address type.