RabbitMQ SSL handshake connection error on any connection attempt with certificate authentication
I am currently trying to migrate some Java components so that the RabbitMQ connection is authenticated through SSL client certificates instead of the PLAIN method, but after a few days I am still struggling with it since all attempts to connect Java components are done with a handshake. A look at RabbitMQ: Handshaking error when trying to use SSL or RabbitMQ SSL certificates giving handshake when using SpringAMQP unfortunately did not give me any results.
The environment I'm trying to make this work is a "legacy" virtual VM running Ubuntu LTS 14.04, which is pretty much the environment I want to deploy to.
The rabbitmqctl report output looks like this:
Status of node 'rabbit@developer-VirtualBox' ...
[{pid,23352},
{running_applications,
[{rabbitmq_management,"RabbitMQ Management Console","3.4.2"},
{rabbitmq_web_dispatch,"RabbitMQ Web Dispatcher","3.4.2"},
{webmachine,"webmachine","1.10.3-rmq3.4.2-gite9359c7"},
{mochiweb,"MochiMedia Web Server","2.7.0-rmq3.4.2-git680dba8"},
{rabbitmq_management_agent,"RabbitMQ Management Agent","3.4.2"},
{rabbit,"RabbitMQ","3.4.2"},
{ssl,"Erlang/OTP SSL application","5.3.2"},
{public_key,"Public key infrastructure","0.21"},
{crypto,"CRYPTO version 2","3.2"},
{asn1,"The Erlang ASN1 compiler version 2.0.4","2.0.4"},
{os_mon,"CPO CXC 138 46","2.2.14"},
{inets,"INETS CXC 138 49","5.9.7"},
{rabbitmq_auth_mechanism_ssl,
"RabbitMQ SSL authentication (SASL EXTERNAL)","3.4.2"},
{amqp_client,"RabbitMQ AMQP Client","3.4.2"},
{xmerl,"XML parser","1.3.5"},
{mnesia,"MNESIA CXC 138 12","4.11"},
{sasl,"SASL CXC 138 11","2.3.4"},
{stdlib,"ERTS CXC 138 10","1.19.4"},
{kernel,"ERTS CXC 138 10","2.16.4"}]},
{os,{unix,linux}},
{erlang_version,
"Erlang R16B03 (erts-5.10.4) [source] [64-bit] [async-threads:30] [kernel-poll:true]\n"},
{memory,
[{total,42662672},
{connection_readers,0},
{connection_writers,0},
{connection_channels,0},
{connection_other,5264},
{queue_procs,2632},
{queue_slave_procs,0},
{plugins,411368},
{other_proc,14374616},
{mnesia,59360},
{mgmt_db,124224},
{msg_index,34312},
{other_ets,1135040},
{binary,42680},
{code,21795549},
{atom,793505},
{other_system,3884122}]},
{alarms,[]},
{listeners,[{clustering,25672,"::"},{amqp,5672,"::"},{'amqp/ssl',5671,"::"}]},
{vm_memory_high_watermark,0.4},
{vm_memory_limit,1658211532},
{disk_free_limit,50000000},
{disk_free,43703377920},
{file_descriptors,
[{total_limit,924},{total_used,4},{sockets_limit,829},{sockets_used,2}]},
{processes,[{limit,1048576},{used,191}]},
{run_queue,0},
{uptime,12}]
Cluster status of node 'rabbit@developer-VirtualBox' ...
[{nodes,[{disc,['rabbit@developer-VirtualBox']}]},
{running_nodes,['rabbit@developer-VirtualBox']},
{cluster_name,<<"rabbit@developer-VirtualBox">>},
{partitions,[]}]
Application environment of node 'rabbit@developer-VirtualBox' ...
[{amqp_client,[{prefer_ipv6,false},{ssl_options,[]}]},
{asn1,[]},
{crypto,[]},
{inets,[]},
{kernel,
[{error_logger,tty},
{inet_default_connect_options,[{nodelay,true}]},
{inet_dist_listen_max,25672},
{inet_dist_listen_min,25672}]},
{mnesia,[{dir,"/var/lib/rabbitmq/mnesia/rabbit@developer-VirtualBox"}]},
{mochiweb,[]},
{os_mon,
[{start_cpu_sup,false},
{start_disksup,false},
{start_memsup,false},
{start_os_sup,false}]},
{public_key,[]},
{rabbit,
[{auth_backends,[rabbit_auth_backend_internal]},
{auth_mechanisms,['EXTERNAL']},
{backing_queue_module,rabbit_variable_queue},
{channel_max,0},
{cluster_keepalive_interval,10000},
{cluster_nodes,{[],disc}},
{cluster_partition_handling,ignore},
{collect_statistics,fine},
{collect_statistics_interval,5000},
{default_permissions,[<<".*">>,<<".*">>,<<".*">>]},
{default_user,<<"guest">>},
{default_user_tags,[administrator]},
{default_vhost,<<"/">>},
{delegate_count,16},
{disk_free_limit,50000000},
{enabled_plugins_file,"/etc/rabbitmq/enabled_plugins"},
{error_logger,
{file,"/var/log/rabbitmq/rabbit@developer-VirtualBox.log"}},
{frame_max,131072},
{halt_on_upgrade_failure,true},
{handshake_timeout,10000},
{heartbeat,580},
{hipe_compile,false},
{hipe_modules,
[rabbit_reader,rabbit_channel,gen_server2,rabbit_exchange,
rabbit_command_assembler,rabbit_framing_amqp_0_9_1,rabbit_basic,
rabbit_event,lists,queue,priority_queue,rabbit_router,rabbit_trace,
rabbit_misc,rabbit_binary_parser,rabbit_exchange_type_direct,
rabbit_guid,rabbit_net,rabbit_amqqueue_process,
rabbit_variable_queue,rabbit_binary_generator,rabbit_writer,
delegate,gb_sets,lqueue,sets,orddict,rabbit_amqqueue,
rabbit_limiter,gb_trees,rabbit_queue_index,
rabbit_exchange_decorator,gen,dict,ordsets,file_handle_cache,
rabbit_msg_store,array,rabbit_msg_store_ets_index,rabbit_msg_file,
rabbit_exchange_type_fanout,rabbit_exchange_type_topic,mnesia,
mnesia_lib,rpc,mnesia_tm,qlc,sofs,proplists,credit_flow,pmon,
ssl_connection,tls_connection,ssl_record,tls_record,gen_fsm,ssl]},
{log_levels,[{connection,info}]},
{loopback_users,[<<"guest">>]},
{mnesia_table_loading_timeout,30000},
{msg_store_file_size_limit,16777216},
{msg_store_index_module,rabbit_msg_store_ets_index},
{plugins_dir,
"/usr/lib/rabbitmq/lib/rabbitmq_server-3.4.2/sbin/../plugins"},
{plugins_expand_dir,
"/var/lib/rabbitmq/mnesia/rabbit@developer-VirtualBox-plugins-expand"},
{queue_index_max_journal_entries,65536},
{reverse_dns_lookups,false},
{sasl_error_logger,
{file,"/var/log/rabbitmq/rabbit@developer-VirtualBox-sasl.log"}},
{server_properties,[]},
{ssl_allow_poodle_attack,false},
{ssl_apps,[asn1,crypto,public_key,ssl]},
{ssl_cert_login_from,distinguished_name},
{ssl_handshake_timeout,5000},
{ssl_listeners,[5671]},
{ssl_options,
[{cacertfile,"/home/developer/rabbitmqcert/devcafiles/cacert.pem"},
{certfile,"/home/developer/rabbitmqcert/rabbitmq.public.pem"},
{keyfile,"/home/developer/rabbitmqcert/rabbitmq.private.pem"},
{verify,verify_peer},
{ssl_cert_login_from,organization},
{fail_if_no_peer_cert,true}]},
{tcp_listen_options,
[binary,
{packet,raw},
{reuseaddr,true},
{backlog,128},
{nodelay,true},
{linger,{true,0}},
{exit_on_close,false}]},
{tcp_listeners,[5672]},
{trace_vhosts,[]},
{vm_memory_high_watermark,0.4},
{vm_memory_high_watermark_paging_ratio,0.5}]},
{rabbitmq_auth_mechanism_ssl,[{name_from,distinguished_name}]},
{rabbitmq_management,
[{http_log_dir,none},
{listener,[{port,15672}]},
{load_definitions,none},
{rates_mode,basic},
{sample_retention_policies,
[{global,[{605,5},{3660,60},{29400,600},{86400,1800}]},
{basic,[{605,5},{3600,60}]},
{detailed,[{10,5}]}]}]},
{rabbitmq_management_agent,[]},
{rabbitmq_web_dispatch,[]},
{sasl,[{errlog_type,error},{sasl_error_logger,false}]},
{ssl,
[{protocol_version,['tlsv1.2','tlsv1.1',tlsv1,sslv3]},
{versions,['tlsv1.2','tlsv1.1']}]},
{stdlib,[]},
{webmachine,[{error_handler,rabbit_webmachine_error_handler}]},
{xmerl,[]}]
Connections:
Channels:
Queues on /:
Exchanges on /:
name type durable auto_delete internal arguments policy
direct true false false []
amq.direct direct true false false []
amq.fanout fanout true false false []
amq.headers headers true false false []
amq.match headers true false false []
amq.rabbitmq.log topic true false true []
amq.rabbitmq.trace topic true false true []
amq.topic topic true false false []
Bindings on /:
Consumers on /:
Permissions on /:
user configure write read
O=dev,CN=rules .* .* .*
guest .* .* .*
Policies on /:
Parameters on /:
After going through it, I see that:
- _Rabbitmq_auth_mechanism_ssl_ plugin is active
- I am listening for SSL connections on port 5671
- The only accepted authorization mechanism is EXTERNAL
- I only accept TLS v1.1 and TLS v1.2
- There is a user _O = dev, CN = rules_ corresponding to the subject in the client certificate. This user does not have a password associated with it (cleared it with _sudo rabbitmqctl clear_password "O = dev, CN = rules" _ and the control screen reflects this)
- The CA certificate file is read from _ / home / developer / rabbitmqcert / devcafiles / cacert.pem_. This corresponds to a CA built for dev purposes, with all public keys (both server and client) used when subscribing to it
For simplicity (and sanity), I tried to connect to the RabbitMQ instance using the following Java code:
package com.rabbitmq.sample;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.security.KeyManagementException;
import java.security.KeyStore;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.security.UnrecoverableKeyException;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import javax.net.ssl.KeyManager;
import javax.net.ssl.KeyManagerFactory;
import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.DefaultSaslConfig;
import com.rabbitmq.client.QueueingConsumer;
public class CertificateAuthenticatedRabbitMQClientExample {
private static final String CLIENT_CERTIFICATE_PASSWORD = "MySecretPassword";
private static final String QUEUE_USED = "sampleQueue";
public static void main(String[] args) throws UnrecoverableKeyException, NoSuchAlgorithmException, KeyStoreException, IOException, CertificateException, KeyManagementException, InterruptedException {
SSLContext sslContext = SSLContext.getInstance("TLSv1.2");
KeyManager[] clientKeyManagerList = null;
try(FileInputStream clientCertificateInputStream = new FileInputStream(new File("/home/developer/rabbitmqcert/ruleprocessing.password.p12"))) {
KeyStore clientKeStore = KeyStore.getInstance("PKCS12"); //Create a clean KeyStore
clientKeStore.load(clientCertificateInputStream, CLIENT_CERTIFICATE_PASSWORD.toCharArray()); //Load the client certificate into the keystore
KeyManagerFactory clientSSLKeyManagerFactory = KeyManagerFactory.getInstance("SunX509");
clientSSLKeyManagerFactory.init(clientKeStore, CLIENT_CERTIFICATE_PASSWORD.toCharArray());
clientKeyManagerList = clientSSLKeyManagerFactory.getKeyManagers(); //Get list of key managers (in essence, only the keystore with the client certificate)
}
TrustManager[] clientTrustManagerList = {
new X509TrustManager() {
//Dummy trust store that trusts any server you connect to.
//For demo purposes only
@Override
public void checkClientTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {}
@Override
public void checkServerTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {}
@Override
public X509Certificate[] getAcceptedIssuers() { return null;}
}
};
sslContext.init(clientKeyManagerList, clientTrustManagerList, null); //Initialize SSL context with the key and trust managers we've created/loaded before
ConnectionFactory rabbitMqConnectionFactory = new ConnectionFactory(); //Create factory
rabbitMqConnectionFactory.setHost("localhost");
rabbitMqConnectionFactory.setPort(5671);
rabbitMqConnectionFactory.setSaslConfig(DefaultSaslConfig.EXTERNAL); //Set authentication method as SSL auth
rabbitMqConnectionFactory.useSslProtocol(sslContext); //Set the created SSL context as the one to use
Connection rabbitMqOutboundConnection = null;
Channel rabbitMqOutboundChannel = null;
try {
rabbitMqOutboundConnection = rabbitMqConnectionFactory.newConnection();
rabbitMqOutboundChannel = rabbitMqOutboundConnection.createChannel();
rabbitMqOutboundChannel.queueDeclare(QUEUE_USED, false, false, false, null);
rabbitMqOutboundChannel.basicPublish("", QUEUE_USED, null, "This is a sample message".getBytes());
System.out.println("Message successfully sent to queue");
}finally{
if(rabbitMqOutboundChannel != null) {
rabbitMqOutboundChannel.close();
}
if(rabbitMqOutboundConnection != null) {
rabbitMqOutboundConnection.close();
}
}
Connection rabbitMqInboundConnection = null;
Channel rabbitMqInboundChannel = null;
try {
rabbitMqInboundConnection = rabbitMqConnectionFactory.newConnection();
rabbitMqInboundChannel = rabbitMqInboundConnection.createChannel();
rabbitMqInboundChannel.queueDeclare(QUEUE_USED, false, false, false, null);
QueueingConsumer rabbitMqQueueConsumer = new QueueingConsumer(rabbitMqInboundChannel);
rabbitMqInboundChannel.basicConsume(QUEUE_USED, true, rabbitMqQueueConsumer);
QueueingConsumer.Delivery deliveryResult = rabbitMqQueueConsumer.nextDelivery();
System.out.println("Message read from the queue: " + new String(deliveryResult.getBody()));
}finally{
if(rabbitMqInboundChannel != null) {
rabbitMqInboundChannel.close();
}
if(rabbitMqInboundConnection != null) {
rabbitMqInboundConnection.close();
}
}
}
}
However, I am always greeted with an exception on the "main" thread java.net.SocketException: Broken pipe error on execution and looking at the RabbitMQ server log I see:
=INFO REPORT==== 29-Dec-2014::15:55:30 ===
accepting AMQP connection <0.644.0> (127.0.0.1:35299 -> 127.0.0.1:5671)
=ERROR REPORT==== 29-Dec-2014::15:55:30 ===
SSL: certify: ssl_handshake.erl:1343:Fatal error: handshake failure
Based on what I've seen on the internet and here I tried to change the value of the check to verify_none, but in doing so I get this:
=INFO REPORT==== 29-Dec-2014::14:51:13 ===
accepting AMQP connection <0.311.0> (127.0.0.1:35271 -> 127.0.0.1:5671)
=ERROR REPORT==== 29-Dec-2014::14:51:17 ===
closing AMQP connection <0.311.0> (127.0.0.1:35271 -> 127.0.0.1:5671):
{handshake_error,starting,0,
{amqp_error,access_refused,
"EXTERNAL login refused: no peer certificate",
'connection.start_ok'}}
Following the guidelines in the SSL Troubleshooting section , I tried to connect s_client to the server again, results depending on the validation value. If the verify parameter is set to verify_peer, I get the following:
developer@developer-VirtualBox:~/rabbitmqcert$ openssl s_client -tls1_2 -connect localhost:5671 -cert ruleprocessing.public.pem -key ruleprocessing.private.pem -CAfile devcafiles/cacert.pem
CONNECTED(00000003)
depth=1 CN = RabbitMQCA
verify return:1
depth=0 CN = rabbitmq, O = dev
verify return:1
139797055162016:error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure:s3_pkt.c:1260:SSL alert number 40
139797055162016:error:1409E0E5:SSL routines:SSL3_WRITE_BYTES:ssl handshake failure:s3_pkt.c:596:
---
Certificate chain
0 s:/CN=rabbitmq/O=dev
i:/CN=RabbitMQCA
1 s:/CN=RabbitMQCA
i:/CN=RabbitMQCA
---
Server certificate
-----BEGIN CERTIFICATE-----
<<OMMITED>>
-----END CERTIFICATE-----
subject=/CN=rabbitmq/O=dev
issuer=/CN=RabbitMQCA
---
Acceptable client certificate CA names
/CN=RabbitMQCA
---
SSL handshake has read 1646 bytes and written 2103 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : AES256-SHA256
Session-ID: FC972B9A5D3EC359DC0467C8F02410E3AD66DA151C4411C0D5892115A439431A
Session-ID-ctx:
Master-Key: E4FE793C71692852F6F3C4E9C5CB17774D8A50511338EF2E75691DC0DC2119F56611FC959C12429BBAFD46EC760ED713
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1419871438
Timeout : 7200 (sec)
Verify return code: 0 (ok)
---
As you can see, the handshake error occurs at the beginning, but then seems to recover.
If the verification is set to verify_none, I get this:
CONNECTED(00000003)
depth=1 CN = RabbitMQCA
verify return:1
depth=0 CN = rabbitmq, O = dev
verify return:1
---
Certificate chain
0 s:/CN=rabbitmq/O=dev
i:/CN=RabbitMQCA
1 s:/CN=RabbitMQCA
i:/CN=RabbitMQCA
---
Server certificate
-----BEGIN CERTIFICATE-----
<<OMMITED>>
-----END CERTIFICATE-----
subject=/CN=rabbitmq/O=dev
issuer=/CN=RabbitMQCA
---
No client certificate CA names sent
---
SSL handshake has read 1666 bytes and written 663 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : AES256-SHA256
Session-ID: C4156551790BA116DC38A981728A71768D0B53AAEBEE969A4DA150746E5373FB
Session-ID-ctx:
Master-Key: 3470DD0C0247B94EA784C3CEF94888C160205E9F06C14869B564A00AF5E4F7FAF5B4FC977E290B80DBCD140133F75AC0
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1419871570
Timeout : 7200 (sec)
Verify return code: 0 (ok)
---
This time, the handshake error does not occur at the beginning.
As a side note, since I'm not very sure about the "clean" VM I got, I actually tried to create a new VM (same Ubuntu version), install RabbitMQ (same version), set it up pretty much like this same (the only thing that has changed is the location for the certificates) and runs the same client code (the certificate path has been changed). The end result was successful. Unfortunately, there is not much I can do to get this VM to be used as a VM at this time.
TL; DR; After configuring RabbitMQ server running in Ubuntu VM to accept SSL connections from Java client using certificate authentication, all connection attempts fail with handshake error as reason if verify = verify_peer or EXTERNAL login refused: no peer certificate if verify = verify_none
+3
source to share