Override DNS for specific domains, like the hosts file, but without using the Hosts file
I need to fire a series of parallel web requests from the same server to a specific domain, but control which IP address these requests actually go to. I originally came up with a scheme where I would request the required IP address and then manually set the header Host: www.example.com
in the request and use a series of handlers to make sure the issued redirects match the same pattern.
This seemed to work for a while, but lately I've been having problems redirecting to HTTPS. The handshake will fail and the request is in turn. I have tried to disable SSL verification in various ways, including:
local $ENV{ PERL_LWP_SSL_VERIFY_HOSTNAME } = 0;
local $ENV{ HTTPS_DEBUG } = 1;
$ua->ssl_opts(
SSL_ca_file => Mozilla::CA::SSL_ca_file(),
verify_hostname => 0,
SSL_verify_mode => 0x00,
);
IO::Socket::SSL::set_ctx_defaults(
SSL_verifycn_scheme => 'www',
SSL_verify_mode => 0,
);
I also tried using LWP :: UserAgent :: DNS :: Hosts to fix the problem, but it persists.
<edit> I should note that the reason disabling peer authentication for SSL does not solve the problem, most likely because for some reason requesting this way actually causes the handshake to fail rather than fail verification. dot </edit>
One thing that works is to write in /etc/hosts
to point to the domain at the corresponding IP address, however this is impractical because I might need to run dozens or hundreds of tests in parallel, on the same domain.
Is there a way to emulate the functionality of adding an entry to /etc/hosts
that does not require a special IP request and Host: ...
HTTP header override ?
EDIT: SSL debug info
DEBUG: .../IO/Socket/SSL.pm:1914: new ctx 140288835318480
DEBUG: .../IO/Socket/SSL.pm:402: socket not yet connected
DEBUG: .../IO/Socket/SSL.pm:404: socket connected
DEBUG: .../IO/Socket/SSL.pm:422: ssl handshake not started
DEBUG: .../IO/Socket/SSL.pm:455: not using SNI because hostname is unknown
DEBUG: .../IO/Socket/SSL.pm:478: set socket to non-blocking to enforce timeout=180
DEBUG: .../IO/Socket/SSL.pm:491: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:501: ssl handshake in progress
DEBUG: .../IO/Socket/SSL.pm:511: waiting for fd to become ready: SSL wants a read first
DEBUG: .../IO/Socket/SSL.pm:531: socket ready, retrying connect
DEBUG: .../IO/Socket/SSL.pm:491: Net::SSLeay::connect -> -1
DEBUG: .../IO/Socket/SSL.pm:1388: SSL connect attempt failed with unknown error
DEBUG: .../IO/Socket/SSL.pm:497: fatal SSL error: SSL connect attempt failed with unknown error error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
DEBUG: .../IO/Socket/SSL.pm:1948: free ctx 140288835318480 open=140288835318480
DEBUG: .../IO/Socket/SSL.pm:1953: free ctx 140288835318480 callback
DEBUG: .../IO/Socket/SSL.pm:1956: OK free ctx 140288835318480
And in response I get:
Can't connect to redacted.org:443
SSL connect attempt failed with unknown error error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure at /System/Library/Perl/Extras/5.18/LWP/Protocol/http.pm line 51.
This doesn't work that well on our server (using an old legacy version of Perl, which I won't cover here as it seems out of place).
The server initially responds to a non-HTTPS request with a 301 redirect to the HTTPS site. Then it crashes. I'll post a replay code with the specific details of my request, but any site that redirects non-HTTPS traffic to HTTPS should suffice.
use IO::Socket::SSL qw/ debug4 /;
use LWP::UserAgent;
use LWP::UserAgent::DNS::Hosts;
use HTTP::Request;
use Mozilla::CA;
use Data::Dumper;
LWP::UserAgent::DNS::Hosts->register_hosts(
'recacted.org' => '127.0.0.1', # no I am not redirecting to loopback in reality, this is anonymized
'www.redacted.org' => '127.0.0.1',
);
LWP::UserAgent::DNS::Hosts->enable_override;
my $ua = LWP::UserAgent->new;
$ua->ssl_opts( SSL_ca_file => Mozilla::CA::SSL_ca_file() );
my $request = HTTP::Request->new(GET => 'http://redacted.org/');
my $response = $ua->request($request);
print $response->content; #Dumper ( $response->is_success ? $response->headers : $response );
Again, this is not production code, enough code to reproduce the problem. It looks like it has nothing to do with SSL validation, but also the inability to negotiate the request, presumably because it LWP::UserAgent::DNS::Hosts
does exactly what I am doing: changing the target target to the correct IP address and then Host: ...
manually recording . Why this is causing the SSL connection to fail I don't know.
When debugging a local machine
openssl version -a: 1.0.2j 26 Sep 2016
IO::Socket::SSL->VERSION == 1.966
Net::SSLeay->VERSION == 1.72
On our server
openssl version -a: 1.0.1t 3 May 2016
IO::Socket::SSL->VERSION == 1.76
Net::SSLeay->VERSION == 1.48
source to share
Considering that it works with an explicit file /etc/hosts
, but not replacement PeerAddr
or use LWP::UserAgent::DNS::Hosts
, it looks like a problem with the SNI extension. This TLS extension is used to provide the TLS server with the requested hostname (similar to the HTTP header Host
) so that it can select the appropriate certificate. If this SNI extension is missing, some servers return the default certificate and others throw an error, as in this case.
The fix is to specify the hostname using SSL_hostname
in ssl_opts
. A fix like this could probably help with LWP::UserAgent::DNS::Hosts
, that is, in LWP / Protocol / https / hosts.pm :
12 if (my $peer_addr = LWP::UserAgent::DNS::Hosts->_registered_peer_addr($host)) {
13 push @opts, (
14 PeerAddr => $peer_addr,
15 Host => $host,
16 SSL_verifycn_name => $host,
NEW SSL_hostname => $host, # for SNI
17 );
18 }
source to share