Unable to access insecure https with selenium using browsermob proxy
I am trying to implement a browserermob proxy with my selenium (chrome) environment to automatically test the UI to intercept responses and other networks.
Description:
Selenium webdriver using browsermob proxy and it works fine - HTTP and HTTPS secure url are ok. When I try to navigate to an unsecured HTTPS url I get this chrome error: ERR_TUNNEL_CONNECTION_FAILED
Here's my python code:
class Browser(object):
display = None
browser = None
def __init__(self, implicitly_wait_seconds=10, is_visible=True, display_size=None, browser_name='chrome'):
if not is_visible:
self.display = Display(display_size)
self.server = Server('/home/erez/Downloads/browsermob-proxy-2.1.4/bin/browsermob-proxy')
self.server.start()
self.proxy = self.server.create_proxy()
self.capabilities = DesiredCapabilities.CHROME
self.proxy.add_to_capabilities(self.capabilities)
self.proxy.new_har("test", options={'captureHeaders': True, 'captureContent': True})
self.start_browser(display_size, implicitly_wait_seconds, browser_name)
def __enter__(self):
return self
def __exit__(self, _type, value, trace):
self.close()
def start_browser(self, display_size, implicitly_wait_seconds=10, browser_name='chrome'):
if browser_name == 'chrome':
chrome_options = Options()
# chrome_options.add_argument("--disable-extensions")
chrome_options.add_experimental_option("excludeSwitches", ["ignore-certificate-errors"])
chrome_options.add_argument("--ssl-version-max")
chrome_options.add_argument("--start-maximized")
chrome_options.add_argument('--proxy-server=%s' % self.proxy.proxy)
chrome_options.add_argument('--ignore-certificate-errors')
chrome_options.add_argument('--allow-insecure-localhost')
chrome_options.add_argument('--ignore-urlfetcher-cert-requests')
self.browser = webdriver.Chrome(os.path.dirname(os.path.realpath(__file__)) + "/chromedriver",
chrome_options=chrome_options, desired_capabilities=self.capabilities)
self.browser.implicitly_wait(implicitly_wait_seconds)
source to share
I faced the same issue for SSL proxy using BrowserMob Proxy. To do this, you need to install the certificate in your browser, which was defined in this link
Go to the end of the document and you will see a section titled "SSL Support". Install ca-certificate-rsa.cer in your browser and there will be no more problems with the SSL proxy.
source to share
If installing browser / test server certificates won't do the job as in my case, not the most elegant way, but does the job:
You can work around the ERR_TUNNEL_CONNECTION_FAILED error by passing the trustAllServers parameter to the proxy instance, which will disable certificate validation for upstream servers. Unfortunately, as far as I know, this functionality was not implemented in the Browsermobs Python wrapper.
However, you can start a proxy with a parameter through the Browsermobs API (see REST API @ https://github.com/lightbody/browsermob-proxy/blob/master/README.md ). In the case of Python, queries might be the way to go.
Here's a snippet:
import json
import requests
from browsermobproxy import Server
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
# Start the proxy via python
server = Server('/path_to/bin/browsermob-proxy')
server.start()
# Start proxy instance with trustAllServers set to true, store returned port number
r = requests.post('http://localhost:8080/proxy', data = {'trustAllServers':'true'})
port = json.loads(r.text)['port']
# Start Chromedriver, pass on the proxy
chrome_options = Options()
chrome_options.add_argument('--proxy-server=localhost:%s' % port)
driver = webdriver.Chrome('/path_to/selenium/chromedriver', chrome_options=chrome_options)
# Get a site with untrusted cert
driver.get('https://site_with_untrusted_cert')
Also, if you need to access HAR data, you need to do so via the REST API as well:
requests.put('http://localhost:8080/proxy/%s/har' % port, data = {'captureContent':'true'})
r = requests.get('http://localhost:8080/proxy/%s/har' % port)
Of course, since you disable security features, this setting should only be used for limited testing.
source to share