Clever Geek Handbook

Can't use selenium with proxy

The script I am doing is to change my IP several times and visit the website using the Tor Browser. I got the IP changes to work, but I get an error when using Selenium with a proxy. My code:

import socket
import socks
import httplib    
from subprocess import check_call
from selenium import webdriver
from selenium.webdriver.common.proxy import *
from selenium.webdriver.support.wait import WebDriverWait

def connectTor():
    socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5,"127.0.0.1",9150,True)
    socket.socket = socks.socksocket

def newIdentity():        
    check_call(["killall","-HUP", "tor"])
    connectTor()

def showIP():
    conn = httplib.HTTPConnection("my-ip.herokuapp.com")
    conn.request("GET","/")
    response = conn.getresponse()
    print (response.read())

def process():
    url = "https://www.google.bg"
    port = "8118" #The Privoxy (HTTP) port
    myProxy = "127.0.0.1:"+port
    proxy = Proxy({
        'proxyType': ProxyType.MANUAL,
        'httpProxy': myProxy,
        'ftpProxy': myProxy,
        'sslProxy': myProxy,
        'noProxy': ''
    })

    browser = webdriver.Firefox(proxy=proxy)
    browser.get(url)
    WebDriverWait(browser, 10)

    browser.close()

def main():
    connectTor()    
    print("Connected to Tor")    
    showIP()    
    process()

    print("Hew Id is")
    newIdentity()
    showIP()
    process()

main()

The trail I am getting:

Traceback (most recent call last):
  File "/home/peter/.spyder2/.temp.py", line 60, in <module>
    main()
  File "/home/peter/.spyder2/.temp.py", line 53, in main
    process()
  File "/home/peter/.spyder2/.temp.py", line 43, in process
    browser = webdriver.Firefox(proxy=proxy)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py", line 59, in __init__
    self.binary, timeout),
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/extension_connection.py", line 47, in __init__
    self.binary.launch_browser(self.profile)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 66, in launch_browser
    self._wait_until_connectable()
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 97, in _wait_until_connectable
    while not utils.is_connectable(self.profile.port):
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/common/utils.py", line 43, in is_connectable
    socket_.connect(("127.0.0.1", port))
  File "/usr/lib/python2.7/dist-packages/socks.py", line 369, in connect
    self.__negotiatesocks5(destpair[0],destpair[1])
  File "/usr/lib/python2.7/dist-packages/socks.py", line 236, in __negotiatesocks5
    raise Socks5Error(ord(resp[1]),_generalerrors[ord(resp[1])])
TypeError: __init__() takes exactly 2 arguments (3 given)

After changing the code as suggested by Louis, I get an error in the browser:

The proxy server is refusing connections. Firefox is configured to use a proxy server that refuses connections.

The output I get is:

Connected to Tor
78.108.63.46

Hew Id is
tor(991): Operation not permitted
62.212.89.116
+3


source to share


1 answer


The problem is that in your code, after you set up the socks proxy, this is valid for everything that follows . So when the Selenium client (your script) tries to talk to the Selenium server (your browser), it tries to use the socks proxy. You need to narrow down your proxy usage only where you need it and when you check your IP address. So change connectTor

to this:

import contextlib

@contextlib.contextmanager
def connectTor():
    socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, "127.0.0.1", 9150, True)
    # Save the old value so that we can restore it later.
    old_socket = socket.socket
    # We are now using the proxy for all connections.
    socket.socket = socks.socksocket
    yield # Let the contents of the `with` block that will use this function execute.
    # We are no longer using the proxy.
    socket.socket = old_socket

And change yours main

to:

def main():
    with connectTor():
        print("Connected to Tor")
        showIP()
    process()

    print("Hew Id is")
    newIdentity()
    with connectTor():
        showIP()
    process()


With this code, the socks proxy setting is only valid inside blocks with connectTor()

. The decorator contextmanager

and how it works is documented here . The changes I mentioned above work (I tested them), but I have never used the library socks

. I'm pretty sure there is a better way to code connectTor

as a context manager than what I did, but at least now you have an idea of ​​what the problem is and how to fix it.

You also need to set your proxy for Selenium with a prefix http://

, so:

myProxy = "http://127.0.0.1:" + port
0


source






More articles:


All Articles