Popup handling in Selenium phantomJS

I am creating a web crawler in python selenium phantomJS that visits links, takes a screenshot of the entire webpage and saves it to AWS s3. Unfortunately, on some sites I get stuck with the "Please subscribe to our newsletter" popup and despite my best efforts, I cannot find a solution to shut it down. This is how it looks:

oop up example

From their documentation ( http://selenium-python.readthedocs.io/api.html#module-selenium.webdriver.common.alert ) I tried alert, but it doesn't seem to work. I need to handle both HTML and JavaScript popups.

Here's a shorthand example of the code I'm working with:

from selenium import webdriver
from pyvirtualdisplay import Display
display = Display(visible=0, size=(1600, 1200)) # Xvfb headless display
display.start()
capabilities = {'handlesAlerts': 'True'}
driver = webdriver.PhantomJS(service_log_path=os.path.devnull, desired_capabilities=capabilities)
driver.get('https://www.everlane.com/products/mens-slim-fit-oxford-shirt2-grey?collection=mens-shirt-shop')
alert = driver.switch_to_alert()
alert.accept()
driver.save_screenshot()

      

I also tried:

driver.switch_to_default_content()

Keyboard "ESC", "ENTER", "F4" also do not work.

I can find the XPath element and click to close, but I need a general purpose solution because I am working with multiple sites and therefore multiple XPaths.

Any help is appreciated! I may not be the only one with this problem, so I hope others find it useful. Thank you.

+3


source to share





All Articles