Ruby selenium web driver - get google graphic content
I am using ruby selenium web driver and trying to get content from google knowledge graph which is on top right site from search results on first page of google search results in <div class="xpdopen">
@driver = Selenium::WebDriver.for :phantomjs
@driver.manage.timeouts.implicit_wait = 10
@driver.get "http://google.com"
element = @driver.find_element :name => "q"
element.send_keys "BMW"
element.submit
content = @driver.find_element(:class, 'xpdopen')
But selenium cannot find this element and raises an error
#<Selenium::WebDriver::Error::NoSuchElementError: {"errorMessage":"Unable to find element with class name 'xpdopen'"
When I tried in chrome js console $('.xpdopen')
it immediately found this item
I have also tried
@driver.execute_script("return document.getElementsByClassName('xpdopen');")
but it cannot find this item
I also linked @driver.page_source
and <div class="xpdopen">
not to the page source, but I can see this in the chrome console. Why?
How can I get this item with selenium?
Here are the results I get from pry:
[21] pry(main)> @driver = Selenium::WebDriver.for :phantomjs
=> #<Selenium::WebDriver::Driver:0x..f822d288ec7f0a708 browser=:phantomjs>
[22] pry(main)> @driver.manage.timeouts.implicit_wait = 10
=> 10
[23] pry(main)> @driver.get "http://google.com"
=> {}
[24] pry(main)> element = @driver.find_element :name => "q"
=> #<Selenium::WebDriver::Element:0x..f389f4a8876f601e id=":wdc:1434526425103">
[25] pry(main)> element.send_keys "BMW"
=> nil
[26] pry(main)> element.submit
=> {}
[27] pry(main)> sleep 10
=> 10
[28] pry(main)> content = @driver.find_element(:xpath, '//*[@id="rhs_block"]/ol/li/div[1]/div')
Selenium::WebDriver::Error::NoSuchElementError: {"errorMessage":"Unable to find element with xpath '//*[@id=\"rhs_block\"]/ol/li/div[1]/div'","request":{"headers":{"Accept":"application/json","Accept-Encoding":"gzip;q=1.0,deflate;q=0.6,identity;q=0.3","Connection":"close","Content-Length":"67","Content-Type":"application/json; charset=utf-8","Host":"127.0.0.1:8929","User-Agent":"Ruby"},"httpVersion":"1.1","method":"POST","post":"{\"using\":\"xpath\",\"value\":\"//*[@id=\\\"rhs_block\\\"]/ol/li/div[1]/div\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/2f3cf350-14c3-11e5-9f8e-4173e8049986/element"}} (org.openqa.selenium.NoSuchElementException)
and
[29] pry(main)> content = @driver.find_element(:css, "#rhs_block > ol > li > div.kp-blk._Jw._Rqb._RJe > .xpdopen")
Selenium::WebDriver::Error::NoSuchElementError: {"errorMessage":"Unable to find element with css selector '#rhs_block > ol > li > div.kp-blk._Jw._Rqb._RJe > .xpdopen'","request":{"headers":{"Accept":"application/json","Accept-Encoding":"gzip;q=1.0,deflate;q=0.6,identity;q=0.3","Connection":"close","Content-Length":"113","Content-Type":"application/json; charset=utf-8","Host":"127.0.0.1:8929","User-Agent":"Ruby"},"httpVersion":"1.1","method":"POST","post":"{\"using\":\"css selector\",\"value\":\"#rhs_block \\u003e ol \\u003e li \\u003e div.kp-blk._Jw._Rqb._RJe \\u003e .xpdopen\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/2f3cf350-14c3-11e5-9f8e-4173e8049986/element"}} (org.openqa.selenium.NoSuchElementException)
and just to show that it finds other elements on the same page without issue:
[30] pry(main)> results = @driver.find_elements(:xpath, "//p/a")
=> [#<Selenium::WebDriver::Element:0x6f6a74631e2b7010 id=":wdc:1434527087873">,
#<Selenium::WebDriver::Element:0x7b6d276448081688 id=":wdc:1434527087874">,
#<Selenium::WebDriver::Element:0x..f9504a4171b03970a id=":wdc:1434527087875">,
#<Selenium::WebDriver::Element:0x..fa6e0158aa8d24e2a id=":wdc:1434527087876">,
#<Selenium::WebDriver::Element:0x327bf842e4399368 id=":wdc:1434527087877">,
#<Selenium::WebDriver::Element:0x..fae292d7ca211ab32 id=":wdc:1434527087878">,
#<Selenium::WebDriver::Element:0x129a58eb5ed6ee9c id=":wdc:1434527087879">,
#<Selenium::WebDriver::Element:0x46ef3b45800e63e0 id=":wdc:1434527087880">,
#<Selenium::WebDriver::Element:0x26bfb47f8ad498ea id=":wdc:1434527087881">,
#<Selenium::WebDriver::Element:0x..f03756c2924a2974 id=":wdc:1434527087882">,
#<Selenium::WebDriver::Element:0xfba93aab4b32af8 id=":wdc:1434527087883">]
I took a screenshot and found out that phantomjs does not display (has no content) knowledge graph
Screenshot of phantoms
Screenshot from Firefox
Why doesn't phantomjs have a content knowledge graph?
source to share
Obviously the css won't know where to find the xpdopen class on its own, you need to pass the entire path to the element:
Xpath:
content = @driver.find_element(:xpath, "//*[@id="rhs_block"]/ol/li/div[1]/div")
Css:
content = @driver.find_element(:css, "#rhs_block > ol > li > div.kp-blk._Jw._Rqb._RJe > .xpdopen")
source to share