Python requests sometimes return an empty list
So I was trying to clear "2005 - 2013" from "Drink from 2005 to 2013" and at first this code worked for me, but now I only get empty lists, but my requests still have a 200 status code
import requests, lxml.html, csv
headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36'}
page = requests.get('http://www.cellartracker.com/wine.asp?
iWine=91411',headers=headers)
print(page.status_code)
html = lxml.html.fromstring(page.content)
content_divs = html.xpath('//a[@title="Source: Community"]/text()')
print(content_divs)
Not sure if I start using selenium to do this because this is a js site and if that's not how to do it then some basic help would be helpful! Thank!!
+3
source to share
1 answer
Selenium use
from selenium import webdriver
url = "https://www.cellartracker.com/wine.asp?iWine=91411"
driver = webdriver.Chrome(executable_path="chromedriver2.25")
driver.get(url)
list = driver.find_elements_by_xpath("//li[contains(.,'review')]")
for item in list:
print(item.text)
print("---")
Output:
Options
1/4/2014 - REUBENSHAPCOTT WROTE:
91 Points
Delicious! Had no idea that Australia made port this good, and affordable. Terrific, smooth fig and plum. Aged and neither sharp nor grapey. If you see it, buy it.
Do you find this review helpful? Yes - No / Comment
---
Options
1/20/2013 - LISAADAM WROTE:
85 Points
The wine looks Tawny colored.
Do you find this review helpful? Yes - No / Comment
---
Options
12/22/2012 - WINEAGGREGATE LIKES THIS WINE:
90 Points
Molasses, pepper, butterscotch candy that been roasted a bit. Very nice.
Do you find this review helpful? Yes - No / Comment
---
Options
10/30/2011 - GTI2TON WROTE:
87 Points
Sweeter than average tawny and straightforward, but still has nice richness in its raisin and light carmel notes. Good value.
Do you find this review helpful? Yes - No / Comment
+1
source to share