Repetition of data after web scraping using python and nice soup
I am trying to clear data from the Garmin golf site. I would like to get the name of the golf course and the address, but after running the script. I noticed that my codes are just repeating the first page data over and over. I also noticed that the page numbers on the site don't start with 1, but 10 for the second page. How can I extract data from this site and get everything, and not repeat only the first page.
import csv
import codecs
import requests
from bs4 import BeautifulSoup
courses_list= []
for i in range(10):
url = "http://sites.garmin.com/clsearch/courses?browse=1&country=US&lang=en&per_page={}".format(i)
r = requests.get(url)
soup = BeautifulSoup(r.content)
g_data2=soup.find_all("div",{"class":"result"})
for item in g_data2:
try:
name= item.contents[3].find_all("div",{"class":"name"})[0].text
print name
except:
name=''
try:
address= item.contents[3].find_all("div",{"class":"location"})[0].text
except:
address=''
course=[name,address]
courses_list.append(course)
with open ('G_Final.csv','a') as file:
writer=csv.writer(file)
for row in courses_list:
writer.writerow([s.encode("utf-8") for s in row])
+3
source to share