Clever Geek Handbook

Download all files on the website

I need to download all files from these links where only the name of the suburb is stored in each link

Just link https://www.data.vic.gov.au/data/dataset/2014-town-and-community-profile-for-thornbury-suburb

All files under this search link: https://www.data.vic.gov.au/data/dataset?q=2014+town+and+community+profile

Any possibilities?

Thank:)

+3


source to share


2 answers


You can download a file like this

import urllib2
response = urllib2.urlopen('http://www.example.com/file_to_download')
html = response.read()


To get all links on the page

from bs4 import BeautifulSoup

import requests
r  = requests.get("http://site-to.crawl")
data = r.text
soup = BeautifulSoup(data)

for link in soup.find_all('a'):
    print(link.get('href'))
+5


source


you can do it in two ways: 1.with the 2.by program code

1.by code

using Web Scrapper libraries like BeautifulSoup, JSoup



if it doesn't work, or you need to click buttons so you can use Selenium.

2.by software

use the Internet Download Manager function to do this

0


source






More articles:


All Articles