Python Download Multiple Files From Web Page Save UPDATED

March 05, 2022 Post a Comment

Python Download Multiple Files From Web Page Save

· v min read · Updated sep 2021 · Web Scraping

Take you ever wanted to download all images on a sure web page? In this tutorial, you will larn how you can build a Python scraper that retrieves all images from a web page given its URL and downloads them using requests and BeautifulSoup libraries.

To get started, nosotros need quite a few dependencies, let's install them:

          pip3 install requests bs4 tqdm

Open up a new Python file and import necessary modules:

          import requests import os from tqdm import tqdm from bs4 import BeautifulSoup as bs from urllib.parse import urljoin, urlparse

First, let's make a URL validator, that makes certain that the URL passed is a valid one, equally there are some websites that put encoded data in the place of a URL, so we need to skip those:

          def is_valid(url):     """     Checks whether `url` is a valid URL.     """     parsed = urlparse(url)     return bool(parsed.netloc) and bool(parsed.scheme)

urlparse() part parses a URL into six components, we just need to encounter if the netloc (domain name) and scheme (protocol) are there.

2nd, I'1000 going to write the core function that grabs all image URLs of a spider web folio:

          def get_all_images(url):     """     Returns all prototype URLs on a single `url`     """     soup = bs(requests.go(url).content, "html.parser")

The HTML content of the web page is in soup object, to extract all img tags in HTML, we need to utilise soup.find_all("img") method, permit's see it in action:

                      urls = []     for img in tqdm(soup.find_all("img"), "Extracting images"):         img_url = img.attrs.become("src")         if non img_url:             # if img does not contain src attribute, just skip             go along

This will remember all img elements as a Python list.

I've wrapped it in a tqdm object just to print a progress bar though. To grab the URL of an img tag, at that place is a src aspect. However, in that location are some tags that do not contain the src attribute, we skip those past using the continue argument in a higher place.

Now nosotros demand to make sure that the URL is accented:

                      # brand the URL absolute by joining domain with the URL that is just extracted         img_url = urljoin(url, img_url)

There are some URLs that contains HTTP Go primal-value pairs that we don't like (that ends with something like this "/epitome.png?c=3.ii.v"), permit'south remove them:

                      try:             pos = img_url.alphabetize("?")             img_url = img_url[:pos]         except ValueError:             pass

Nosotros're getting the position of '?' character, so removing everything after information technology, if in that location isn't any, it volition raise ValueError, that'due south why I wrapped it in endeavor/except block (of form you tin implement it in a better way, if so, please share with united states of america in the comments below).

At present let's make sure that every URL is valid and returns all the epitome URLs:

                      # finally, if the url is valid         if is_valid(img_url):             urls.append(img_url)     return urls

At present that we accept a part that grabs all image URLs, nosotros need a function to download files from the web with Python, I brought the post-obit function from this tutorial:

          def download(url, pathname):     """     Downloads a file given an URL and puts it in the binder `pathname`     """     # if path doesn't exist, make that path dir     if not os.path.isdir(pathname):         bone.makedirs(pathname)     # download the trunk of response by chunk, not immediately     response = requests.get(url, stream=True)     # get the total file size     file_size = int(response.headers.get("Content-Length", 0))     # get the file proper noun     filename = os.path.bring together(pathname, url.divide("/")[-one])     # progress bar, changing the unit to bytes instead of iteration (default by tqdm)     progress = tqdm(response.iter_content(1024), f"Downloading {filename}", total=file_size, unit="B", unit_scale=True, unit_divisor=1024)     with open(filename, "wb") as f:         for information in progress.iterable:             # write data read to the file             f.write(data)             # update the progress bar manually             progress.update(len(data))

The to a higher place part basically takes the file url to download and the pathname of the folder to relieve that file into.

Related: How to Convert HTML Tables into CSV Files in Python.

Finally, here is the main part:

          def main(url, path):     # get all images     imgs = get_all_images(url)     for img in imgs:         # for each paradigm, download it         download(img, path)

Getting all image URLs from that folio and download each of them i by one. Let's examination this:

          principal("https://yandex.com/images/", "yandex-images")

This will download all images from that URL and stores them in the folder "yandex-images" that will exist created automatically.

Note though, there are some websites that load their data using Javascript, in that case, y'all should utilize requests_html library instead, I've already made another script that makes some tweaks to the original ane and handles Javascript rendering, cheque it here.

Alright, nosotros're done! Here are some ideas you lot tin implement to extend your code:

Extracting all links on a web page and downloading all images on each.
Download every PDF file on a given website.
Apply multi-threading to accelerate the download (since this is a heavy IO chore).
Use proxies to foreclose sure websites from blocking your IP address.

Want to Learn More than well-nigh Spider web Scraping?

Finally, if you want to dig more into web scraping with dissimilar Python libraries, not just BeautifulSoup, the below courses will definitely be valuable for yous: