Batch scraping

In the last section, we have learned how to define various recipes. However, up to now, the scope of scraping is locked when defining recipe, such as the start URL, the keywords to search, or page number to scrape.

In this section, we will show how to make dynamic recipes which accept dynamic arguments when starting and the recipes will take these arguments into scraping.

In this section, we will learn how to

  • scrape urls in lists easily
  • scrape the query result of a keyword list
  • scrape the result of a keyword list, and with custom number of pages in result list to scrape for each keyword
  • scrape the list of products from one website, then search each product in another wesbite, and scrape the result on the second website.