Collecting Web Data - APIs & Web Scraping

Web Scraping

Web scraping is a powerful, but complex approach to collecting information from websites.  At its core, "web scraping" broadly encompasses activities in which you use software or programmatic scripts to systematically find, download, and organize information found on the internet.  Web scraping is often the best approach to collecting data from the web when the following conditions are met:

  • The data is not otherwise available via APIs or accessible datasets.
  • A systematic approach is necessary to...
    • collect large volumes of organized data.
      AND / OR
    • collect data that changes or updates on a regular basis.

There are a myriad of tools and considerations to keep in mind when pursuing a web scraping project.  Use the links below or in the navigation-menu for further info.