Collecting Web Data - APIs & Web Scraping

About Pre-built Web Scraping Tools

There are many different web scraping tools and software packages available on the internet.  Depending on your research objectives, these tools may or may not be useful to you.  If you are looking into these tools, there are a few core questions you should ask yourself in order to determine if any individual tool is appropriate for you: 

  • Ease of use - How easy or hard is it to install the software, learn its functions, and start collecting data?  Some tools are extremely intuitive, while others are not.  For any individual tool, try to find "examples", "guides", or "tutorials" on the developer's website.  These will give a good impression about how easy or hard to use the program will be.
  • Functionality and customization - How much functionality does the software tool offer and how well does that align with your research needs?  Does the tool give you raw data or pre-made reports?  Can the web-scraper be automated or do you need to manually activate the software every time?  Depending on your research project, you will want to closely inspect whether a web scraping tool will actually provide the data you need in the format that you need.
  • Cost - How much does the tool cost?  The cost to use any given tool can vary wildly.  Make sure to you fully understand the costs before signing up to use a tool.  These tools are not easy to build or maintain, so you should not be surprised if a tool that does exactly what you need will also cost money.  Some tools are wholly free to use.  Some tools offer a "free" version of the product that is severely limited, while the paid version provides full functionality.  Additionally, the cost for any given tool could be a single flat fee, a monthly subscription, or a use-based fee system.
  • Privacy policies - What privacy policies and data protections are in place?  Every company and organization that develops these tools has different privacy policies.  In some cases, web scraping tools will process and store data on their servers before sending it to your computer.  If you are working with data that is in any way sensitive (e.g. collecting user-comments on a social media website), it is critical that you evaluate the privacy policies and protections that are (or are not) associated with a web scraping tool and the software developer.

List of Web Scraping Tools

Below is a short, non-exhaustive list of popular web scraping tools and software packages: