For me, web scraping has always carried an illicit thrill. It's a combination of a jewel heist, jigsaw puzzle, and lockpicking.
I began web-scraping 13 years ago, when I was starting out as a data journalist. I understood scraping but had never coded a functioning scraper; learning Python was daunting.
In 2013, Duke University's Reporters' Lab asked if I could test and review enterprise scraping software. I jumped at the chance to learn how to crawl, parse and extract unstructured data from websites. It enabled me to land scoops that helped me gain a foothold in the industry. At the time, you needed at least basic coding skills to be able to use the tools I reviewed. And as much as I loved web scraping, it was a pain in the ass, even with coding skills and premium tools.
That’s no longer the case. Today, you can scrape webpages, including those with features like dropdown menus and infinite scroll, or protections like Cloudflare, without technical expertise or a big budget. There are browser extensions, AI-assisted tools, and other options that put scraping in the hands of any investigator. To help you decide which scraper is the right fit for your needs, I put seven options to the test.
Table of Contents
Upgrade to read the rest
Become a paying member of Indicator to access all of our content and our monthly members-only workshop. Support independent media while building your skills.
UpgradeA membership gets you:
- Everything we publish, plus archival content, including the Academic Library
- Detailed resources like, "The Indicator Guide to connecting websites together using OSINT tools and methods"
- Live monthly workshops and access to all recordings and transcripts
