The Indicator Guide to building your own reverse image search engine

Ishaan Jhaveri investigates visual stories using open-source and computational techniques. He’s currently a computational reporter and producer with the BBC’s Eye Investigations team in London and previously worked as a reporter on the New York Times’ Visual Investigations team in New York.

Reverse image search is a core technique in digital investigations. It is used by fact-checkers and journalists all over the world to identify old footage being shared in misleading contexts, as part of geolocation, and was even used last year by Sherwood News to try to match the faux gold decor that was installed in a recent Oval Office refresh. 

In 2019, Aric Toler published a comprehensive reverse image search guide for Bellingcat, much of which remains relevant. It identified the relative strengths of reverse image search engines developed by Google, Bing, Yandex and TinEye. Bellingcat’s guide and the examples and tools cited above focus on reverse image searching against the public internet — looking only on publicly available, openly indexable websites for images that are visually similar to a query image. But this is not the only use case for reverse image search.

What if you have a proprietary dataset of images, or you’ve collected images from a source that’s not indexed by popular reverse image search sites? Or what if you need to constrain the universe of results returned for an image search?

In this guide I will show step-by-step instructions for using two tools that allow you to index a dataset of images (or videos) of any size and run reverse image search queries against it. Both approaches keep all data (indexed data + query data) private and run all processing locally on your computer. One tool is free but requires light coding (that an LLM could easily help with); the other is code-free and allows 40 free searches before charging $49/month. 

The approaches I explain in this guide help filter large troves of images into much smaller sets of results that you can then manually peruse. I’ve used them to search images scraped from Russian social media sites to identify soldiers active in the ongoing invasion of Ukraine; to compare features inside buildings that are used by people for illegal activities to images scraped from local housing catalogs; and to match Discord users to Steam users by their profile pictures.

You don't need advanced technical skills to do the same. Here’s how.