In July’s episode of Show & Tell, Hal Triedman showed us how easy it can be to poison an AI research agent.
Hal is a privacy and security researcher at Cornell Tech and former senior privacy engineer at the Wikimedia Foundation. He walked us through a recent research project he ran with colleagues Tingwei Zhang and Vitaly Shmatikov. Their draft paper, “Deep-Research Agents Can Be Poisoned via User-Generated Content,” details how deep-research agents often rely on a handful of Reddit and Wikipedia pages across an entire topic, no matter how a user phrases the question. They appended as few as 13 words to one of those pages, which caused the agents to recommend a product, app, or investment that doesn't exist, like the cryptocurrency BananaCoin or dating app for divorced men over 50 called SilverPath. The team ran this in a simulated environment rather than on the live web to avoid pushing the made up products they were trying to make the AI believe were real.
These attacks are increasingly relevant as brands, marketers, and deceptive actors try to get AI models to reference their products and content, just as they’ve long done with search engines. For specific topics where UGC sites feature prominently on Google, this can be a relatively easy attack to pull off – Hal said success rates ranged between 20 and 40%. Subreddits are already having to take countermeasures to avoid getting spammed.
The attacks are also cheap to execute. “If you're an attacker, this costs two cents per run,” Hal said. “You could do large-scale evaluation of a ton of queries for a couple hundred dollars, which is basically nothing.”
You can watch the episode above or on our YouTube channel.
In the “Show” half of the episode, Hal shared several tools that can be useful when investigating Wikipedia. He showed Pageviews Analysis, which lets you see traffic to specific pages on Wikipedia over time, as well as a project he worked on that displays pageviews by country, project, and more. Hal also recommends the Wikimedia Enterprise API, whose free tier is sufficient for many investigative purposes.
Other resources include the discussion forum Village Pump, the newsletter The Signpost, and of course talk pages for people or topics of interest.
About Show & Tell
Show & Tell is a monthly podcast hosted by Indicator co-founders Alexios Mantzarlis and Craig Silverman.
We talk about digital investigations with leading journalists, OSINT investigators, threat analysts, and trust and safety researchers. We aim to be as practical as possible by having our guests walk us through one or more investigations or research projects they actually worked on.
This is our sixth episode. Our previous episodes featured:
Omer Benjakob, disinformation and cyber reporter for Haaretz.
Barbara Marcolini, visual investigator with Amnesty International’s Evidence Lab.
Clara Jiménez Cruz, CEO of the Spanish nonprofit Maldita.es.
Jeff Horwitz, an investigative reporter for Reuters.
Kolina Koltai, a senior researcher and trainer at Bellingcat.
We hope you enjoy and learn from the conversation with Hal. Catch every episode by subscribing to our YouTube channel. We’re also on Spotify and Apple Podcasts.
Have a suggestion for who we should feature next? Email us.
