Grokipedia cites a Nazi forum and fringe conspiracy websites

Grokipedia’s entry for “Clinton body count,” a conspiracy theory that falsely links the deaths of multiple people to Bill and Hillary Clinton, cites an unlikely source:

InfoWars is the fringe conspiracy website operated by Alex Jones, the man who owes $1.4 billion to the families of the victims of the Sandy Hook Elementary School shooting for repeatedly calling the mass tragedy a hoax. Jones’ website is cited 34 times on Grokipedia, the AI-powered encyclopedia Elon Musk launched late last month to counter Wikipedia, which he has accused of being an “extension of legacy media propaganda.”

By comparison, InfoWars has never been cited as a source on Wikipedia. It is one of several fringe websites that were never used as a source or only cited a single time in Wikipedia articles that have a bigger footprint on Grokipedia. Others include white nationalist forum Stormfront (42 Grokipedia citations), the anti-vaccine website LifeSiteNews (100 citations), and the conspiracy websites Global Research and VoltaireNet (51 and 45 citations respectively). 

Grokipedia’s undiscerning approach to sourcing is far bigger than a few hundred citations to fringe websites. The AI-infused encyclopedia cites domains that Wikipedia editors have deemed “generally unreliable,” “blacklisted,” or “deprecated” 2.6 million times across its almost 890,000 entries. That’s 6.0% of all citations, and twice the share of the same sources on English Wikipedia. Reviewing citations using a comprehensive set of domain ratings compiled by Hause Lin and other researchers in 2023 also suggests Grokipedia overrelies on low quality sources compared to Wikipedia.

We know all this thanks to my colleague Hal Triedman, a privacy and security researcher at Cornell Tech¹. Hal scraped 99.8% of Grokipedia’s corpus between Oct 28 and 30 and compared its citations and text to the equivalent entries on Wikipedia. He invited me to review the data and we will be publishing our findings in an upcoming preprint on arXiv;² this blog serves as a preview of our results with a generous sprinkling of my less academically rigorous speculation about what they mean.

This exercise also created a first-of-its-kind dataset that we hope other analysts and researchers will use to provide a deeper understanding of Grokipedia (GitHub | Hugging Face).

Hal was motivated to run this analysis because “Wikipedia is one of the most fascinating cultural objects that humanity has ever created. As an object of study, it has informed our understanding of language, culture, and how networks function. My hope is that, by making Grokipedia equally accessible and specifying how it differs from Wikipedia, we can have a meaningful discussion about how we construct factually-grounded encyclopedias.”

Elon Musk has said that Grokipedia’s goal is “the truth, the whole truth and nothing but the truth.” He has been less clear about how his AI-generated Wikipedia alternative was compiled in practice and how it differs from the crowdsourced encyclopedia. Thanks to the data Hal collected, we can start to figure this out.

I spent a week looking at the data and comparing several entries on both platforms. My assessment is that Grokipedia has made significant editorial decisions around sourcing and topic treatment that lend credence to Matteo Wong’s argument in The Atlantic that it is “the next step in Musk’s propaganda machine.”

This doesn’t mean that Grokipedia is a wholesale rewriting of Wikipedia, or a completely original version unto itself. In fact, we found that Grokipedia largely emulates Wikipedia in over half of its entries, seemingly content to transcribe pages verbatim from a rival that Musk has dismissed as biased. This makes Grokipedia often consistent with Wikipedia in terms of content and source quality. The concern is that a smaller portion of entries — which are often about more sensitive topics — have been significantly rewritten to highlight a specific narrative. In the process, Grokipedia has introduced a large swath of less reputable sources.