In a shock development - AI companies are failing to benchmark what matters in newsrooms. More excellent work from our Tow Center research team @klaudia.bsky.social and @aisvarya17.bsky.social

In our latest piece for CJR, @aisvarya17.bsky.social and I argue that third-party evaluation of AI models is not just a technical matter; it’s a matter of accountability. www.cjr.org/analysis/jou...

The performance tests used by AI companies don’t measure what matters in the newsroom. By @klaudia.bsky.social and @aisvarya17.bsky.social for @towcenter.bsky.social. www.cjr.org/analysis/jou...

The Tow Center has looked into how good LLMs are at verifying and geolocating an image. Gentle reminder: "on multiple occasions models blatantly lied about the steps they took to arrive at an answer to our query" www.cjr.org/tow_center/w... #ai #osint

We asked seven different AI models to identify the location, date and source of photos taken by photojournalists, they collectively only answered 14 out of 280 queries completely correctly. @aisvarya17.bsky.social @klaudia.bsky.social @columjournreview.bsky.social www.cjr.org/tow_center/w...

New Research: Can AI models be used to fact-check images? www.cjr.org/tow_center/w... I cannot summarize this piece better than @klaudia.bsky.social did in this thread. The methodology and data from the experiment are linked in the article. Reach out with any questions, comments, or feedback!

NEW in @columjournreview.bsky.social: Why AI models are bad at fact-checking photos www.cjr.org/tow_center/w...

Great work by @aisvarya17.bsky.social and @klaudia.bsky.social. Read more below! www.cjr.org/tow_center/w...

More testing of LLMs for journalism, this time on identifying image location, date, and source. By @aisvarya17.bsky.social and @klaudia.bsky.social with some of my own commentary. www.cjr.org/tow_center/w... @columjournreview.bsky.social

The Future of News and Search: The Tow Center for Digital Journalism interviewed news and tech industry representatives about AI’s impact on platforms and publishers. They expressed some hope and a lot of trepidation. By Klaudia Jaźwińska. www.cjr.org/the_media_to...

SPARC has invited @klaudia.bsky.social & @aisvarya17.bsky.social (Tow Center for Digital Journalism at Columbia U) @davehansen.bsky.social (Authors Alliance) & @petersuber.bsky.social (Harvard U) to discuss AI issues on May 29 at 1 pm EDT. SPARC members can register at: sparcopen.org/event/sparc-...

I wrote about the report - and publishers' anxiety around the decline of traditional search - in today's @columjournreview.bsky.social newsletter www.cjr.org/the_media_to...

A new report I co-authored for the @towcenter.bsky.social is out today! We interviewed representatives from the news and technology industries about AI’s impact on the relationship between platforms and publishers. They expressed some hope and a lot of trepidation. www.cjr.org/the_media_to...

Google's "AI Mode" in action. To the extent anyone uses it, this will be a traffic killer for news: two clicks to actually visit a site and you have to click the link icon for the sidebar to update with the sources so it also limits source credibility signals.

2️⃣ At 10:15 GMT we'll examine the power dynamics emerging between news organisations and AI developers, focusing on content licensing and data valuation Featuring @fedecherubini.bsky.social @felixsimon.bsky.social @agstrait.bsky.social @mattrogerson.bsky.social @klaudia.bsky.social buff.ly/dEmIIrg

Google and OpenAI want Trump to open up the rules. News publishers have some thoughts, by Klaudia Jaźwińska. www.cjr.org/the_media_to...

Columbia Journalism Review tested eight generative AI search tools and found their answers were wrong 60% of the time, and the paid ones actually fared worse than the free ones. Meanwhile, millions of people trust the way they present total bullshit with confident language.

Imagine if all of government reported data like DOGE does -- deleting numbers, altering them without notice, pointing numbers to the wrong things. w/ another awesome @ethansinger.bsky.social graphic: www.nytimes.com/2025/03/13/u...

On a darker note - the hyper partisan dark money funded ‘news’ websites we have studied for 5 years or more @towcenter.bsky.social are taking an ugly AI turn. This was always going to happen - will not be the first or last incidence . www.nytimes.com/2025/03/12/u...

It's heartening to see that our research is being shared widely and there is interest in holding AI tools accountable. But there have been some representations of our research that we would like to address. www.cjr.org/tow_center/w... @klaudia.bsky.social

This post is misleading. We were testing specifically to see whether the chatbots accurately identified the sources of excerpts from news articles. We did not intend to extrapolate these findings to the overall accuracy of the chatbots.

Lets replace search with "AI" then! Totally logical if you ask me. Even more worth it when you know they're exponentially overtaking the airline industry in their carbon footprint. Study: www.cjr.org/tow_center/w...

Thank you for featuring our research!

well yeah, you gotta pay more for the confidently incorrect. regular incorrect is for freemium users

Full study using 8 LLMs and 20 sample publishers to test how AIs cite news. Answer: badly. Even if you have an AI deal, or robot.txt enabled, results are similar . And pro versions more expensive but no better . [email protected] researchers @klaudia.bsky.social and @aisvarya17.bsky.social

AI search engines fail to produce accurate citations in over 60% of tests, according to new Tow Center study

Tow Center for Digital Journalism study found that "AI" chatbots provided incorrect answers to more than 60 percent of queries, with Musk's Grok 3 responding to 94 percent of the queries incorrectly. "Premium chatbots provided more confidently incorrect answers than their free counterparts."

A study of eight AI search engines found they provided incorrect citations of news articles in 60%+ of queries; Grok 3 answered 94% of the queries incorrectly (Columbia Journalism Review) Main Link | Techmeme Permalink

Thank you to @andrewdeck.bsky.social for writing about my and @aisvarya17.bsky.social's research in @niemanlab.org!

A list of words that are disappearing in the Trump administration. One example below. www.nytimes.com/interactive/...

538-style poll collection has now been recreated by a small group of fans and former staff. all fully transparent and public now. return of live public aggregation is imminent docs.google.com/spreadsheets...

Chat-based search apps are often "confidently wrong" in attributing and citing news stories... particularly Grok, which performs abysmally. Terrific study from CJR: www.cjr.org/tow_center/w...

Are you using Generative AI tools as your search engine? @towcenter.bsky.social ran a test to see how often they cite news articles correctly given direct quotes from them. Results are troubling. www.cjr.org/tow_center/w... Great work by @aisvarya17.bsky.social and @klaudia.bsky.social

🔍 AI Search Engines: popular, but potentially problematic. A new study from the @towcenter.bsky.social reveals the hidden pitfalls in AI-powered search, like misattribution, misinformation and missing citations: www.cjr.org/tow_center/w...

Our new analysis of Comscore data shows ChatGPT and Perplexity send negligible traffic to news publishers, preferring sources like academic journals and education sites. Research with Nick Hagar on the GAIN Blog today: generative-ai-newsroom.com/genai-as-new...

@klaudia.bsky.social and I spent a few months studying how generative search tools cite news content. We'd love to hear your thoughts on our findings! @towcenter.bsky.social @columjournreview.bsky.social

Will Elon Musk Break with Community Notes? His own fact-checking system has led to some awkward moments for the X owner–cum-bureaucrat, by Sarah Grevy Gotfredsen. www.cjr.org/the_media_to...

For @columjournreview.bsky.social, @aisvarya17.bsky.social and I tested Grok-3's ability to identify quotes from news publishers. Out of the 200 quotes we tested, 102 returned hallucinated citations and the original source article was correctly cited only nine times. www.cjr.org/the_media_to...

"Grok 3 required training two new massive data centers operating full time for months, and 15x the compute of Grok 2 -yet all these kinds of errors feel awfully familiar"

This year I always had my camera and a small photo box to hand when I was gardening. My children are always finding caterpillars outside, and I wanted to document that a bit this year. With the exception of the garden tiger moth (Arctia caja), they all came from my garden.

New from Tow: AI tools are generating ultra-realistic customized podcasts that can be used for a variety of legitimate purposes. The Tow Center tests how they can also create misleading content and propaganda. @srd212.bsky.social www.cjr.org/tow_center/p...

Tow Center analysis finds that publishers face the risk of their content being misattributed or misrepresented regardless of whether or not they allow OpenAI’s crawlers. @klaudia.bsky.social @aisvarya17.bsky.social @columjournreview.bsky.social www.cjr.org/tow_center/h...

Study of ChatGPT citations makes dismal reading for publishers

This paper is definitely a reaction to OpenAI, Google, Microsoft and others pushing chatbots as information access tools (and as such is a follow up to our 2022 paper "Situating Search"): dl.acm.org/doi/10.1145/... >>

Excited to share the piece @klaudia.bsky.social and I wrote for @columjournreview.bsky.social, where we dive into how the tool surfaces and (mis)attributes publisher content. We'd love to hear your thoughts and any questions you have!