I'm in Seoul for ICML 2026!🇰🇷 If you’re interested in evals, coding and long-horizon agents, or issues such as contamination, reward hacking, and model cheating, please stop by our posters (details below) or say hi if you see me around, I'd love to chat!

Excited to share CoffeeBench!!☕️☕️☕️ We evaluate LLM agents in a 90-day B2B coffee supply-chain economy spanning farmers, roasters, and retailers, where autonomous firms negotiate, manage inventory, set prices, handle invoices, and manage cash flow. arxiv.org/abs/2606.16613 github.com/sakanaai/cof...

Coding agents are evaluated with unit tests: more tests passed = better model. But if tests or feedback are accessible, models may learn to game them. We introduce CapCode to detect suspiciously high scores, and CapReward to discourage them during RL. 🧵1/10

I will be at ICLR 2026 this week! @iclr-conf.bsky.social I would love to chat with people with similar research interests. Please check out our posters at Session 6, Pavilion 3 & Pavilion 4, on Saturday, April 25. See you there!

If you are at ICLR 2026 this week and are interested in model evaluation and financial benchmarks, please stop by our poster at Session 6, Pavilion 3, on Saturday, April 25! 👋🇧🇷

I am very proud of our team for releasing EDINET-Bench, and it is fantastic to see a Japanese financial dataset recognized at #ICLR2026 this week. We need more diverse, non-English datasets to evaluate models in the real world. Paper: openreview.net/forum?id=Dxn...

Tired of KL penalties constraining your model? But don't want your policy to just hack the reward? Try Gradient Regularization! We show it beats a KL penalty in RLHF, RLVR and LLM-as-a-Judge! 🧵1/7

📣We are excited to launch the CapBencher toolkit today! CapBencher caps the best achievable accuracy on purpose. If an LLM scores above the cap, it’s a red flag for leakage, contamination, or leaderboard gaming🚩 If you are creating a new benchmark, you might find it useful👉

Happy to share that our papers were accepted to ICLR 2026!🇧🇷 Big thanks to my co-authors! Scalable oversight: arxiv.org/abs/2510.22500 EDINET-Bench: arxiv.org/abs/2506.08762 Optimal classification error estimation: arxiv.org/abs/2505.20761

We're excited to announce the launch of Google Developer Group AI for Science Japan!🎉 If you're interested, we’d love to have you join our community. GDG AI for Science Japan gdg.community.dev/gdg-ai-for-s...

Reward models do not have the capacity to fully capture human preferences. If they can't represent human preferences, how can we hope to use them to align a language model? In our #COLM2025 "Off-Policy Corrected Reward Modeling for RLHF", we investigate this issue 🧵

Released bibfixer 🎉 A tiny AI tool that cleans & standardizes your BibTeX files using LLMs + web search. No more tedious edits like fixing capitalization (ai -> AI), swapping arXiv for the conference version, or expanding "and others" into full author lists. Let bibfixer do the grunt work for you!

EDINET-Bench: Evaluating LLMs on Complex Financial Tasks using Japanese Financial Statements Paper: pub.sakana.ai/edinet-bench/ We just released a Japanese financial benchmark designed to evaluate the performance of AI Agents on challenging financial tasks like accounting fraud detection.

Excited to announce EDINET-Bench, a financial LLM benchmark built from 40k annual reports in Japan! It features accounting fraud detection, earnings forecasting, industry classification, and includes our tool edinet2dataset as a foundation for designing new tasks. Hope researchers find it useful!

Our discussion period just started. Authors, please read our instructions carefully. We require responses by June 2. But, what you really want to hear about is stats .... right? -> 🧵