Manifold-Constrained Hyper-Connections [Xie+, 2026] HC expands Transformer's residual stream width to improve expressivity but breaks the identity mapping. mHC constrains residual mappings to be doubly stochastic to stabilize training and enable 27B-scale experiments. arxiv.org/abs/2512.248...

2025年に読んだ本をまとめました。 読んだ本とともに2025年を振り返る | Hippocampus's Garden hippocampus-garden.com/books_2025/

Wrapped up 2025 with a post: my top 5 AI papers of the year. DeepSeek-R1, RLVR limitations, agent benchmarks, long-task metrics, and why LMs hallucinate, plus pointers to related research and write-ups. Top 5 AI Papers of 2025 | Hippocampus’s Garden hippocampus-garden.com/ai_papers_20...

Does RL incentivize reasoning capacity in LLM? [Yue+, 2025, NeurIPS] By evaluating pass@k at large k, the paper shows that RLVR improves sampling efficiency but doesn't introduce new reasoning patterns. Distillation expands the boundary by injecting the teacher distribution. arxiv.org/abs/2504.13837

AIによって知識がコモディティ化していく時代において、人間の価値がどこで生まれるのかを、フロンティア知識と行動のプレミアムという観点から整理しました。 知識のコモディティ化と行動のプレミアム | Hippocampus's Garden hippocampus-garden.com/commodity_kn...

Weight-sparse transformers have interpretable circuits [Gao+, 2025] Training a Transformer with L0 norm fixed—so that most weights are zero—yields disentangled circuits. The role of each weight can be identified and visualized for simple tasks like closing quotation marks. arxiv.org/abs/2511.13653

An Open-Ended Embodied Agent with LLMs [Wang+,2024, TMLR] Voyager is an embodied agent in Minecraft powered by GPT-4. It learns skills (executable code for performing complex actions), guided by a curriculum generated by LLM to encourage exploration. arxiv.org/abs/2305.16291 #NowReading

Evaluating creative reasoning with Sudoku variants [Seely+, 2025] This paper proposes Sudoku-Bench, a memorization-proof benchmark using Sudoku variants to test LLMs’ insight and multi-step reasoning, with expert solving traces from YouTube included in the dataset. arxiv.org/abs/2505.16135

Recursive Reasoning with Tiny Networks [Jolicoeur-Martineau, 2026] Tiny Recursion Model is a 7M-param network that performs internal recursive reasoning, eliminating the complexity of HRMs. With supervised training, it outperforms LLMs on Sudoku and ARC-AGI. arxiv.org/abs/2510.048... #NowReading

Chasing leaderboard scores is a trap. We reproduced Tau-bench across different models and learned: - Hidden configs make apple-to-apple comparisons impossible - Reasoning boosts accuracy, but also latency That’s why you need an internal benchmark. medium.com/alan/benchma...

Wrote a short piece on why pass@k and pass^k tell different stories than mean and variance. If you enjoy thinking about evaluation quirks and probability, you might like it. Pass@k and Pass^k Tell Different Stories from Mean Success Rate | Hippocampus's Garden hippocampus-garden.com/pass_k/

A Sober Look at LM Reasoning [Hochlehnert+, 2025, COLM] The authors evaluated 7B-class open models on math tasks and found that performance varied significantly with decoding parameters (random seeds, temperature etc). Many RL methods failed to reproduce reported gains. arxiv.org/abs/2504.070...

What’s the difference between evaluating LLMs vs. AI agents? Why is it so hard to assess AI agents in real business settings? I’ve written my thoughts and a possible path forward. I plan to keep writing about this topic—please share your thoughts or feedback! medium.com/alan/benchma...

Evaluating AI on real-world economically valuable tasks [Patwardhan+, 2025] OpenAI created a new benchmark, GDPval, asking experts from 9 industries that contribute to the US GDP. The current top model is Claude Opus 4.5, nearing the standards of industry experts. openai.com/index/gdpval/

Meta Agents Research Environments [Andrews+, 2025] ARE is a platform that facilitates async interactions between agents and the environment (including real apps). The authors then upgraded the GAIA benchmark to Gaia2, which runs in a virtual mobile environment built on ARE. arxiv.org/abs/2509.17158

Benchmark for General AI Assistants [Mialon+, 2023] GAIA evaluates AI's fundamental abilities, such as web search, image understanding, and file reading, rather than expert knowledge. While humans achieve 92%, GPT-4 does 15% (30% with plugins). arxiv.org/abs/2311.12983

Some reasoning models disable temperature and top‑p. This short post explains how sampling works and why those knobs are off. Why You Can’t Set Temperature on GPT-5/o3 | Hippocampus's Garden hippocampus-garden.com/llm_temperat...

Why Language Models Hallucinate [Kalai+, 2025] LM pre-training is essentially density estimation and doesn't help prevent hallucination. Post-training can alleviate the issue by rewarding acknowledging uncertainty over guessing, but current benchmarks are not designed so. arxiv.org/abs/2509.04664

Stress Testing MCP-enabled Agents [Yin+, 2025] LiveMCP-101 is a benchmark of 101 real-world tasks that require the coordinated use of multiple MCP tools. As results could change across attempts, it also evaluates the resolution path. GPT-5 achieves a success rate <60%. arxiv.org/abs/2508.15760

Evaluating Conversational Agents in a Dual-Control Environment [Barres+, 2025] To simulate a user who doesn't have perfect information, τ^2-Bench gives tools to the user agent. The new telecom domain is more challenging than the existing airline and retail sectors. arxiv.org/abs/2506.07982

"Should I put this tool description in the prompt or pass it as a parameter?" I've been asked this question a few times, so I wrote a new post. It's not obvious that inside LLMs, everything becomes one token stream. Here's how it actually works 👇 hippocampus-garden.com/llm_serializ...

Subliminal Learning [Cloud+, 2025] This paper studies a phenomenon where LMs transmit behavioral traits (e.g., liking owls) via unrelated data (e.g., a sequence of numbers) during distillation. It doesn't happen when the teacher and student have different base models. arxiv.org/abs/2507.148...

Open Agentic Intelligence [Kimi Team, 2025] Kimi K2 is an MoE with 32B/1T params, which achieves the SOTA performance among open-source non-thinking models. It's powerful in agentic tasks thanks to the RL in an agentic environment. MuonClip helps the stable training. arxiv.org/abs/2507.20534

ACEBench [Chen+, 2025] ACEBench is a benchmark designed to assess LLM's tool calling capability. The dataset is generated by an LLM-based pipeline that includes incomplete instructions and real-world APIs. It compares the model’s tool call outputs with the ground truths. arxiv.org/abs/2501.128...

We developed an AI agent that dynamically investigates insurance claims, automating 30% of support tickets. See how we built it leveraging the tool calling capability👇 Inside Alan’s Claim Agent: How an AI Investigates Claims just like a Human, only Faster medium.com/alan/inside-...

Yet another real-world example of an AI agent that does the job.

『コンピュータの構成と設計』（通称「パタヘネ本」）のまとめと感想です。 書評『コンピュータの構成と設計』 | Hippocampus's Garden hippocampus-garden.com/book_review_...

We often treat databases as a black box, but understanding what’s inside is crucial for becoming a good software engineer. I just finished reading Database Internals and published a review + summary! hippocampus-garden.com/book_review_...

Measuring AI Ability to Complete Long Tasks [Kwa+, 2025] "50%-task-completion time horizon" measures human experts' time to complete tasks that AI models can complete with a 50% success rate. The time horizon for software engineering tasks has been doubling every 7 months. arxiv.org/abs/2503.14499

Developing is the best way to learn. I built an MCP server to connect Claude to Google Sheets — and wrote a guide so you can build your own too! How to Customize Claude with your own MCP Server | Hippocampus's Garden hippocampus-garden.com/claude_mcp/

Benchmark for Tool-Agent-User Interaction in Real-World [Yao+, 2024] τ-bench evaluates LLM agents' human interaction and rule adherence in real-world domains (retail and airline)—key aspects missing from existing evaluations. It also proposes pass^k to measure consistency. arxiv.org/abs/2406.12045

New post on Alan's tech blog. I explained how DeepSeek R1 uses pure reinforcement learning to boost reasoning (no reward models needed). Plus, R1-inspired projects and quick insights from our tests! DeepSeek R1: Demystifying LLMs’ Reasoning Capabilities. medium.com/alan/deepsee...

In late January, DeepSeek shocked the world by dropping an open-weight successor to OpenAI's o1: R1. Their tech report discusses how to incentivize reasoning capability in LLMs. We share our learnings at:

Just finished "AI Engineering" by Chip Huyen—an excellent guide to building AI applications with foundation models. I love how it ties technical and product thinking together, something many technical books miss! hippocampus-garden.com/book_review_...

Simple test-time scaling [Muennighoff+, 2025] The authors reproduced the test-time scaling curve by fine-tuning Qwen2.5-32B with s1K, a set of reasoning traces generated by Gemini. They controlled the response length by appending “Wait” when the model tried to answer early. arxiv.org/abs/2501.193...

Incentivizing Reasoning Capability in LLMs via RL [DeepSeek-AI, 2025] R1-Zero acquired o1-level reasoning capability by directly applying GRPO, which rewards correct answers, to the base model. The team also released the weights of R1 with SFT and distilled models. arxiv.org/abs/2501.129...

📈 In 2024, we automated 20% of customer contacts using AI, while keeping customer satisfaction intact! Check out our latest blog for the behind-the-scenes👇 Bonus: our Marmot image generator is good at joking. Look at the iMac with Windows installed 😂 medium.com/alan/how-we-...

In 2024, we automated 20% of customer contacts with AI while maintaining the same customer satisfaction (the number is still growing!) 🚀 Learn about our journey:

年末年始は、自分のキャリアを見つめ直す良い機会。ということで、今後5年程度の未来を見据え、AI時代を生き抜くための方法について考えていたことをまとめました。 皆さんの生存戦略もぜひ教えてください！ AI時代をどう生き抜くか | Hippocampus's Garden hippocampus-garden.com/ai_and_emplo...

I've wrapped up 2024 with my top 10 favorite deep learning papers of the year! 🔍 Dive into the breakthroughs, insights, and ideas that shaped the field. Year in Review: Deep Learning Papers in 2024 | Hippocampus's Garden hippocampus-garden.com/deep_learnin...

Language Modeling in a Sentence Representation Space [Barrault+, 2024] Large Concept Models predict sentence embeddings instead of tokens. They inherently generalize to languages that their embedding model can and are good at handling long context (e.g., summarization) arxiv.org/abs/2412.08821

The Road Less Scheduled [Defazio+, 2024, NeurIPS] Schedule-Free achieves Pareto-optimal loss curves without any LR scheduling. The theory behind it unifies iterate averaging (theoretically good) and scheduling (empirically strong) , taking the best of both worlds. arxiv.org/abs/2405.15682

Movie Gen [Polyak+, 2024] Movie Gen is a group of Transformer-based Flow Matching models that can collectively generate and edit videos with audio, setting a new SOTA on various tasks. Videos are patchfied by spatiotemporal autoencoders. arxiv.org/abs/2410.13720

RLHF is not the only method for AI alignment. This post introduces modern algorithms like DPO, KTO, and DiscoPOP that offer simpler and more stable alternatives. Evolution of Preference Optimization Techniques | Hippocampus's Garden hippocampus-garden.com/preference_o...