Learning from feedback is instrumental but human preference data can be expensive. How much reward supervision could we get from raw web text instead, without human labels? Our work, a year of incredible effort by @fjxdaisy.bsky.social, advances pure RLHF training across multiple models and tasks.

Introducing EvoLM, a model suite with 100+ decoder-only LMs (1B/4B) trained from scratch, across four training stages — 🟦 Pre-training 🟩 Continued Pre-Training (CPT) 🟨 Supervised Fine-Tuning (SFT) 🟥 Reinforcement Learning (RL)

New work [JSKZ25] w/ Jikai, Vasilis, @shamkakade.bsky.social . We introduce new formulations and tools for evaluating LM capabilities, which help explain observations of post-training behaviors of Qwen-series models. More details: - hanlin-zhang.com/causal-capab... - x.com/_hanlin_zhan...

Highlights from #ICLR2025 — a brief thread 🧵

I want to reshare @brandfonbrener.bsky.social's @NeurIPSConf 2024 paper on CoLoR-Filter: A simple yet powerful method for selecting high-quality data for language model pre-training! With @hlzhang109.bsky.social @schwarzjn.bsky.social @shamkakade.bsky.social

(1/n) 💡How can we speed up the serial runtime of long pre-training runs? Enter Critical Batch Size (CBS): the tipping point where the gains of data parallelism balance with diminishing efficiency. Doubling batch size halves the optimization steps—until we hit CBS, beyond which returns diminish.

LLM self-improvement has critical implications in synthetic data, post-training and test-time inference. To understand LLMs' true capability of self-improvement, we perform large-scale experiments with multiple families of LLMs, tasks and mechanisms. Here is what we found: (1/9)