Thrilled to share this accessible MSR blogpost that summarizes our latest work on building a Science of AI Evaluation, where we manage to both reliably explain and predict success/failure of general-purpose AI models on new, unforeseen tasks and environments!

🚨To continuously foster conceptual & technical innovations for a science of AI Evaluation: An open collaborative community is initiated by Leverhulme Centre for the Future of Intelligence, to adopt and extend our novel methodology. Join us: kinds-of-intelligence-cfi.github.io/ADELE!

To better understand why this matters in high-stakes contexts, you can also check out our previous work. We discuss why predicting model performance (e.g., failures on out-of-distribution languages in machine translation) remains essential in legal contexts.

Understanding and extrapolating benchmark results will become essential for effective policymaking and informing users. New work identifies indicators that have high predictive power in modeling LLM performance. Excited for it to be out!

Thrilled to unlock AI Evaluation with explanatory and predictive power through general ability scales! With a new methodology to -Explain what common benchmarks really measure -Extract explainable ability profiles of AI systems -Predict performance for new task instances, in & out-of-distribution 🧵