👀 Animals have been assigned. Scheduled to print fall 2026! We have iterated on this with over 3k students (and continue to do so). We give our students access to the full draft as part of our evals course (link in bio)

Love that @hamel.bsky.social is putting on a hackathon where the goal is for your agent to score the highest on evaluations, not just do something flashy. click.convertkit-mail2.com/gkumlz753lc5...

If your looking to get started with evals check out this cookbook from @hamel.bsky.social cookbook.openai.com/examples/eva...

"Can I just get an LLM to do my error analysis?" We get this question constantly. The answer is no, and trying is the fastest way to miss critical bugs. Full podcast: youtu.be/BsWxPI9UM4c?...

I recently sat down with Lenny Rachitsky to discuss why AI Evals are becoming the most sought after skill for product builders. As a bonus, we step through an end-to-end example of building an eval in a spreadsheet so everyone can understand. See reply for links.

This one is going to be spicy. 80% of the time I've seen a graph DB in production, it's been an overcomplicated mess (especially in AI applications). In this talk, Jo and I will discuss when GraphDBs are overkill and when they actually make sense. Sign up here:

For technical domains especially, getting non-DSes involved in analyzing outputs is vital. It’s hard to build anything good without it bc v1s almost always have major fail modes. Finding the appropriate system design—let alone optimizing—requires a tight coupling of output analysis and system design

If you're wanted to learn applied AI evals but not sure if its for you, @sh-reya.bsky.social and I put together something that might help. This free email course compiles what we've learned from teaching 2k+ students. It’s 17 emails plus 2 free e-books. Here's the link: ai.hamel.dev/eval-course

I've been eval pilled by @hamel.bsky.social . Everyone is all let's build some MCP servers and ship a minimal AISRE as quickly as possible. And I'm writing a design doc about how to build evals with @runme.dev so we can iterate rapidly on the AI

Last chance to signup for this free lesson with OpenAI on evals, Including a sneak peek of their new eval products! Link: maven.com/p/d2dc30/how...

Can non-data scientists write AI Evals? The answer is nuanced and not just "Yes". @eugeneyan.com and I discuss this in the context of the "analyze-measure-improve" cycle from our course. Links to more resources in the reply

If you are writing evals without error analysis, our course AI Evals for Engineers & PMs is for you. Begins monday next week. Full syllabus in this link: maven.com/parlance-lab...

It is very easy to make mistakes when creating evals for your AI product. @sh-reya.bsky.social and I run through the most common errors in this talk. 35% discount code to our upcoming course in the video notes youtu.be/GL0XhAj5LPE?...

GitHub CoPilot is one of the first commercially successful LLM products (predating ChatGPT). What was the secret? A robust eval suite! In this lightning lesson, John Berryman will reveal the eval techniques (and mistakes) from working on this product maven.com/p/da8264/how...

@hamel.bsky.social & @sh-reya.bsky.social are two of the world's best on evals. They've built evals for 35+ AI apps & helped teams ship confidently. Now they'll teach everything they know on building evals that work. Enrollment closes in 4 days. Secret 35% discount code: maven.com/parlance-lab...

GitHub CoPilot is one of the first commercially successful LLM products (predating ChatGPT). What was the secret? A robust eval suite! In this lightning lesson, John Berryman will reveal the eval techniques (and mistakes) from working on this product maven.com/p/da8264/how...

I keep hearing about the emerging role of AI PM. How is this any different than a normal PM? Is it hype? We are gonna find out in this free lightning lesson. I will ask difficult questions. With @schof.bsky.social and Aman Khan maven.com/p/544677/wha...

I keep hearing about the emerging role of AI PM. How is this any different than a normal PM? Is it hype? We are gonna find out in this free lightning lesson. I will ask difficult questions. With @schof.bsky.social and Aman Khan maven.com/p/544677/wha...

As genAI projects mature, proper evals are becoming table stakes for production deployment. But how do we evaluate probabilistic machines? Looking forward to learning about the latest techniques and best practices from @hamel.bsky.social and Shreya Shankar next month!

Last chance to sign up for this. Recording sent to everyone who signs up. maven.com/p/29a33a/hyb...

@hamel.bsky.social & his wisdom on evals, error analysis, looking at your data is what we need. Here are his 10 Don'ts: • Don't skip error analysis • Don't skip looking at your data • Don't gatekeep who can write prompts • Don't let zero users be a roadblock • Don't be blindsided by criteria drift

If you are building RAG applications, you don't want to miss this. Doug Turnbull is going to show you his tricks he's learned from a decade of optimizing retrieval in search systems, and how that transfers to RAG. Link: maven.com/p/29a33a/hyb...

If you are building RAG applications, you don't want to miss this. Doug Turnbull is going to show you his tricks he's learned from a decade of optimizing retrieval in search systems, and how that transfers to RAG. Link: maven.com/p/29a33a/hyb...

The most critical part of RAG is the R (Retrieval). In this lesson, Doug Turnbull will share how we can go beyond simple hybrid search to optimize retrieval. He'll share his bag of tricks from over a decade of optimizing search systems. maven.com/p/29a33a/hyb...

The most critical part of RAG is the R (Retrieval). In this lesson, Doug Turnbull will share how we can go beyond simple hybrid search to optimize retrieval. He'll share his bag of tricks from over a decade of optimizing search systems. maven.com/p/29a33a/hyb...

I’m excited to teach this lesson on improving retrieval for RAG with @softwaredoug.bsky.social maven.com/p/29a33a/hyb...

Interesting post: A Field Guide to Rapidly Improving AI Products by @hamel.bsky.social

Whenever @hamel.bsky.social drops a new banger of a blog post I make sure to forward it to the rest of my team 🚀 And I encourage everyone working with productionalizing LLM systems to do the same. hamel.dev/blog/posts/f...

Finally got around to trying Answer.ai 's "nbsanity": Works beautifully on the first try! Even renders my interactive Plotly stuff! Just replace "github" in your notebook URL with "nbsanity", as in... nbsanity.com/static/3465a... More info from @hamel.bsky.social : www.answer.ai/posts/2024-1...

In case you missed it, @hamel.bsky.social reviewed Devin. It succeeded on 3/20 assigned tasks.

Thoughts On A Month With Devin (the "AI software engineer") by @hamel.bsky.social "Out of 20 tasks we attempted, we saw 14 failures, 3 inconclusive results, and just 3 successes. More concerning was our inability to predict which tasks would succeed."

Enjoyed the systematic first hand reporting of their experience using Devin by @hamel.bsky.social and team. If you’ve worked with llm coding assistants, the results aren’t surprising, but it points to how far these models still need to go and should be worrying for how effective “agents” will be.

Thoughts On A Month With Devin by @hamel.bsky.social They decided to put it through its paces, testing it against a wide range of real-world tasks. This is their story - a thorough, real-world attempt to work with one of the most hyped AI products of 2024. www.answer.ai/posts/2025-0...

Four steps to use evals effectively in LLM applications (we haven't done the last one but are still getting great results): Eval Driven Development is the new TDD for LLM based applications. Without them, you're flying blind. #cto #llm #ai #tech #dev #genai

New LLM Eval Office Hours, I discuss the importance of doing error analysis before jumping into metrics and tests Links to notes in the YT description youtu.be/ZEvXvyY17Ys?...

This is pretty damn nifty! @hamel.bsky.social @projectjupyter.bsky.social #datascience #jupyternotebooks www.answer.ai/posts/2024-1...

ModernBERT is available as a slot-in replacement for any BERT-like model, with both 139M param and 395M param sizes. It has a 8192 sequence length, is extremely efficient, is uniquely great at analyzing code, and much more. Read this for details: huggingface.co/blog/modern...

Our team at @specstory.com launched our very first product iteration today. What is it? An extension for @cursor_ai that allows you to save and share your composer and chat history. Give it a try at marketplace.visualstudio.com/items?itemNa... and let us know what you think!

Recoded my second office hours on LLM Evals. We talked about observability and how to prioritize writing tests in complex systems Here are the notes: hamel.dev/notes/llm/of... Video: youtu.be/TZwmLXXFbh4?...

Running this notebook from @howard.fm hoping that it removes noise from my timeline nbsanity.com/static/0b3fd...

Make it easier to manually inspect your data! I built a small Shiny for Python web app as recommended by @hamel.bsky.social. I'm getting through my task much faster than previous iterations

I'm proud that we're going public with some positioning on what @honeycomb.io actually believes AI represents: A new, weird, and sometimes janky kind of virtual computer. Stay tuned for a lot more clear-headed posting on applied AI in the coming year. www.honeycomb.io/blog/observa...

Fantastic use of shot-scraper.datasette.io here to. Create social media cards for this new Jupyter Notebook rendering site nbsanity.com

Super exciting update to nbsanity. I've incorporated @simonwillison.net 's shotscraper. Now, all new renders get a fancy social card! This makes nbsanity a nice microblogging utility Examples of rendered notebooks: 1/3 nbsanity.com/static/6a987...

nbsanity now has a bookmarklet nbsanity.com It's a static server that renders public Jupyter notebooks with Quarto

Are you frustrated by how GitHub renders Jupyter notebooks? I have public service that renders GitHub notebooks with Quarto nbsanity.com It now works with gists!

I am holding open office hours on LLM Evals. I recorded the first one which was about evaluating multi-turn chats Notes and recording here: hamel.dev/notes/llm/of...

As R&D staff @ answer.ai, I work a lot on boosting productivity with AI. A common theme that always comes up is the combination of human+AI. This combination proved to be powerful in our new project ShellSage, which is an AI terminal buddy that learns and teaches with you. A 🧵

I updated the list of accounts in the AIEngineering feed (bsky.app/profile/jere...) yesterday by crawling @simonwillison.net and @hamel.bsky.social 's followers. This increased the number of accounts from ~100 to ~1100.

@hamel.bsky.social is 💯: hamel.dev/blog/posts/a... Write for yourself. Assume no one reads it. Write about topics to learn, clarify your thoughts, put out a bat signal. This makes it sustainable. Writing is primarily a single player game; multiplayer benefits (e.g., audience building) are bonuses.