I’ve seen semantic IDs pop up but never bothered to actually look into them. This write up from @eugeneyan.com is a great intro that also illustrates why they’re pretty interesting for mixing recsys and LLMs eugeneyan.com/writing/sema...

I've been nerdsniped by the idea of Semantic IDs. Here's the result of my training runs: • RQ-VAE to compress item embeddings into tokens • SASRec to predict the next item (i.e., 4-tokens) exactly • Qwen3-8B that can return recs and natural language! eugeneyan.com/writing/sema...

Wrote an intro to evals for long-context Q&A systems: • How it differs from basic Q&A • What dimensions & metrics to eval on • How to build llm-evaluators • How to build eval datasets • Benchmarks: narratives, technical docs, multi-docs eugeneyan.com/writing/qa-e...

Wrote an intro to evals for long-context Q&A systems: • How it differs from basic Q&A • What dimensions & metrics to eval on • How to build llm-evaluators • How to build eval datasets • Benchmarks: narratives, technical docs, multi-docs eugeneyan.com/writing/qa-e...

Some thoughts on leadership: eugeneyan.com/writing/lead... • What makes an exceptional leader? • What do exceptional leaders do? • Leadership styles: Commando, soldier, police

The best leaders I’ve worked with operate with perma-urgency. They act like early founders, mindful of existential threats. And they can balance speed, sustainability, and repay tech debt. Ultimately, customers love it and teams thrive when we ship fast to deliver delight.

Had a fun couple of hours this weekend with Codex & Windsurf • Migrated off deprecated jekyll-algolia to official sdk (better indexing) • Added recommendations + relevance scores to each post • Improved site responsiveness; fixed dark mode flicker • Marie Kondo-ed unused files & dead code

In orgs pushing the envelope, there's always a minority that can be counted on to get shit done against all odds, driven by force of will, resourcefulness, influence, etc. When you identify them, vest in them authority, autonomy, and step back and watch them perform miracles.

To better understand MCPs and agentic workflows, I built news-agents to generate a daily news recap. The main agent spawns sub-agents, assigning them news feeds to parse and summarize, and then generates a final overall summary plus analysis. eugeneyan.com/writing/news...

@hamel.bsky.social & @sh-reya.bsky.social are two of the world's best on evals. They've built evals for 35+ AI apps & helped teams ship confidently. Now they'll teach everything they know on building evals that work. Enrollment closes in 4 days. Secret 35% discount code: maven.com/parlance-lab...

The Art of Doing Science and Engineering: Learning to Learn by Richard Hamming only $1.99 for the Kindle version today: amazon.com/dp/B088TMLQDC

Enjoyed this on eval-driven product development from @eugeneyan.com. It chimes with my own experiences building around LLMs and search engines, including the thoughts on automated evaluators. When deconstructed, EDD is just the good old scientific method under a new name

Surround yourself with people whose "work" is their calling, craft, and play. They are intrinsically motivated, are driven to excel and do what's right, and and get so much shit done just because it's fun.

Some of the anti-AI stuff feels a bit like when people would say "don't use Wikipedia as a source." It's just like anything else, a piece of information that you weigh against multiple sources and your own understanding of its likely failure modes

Product evals are misunderstood. Many teams think that adding another tool, metric, or llm-as-judge will solve all their problems and save their product. But that just dodges the hard truth and avoids the real work. Here's how to fix your process instead. eugeneyan.com/writing/eval...

The default state of projects is to drift toward entropy; you need to actively resist & reverse it.

Interesting paper from Google that challenges a core assumption in translation evaluation—a single metric can measure both accuracy & naturalness. They found that the best systems had neural metrics that did not correlate with human preferences. arxiv.org/abs/2503.24013

Had a session with very senior folks on how they build with AI and can’t help thinking there’s no better time to learn, clarify, brainstorm, write, debate, plan, design, code, debug, review, analyze, delegate, play, and in general do more more while doing less with AI—so psyched!

Great list of what the best devs do, such as: • Read the source, docs, error msgs • Simplify problems, write simple code • Get their hands dirty • Write to share & write well • Have beginner's mind & keep learning • Not afraid to say: I don't know endler.dev/2025/best-pr...

@hamel.bsky.social & his wisdom on evals, error analysis, looking at your data is what we need. Here are his 10 Don'ts: • Don't skip error analysis • Don't skip looking at your data • Don't gatekeep who can write prompts • Don't let zero users be a roadblock • Don't be blindsided by criteria drift

Great example of generate -> validate loop + error analysis > "the most effective route to improve outcomes was brute force: retry steps until they passed or reached a limit. We give the validation errors ... to the LLM and built a loop runner"

This is a great list, things that “the best engineers I know” do, stuff like: - understanding things deeply, reading the actual source - being willing to help other people - status doesn’t matter, good ideas come from anywhere endler.dev/2025/best-pr...

Stumbled on the first(?) RAG in NarrativeQA from 2017. Because books & movies were too large for LSTMs to do Q&A on, they embedded 200-word chunks and retrieved similar snippets to answer questions. "Chunking and cosine similarity retrieval is so 2017." arxiv.org/abs/1712.07040

If you were building a Q&A feature (or chatbot) based on very long documents (like books), what evals would you focus on?

Can't wait for when I can vibe code a production recommender system. Until then, here's some system designs: • Retrieval vs. Ranking: eugeneyan.com/writing/syst... • Real-time retrieval: eugeneyan.com/writing/real... • Personalization: eugeneyan.com/writing/patt...

Your favorite AI writer's favorite AI writer

I often get asked: How did I start writing? Why do I write? Who do I write for? What's my process? I procrastinated on this because, honestly, who cares about my writing process? But after repeatedly answering the same qns, I finally wrote this. eugeneyan.com/writing/writ...

Vibe coding works—but only if you know what good code looks like.

What are your favorite resources on translating long documents with LLMs? Generating -> validating -> regenerating translations, etc. Also, identifying defects where gender pronouns, formality, idioms, etc are mistranslated, such as for Spanish, German, etc. Please share! 🙏

Search & recsys have historically drawn inspiration from language modeling, such as Word2vec for item embeddings, and GRUs and BERT for sequential models. LLMs are no different and have led to updates in model architecture, training data & paradigms, etc. eugeneyan.com/writing/recs...

If you look at most of the models we've received from OpenAI, Anthropic, and Google in the last 18 months you'll hear a lot of "Most of the improvements were in the post-training phase." Here's a simple analogy for how so many gains can be made on mostly the same base model:

Been querying gpt-4.5 and it's better in ways we can't quantify yet: creativity, humor, world knowledge, wisdom, nuance, based, etc. Excited about how we'll discover new ways to evaluate gpt-4.5 on these aspects which will also transfer to product / application related evals

At dinner with some builders ytd, we shared how we're using AI Agents. This got me curious about how you're using Agents—please share so I can learn from you🙏 I'll start: I'm using deepresearch more for knowledge discovery/decisions and am trying openhands for feature development. What about you?

I want to follow more ML/AI people on bluesky. Who is awesome?

anthropic = protoss; openi = terran; meta = zerg

agent ≈ model + tools, within a for-loop + environment

one of the greatest joys in life is to be with loved ones that you love more than yourself

now that i'm back in seattle, nothing beats the lullaby of my wife and dog's gentle snoring while in my own bed

Almost all major projects I worked on started by 2-3 people sitting in the same meeting room and coding while debating for hours every day without any other distraction.

What industrial recsys papers have you enjoyed or found useful in the past year or two? Sharing my list: # 1. Integrating LLMs into recsys 1.1. LLM-augmented recommenders • Better Generalization with Semantic IDs: A Case Study in Ranking for Recommendations - arxiv.org/abs/2306.08121

What industrial recsys papers have you enjoyed or found useful in the past year or two? Sharing my list: # 1. Integrating LLMs into recsys 1.1. LLM-augmented recommenders • Better Generalization with Semantic IDs: A Case Study in Ranking for Recommendations - arxiv.org/abs/2306.08121

♥️ it's tricky to separate what i do on the job (at the bookstore i work at) and what i hack on in my personal time. out of abundance of caution, to not discuss possible proprietary info, i won't be sharing more about the backend of aireadingclub.com 😔

Thanks to the hundreds of readers who've tried aireadingclub.com and interacted with Dewey. If you've tried aireadingclub and have feedback, feature ideas, or thoughts on how AI can help you get more out of reading, please comment or dm me 🙏

Few understand this jvns.ca/blog/2014/06...

> Nobody tells you the variables you should be regressing. What's the target? What's the source? Do you notice when results are rubbish? ... That's why I think you need smart people who appear to do something technically easy but actually not so easy. news.ycombinator.com/item?id=1906...

How can AI make reading more enjoyable? What would an AI-powered reading experience look like? Over the holidays, I prototyped aireadingclub.com to explore ideas like: • Context understanding • Creating quizzes • Recap the book so far • Look up & summarize a term

• Work hard • Keep learning • Cherish loved ones • Find people who inspire you • Be kind & egoless • Eat healthy, exercise, sleep well • Read & write • Practice gratitude & meditate • Be present • Enjoy food & nature • Don’t sweat the small stuff • Smile =)

let go of your wants and desires. make space and attention for the present, and for the natural abundance and opportunities to flow into your life.

Thank you, glad it was helpful! Some resources: • Task-specific evals: eugeneyan.com/writing/evals/ • LLM-evaluators: eugeneyan.com/writing/llm-... • Aligning people to data; aligning AI to humans: eugeneyan.com/writing/alig... • Finetuning for hallucination detection: eugeneyan.com/writing/fine...