A common problem I see is people expecting models to solve all their problems in a single try. Instead treat it as an iterative process and work slowly towards building a portfolio of good outputs and reviewing them consistently ivanleo.com/blog/youre-p...

My favourite evaluation is an assert statement and it works surprisingly well. Stupid evals helps us turn vibes into objective metrics - one step at a time. See my favourite stupid evals at ivanleo.com/blog/write-s...

Evals are "too damn expensive" until you: • can't migrate underlying models safely • can't add new features with confidence • can't ship without HITL evals, which takes >100x longer • product development and iteration grinds to a halt • lose customer trust due to poor user experience

Wrote up my thoughts on synthetic data generation www.ivanleo.com/blog/synthet...

Spent some time writing about my takeaways from running loads of experiments as a new data scientists working with LLMs www.ivanleo.com/blog/youre-p...