Dream unlocked: I'm publishing my first book! 🎉🎉🎉 It's called "Master Machine Learning with scikit-learn: A Practical Guide to Building Better Models with Python" Download the first 3 chapters right now: 👉 dataschool.kit.com/mlbook 👈 Thanks for your support 🙏

I got 3rd out of 691 in a tabular kaggle competition – with only neural networks! 🥉 My solution is short (48 LOC) and relatively general-purpose – I used skrub to preprocess string and date columns, and pytabkit to create an ensemble of RealMLP and TabM models. Link below👇

This work is presented at ICML next week. • The paper arxiv.org/html/2502.05... • The python package: pypistats.org/packages/tab... (try it out 🐍) • The source code github.com/soda-inria/t... (100% open source, including pre-training 💞) Longer read (5mn): gael-varoquaux.info/science/tabi... 8/9

👨‍🎓🧾✨#icml2025 Paper: TabICL, A Tabular Foundation Model for In-Context Learning on Large Data With Jingang Qu, @dholzmueller.bsky.social, and Marine Le Morvan TL;DR: a well-designed architecture and pretraining gives best tabular learner, and more scalable On top, it's 100% open source 1/9

My thoughts on the current state of AI progress and the most important developments in 2025: www.dataschool.io/ai-progress-...

Are you familiar with Token Pooling? Models that use late interaction, like ColBERT, ColPali, and ColQwen, gain significant benefits from this pooling technique! By integrating token pooling methods, the number of vectors to store can be reduced. Blog: www.answer.ai/posts/colber...

Efficiently scale long CoT models like DeepSeek when using Best-of-N or Majority Voting by early pruning reasoning chains. Kaggle Discussion: www.kaggle.com/competitions...

I find making your agents safe is just as important as making them smart. 🔒 A good read for building secure AI! arxiv.org/pdf/2503.18813

There will be one day ... in 🇺🇸 or 🇻🇳

Claude finally integrated web search into its results... But with LangChain & LangGraph, you can build a chatbot that integrates web search into ANY model you like! You'll learn how to do that (and much more) in my new AI course... Sign up for EARLY ACCESS: 👉 dataschool.kit.com/agents 👈

A practical way for students to secure jobs and earn money is by developing real-world projects. Researching or engineering LLMs often seems like a field dominated by the big tech! It's still important to learn fundamentals from scratch for growth and problem-solving (e.g be able to fix things)! 😁

My next tutorial on pretraining an LLM from scratch is now out. It starts with a step-by-step walkthrough of understanding, calculating, and optimizing the loss. After training, we update the text generation function with temperature scaling and top-k sampling: www.youtube.com/watch?v=Zar2...

cuDF-pandas (%load_ext cudf.pandas) with Rapids ... work similarly and super cool to see we will be able to speed up scikit-learn

Scikit-learn accelerated 🚀 My company has a bunch of unused T4 GPUs because the LLMs are too big for AI teams run exps. Now the data science team finally has a reason to ask for them! 🤣 developer.nvidia.com/blog/nvidia-...

In honor of March Madness 🏀, I've got a new blog post: www.dataschool.io/pandas-strea... Learn how to identify & analyze scoring streaks using pandas operations: - shift() - cumsum() - boolean math - groupby()

Many good advices/best practices for missing value imputation in the paper! I now have a much deeper appreciation for Data School's course and regard it as the best scikit-learn course. Master Machine Learning with scikit-learn: courses.dataschool.io/master-machi...

"Some people today are discouraging others from learning programming on the grounds AI will automate it. This advice will be seen as some of the worst career advice ever given." -- Andrew Ng, legendary AI researcher Source: www.deeplearning.ai/the-batch/is...

A recent talk, fully in a vscode: 100% code on data wrangling for machine learning with @skrub-data.bsky.social www.youtube.com/watch?v=hdWW... super powerful to easily assemble production-ready pipelines in easy syntax

Yesterday, Google released Gemma 3, their latest open-weight LLM. Finally, a new addition to the "Big 5" of open-weight models (Gemma, Llama, DeepSeek, Qwen, and Mistral). I just went through the Gemma 3 report and experimented a bit with the models, and there are plenty of interesting tidbits:

Just uploaded my "Coding Attention Mechanisms" tutorial. A 2h15m session on coding attention mechanisms to understand how the engine of LLMs works: self-attention → parameterized self-attention → causal self-attention → multi-head self-attention www.youtube.com/watch?v=-Ll8...

I just shared a new article, "The State of Reasoning Models", where I am exploring 12 new research articles on improving the reasoning capabilities of LLMs (all published after the release of DeepSeek R1): magazine.sebastianraschka.com/p/state-of-l... Happy reading!

A couple months ago @dataschool.io wrote about a tool he uses to chat with different LLM models without paying a monthly subscription to all of them. The tool is called Typing Mind and I decided to pay $30 for lifetime access. It was well worth it. Kevin's post 👇 www.dataschool.io/save-money-o...

19 professionals (in a variety of fields) evaluated OpenAI's Deep Research vs Google's Deep Research. OpenAI was the clear winner 🏆 Neat study by @binarybits.bsky.social, read more here: www.understandingai.org/p/these-expe...

A new tutorial in my “Build A Large Language Model From Scratch” series is now live (www.youtube.com/watch?v=341R...) - Tokenizing raw text and converting tokens into token IDs - Applying byte pair encoding - Setting up data loaders in PyTorch for efficient training

Comprehensive !! mlcontests.com/state-of-mac...

Another great read on reasoning models! 🧠 Small LMs struggle to learn from long or complex CoTs from larger teachers. 🔍 Why? The reasoning complexity may be too overwhelming. 🚀 Solution: Mix simple & complex CoTs! 📈 Results: Clear gains over complex CoTs alone! Arxiv: arxiv.org/pdf/2502.121...

OMGGG!!!!!!!!!!! 100+ pages book to fully understand distributed training. Author: "🎯 Our goal: democratize knowledge about LLM training at scale. Whether you're starting with 1 GPU or orchestrating thousands, this guide walks you through the journey step by step." huggingface.co/spaces/nanot...

Here's my personal guide to Bluesky happiness 🌈 1. Be very selective about who I follow (42 people at the moment) 2. Read everything in the "Following" feed, and nothing else 3. Trust that the people I follow will surface other good people to follow 4. Unfollow anyone who no longer makes the cut

It's 2025, and I’ve finally updated my Python setup guide to use uv + venv instead of conda + pip! Here's my go-to recommendation for uv + venv in Python projects for faster installs, better dependency management: github.com/rasbt/LLMs-f... (Any additional suggestions?)

Wondering about the differences between "Jupyter", "Jupyter Notebook", "JupyterLab", "IPython", "Colab", and other related terms? I'll explain these terms (and more) in 5 minutes: ▶️ www.youtube.com/watch?v=TDlG...

Reading this paper, you feel the power of RL. RL is great, but not all we need. A blend of RL, inference time scaling, good pretraining, and domain-specific tuning is often the way to go! arxiv.org/pdf/2502.06807

LLM Trick - Finding the Near Optimal Temperature for Multi-Sample Inference (e.g., Majority Vote, Best-of-N) - Notebook (implement paper): 🤖https://github.com/ducnh279/Optimize-Temperature-for-LLM-Inference/blob/main/optimize-temperature-for-llm-inference.ipynb

When I first started learning about LLMs, I used this cheat sheet (community.openai.com/t/cheat-shee...) to choose sampling "temperature." Now, I’ve found another useful technique to add to my bag of tricks: arxiv.org/pdf/2502.05234.

Have you ever wondered what are the differences between 2 types of knowledge distillation (for LLMs) these days? Link: developer.nvidia.com/blog/how-to-...

My learning project: Implement a Reasoning Model with GRPO from scratch Code: github.com/ducnh279/grp... - Base model: Qwen2.5-1.5B - Dataset: GSM8K (math) - Reward functions: Format, Accuracy - Objective: GRPO - PEFT: LoRA

Maybe a hot take, but what about the following advice to the next gen: Don't get an AI degree; the curriculum will be outdated before you graduate. Instead, study math, stats, or physics as your foundation, and stay current with AI through code-focused books, blogs, and papers.

𝐂𝐨𝐬𝐢𝐧𝐞 𝐫𝐞𝐰𝐚𝐫𝐝 stabilizes training & incentivizes the model to generate responses that are both accurate and reasonably lengthy. To prevent reward hacking, a repetition penalty is added to the cosine reward. Paper: arxiv.org/pdf/2502.03373 Hackable Code: github.com/eddycmu/demy...

Comparing: - Software Developers - ML Engineers - LLM Developers - Prompt Engineers Read here: www.louisbouchard.ai/llm-develope... Image & article by Louis-François Bouchard

Just read the s1: Simple Test-Time Scaling paper. Super interesting approach to improving reasoning models! TL;DR: 1. SFT on 1k curated examples w/ reasoning traces. 2. Control response length w/ budget forcing: "Wait" tokens → longer reasoning/self-correction. "Final Answer:" → enforce stopping.

I just finished writing up my take on reasoning models: magazine.sebastianraschka.com/p/understand... Here, I 1. Discuss the advantages & disadvantages of reasoning models 2. Of course, describe and discuss DeepSeek R1 3. Describe the 4 main ways to building & improving reasoning models

The best way to understand GRPO is to code it from scratch without HF Trainer. 😂

Want to start with RL basics for training reasoning LLMs using GRPO? Check out: superb-makemake-3a4.notion.site/group-relati...

This blog post is now available in video form: www.youtube.com/watch?v=rOKX... (Hopefully I did a good job pronouncing Andrew Ng & Sebastian Raschka 🤷‍♂️)

Want to keep up with the latest developments in AI? Check out the 7 newsletters I personally read & recommend for AI news, insights, and analysis: 👉 www.dataschool.io/best-ai-news... 👈 Includes: @simonwillison.net @emollick.bsky.social @sebastianraschka.com @binarybits.bsky.social

But before I get to the reasoning model space... if you are looking to do some focused offline reading this weekend, I just re-compiled my take on the "noteworthy AI research papers of 2024" into one PDF-export-friendly 47-page mega-post with TOC and all: sebastianraschka.com/blog/2025/ll...

Wondering what you can build with LangChain & LangGraph? Browse the @langchain.bsky.social feed for inspiration! Here's an example 👇 LangChain & LangGraph are both powerful and complex, but my new course gets you going in under an hour: courses.dataschool.io/build-an-ai-...

🚀 Launching now: "Build an AI chatbot with Python" In this course, you'll create your first AI app in 60 minutes using LangChain & LangGraph ⚡ Why choose this course? - Beginner-friendly - Hands-on learning - Up-to-date technology Get lifetime access for $9: courses.dataschool.io/build-an-ai-...

So exciting!!! My new AI course launches in 24 hours 🚀 Sign up here if you want to get notified by email: courses.dataschool.io/subscribe

Deepseek R1 models - huggingface.co/deepseek-ai/... "Performance on par with OpenAI-o1" 🔥🔥🔥