Just launched the Wikidata dumps parsing. No more refactors, time for big plans.

Maybe it’s me but being a mess of different historical periods feels pretty on brand for a Homeric adaptation.

Funnily enough, my partner just made the same point to a semi-well-known French researcher complaining about widespread training on copyrighted data. And this was the answer.

AI discourse on the good site. AI discourse on the bad site.

French investigative media Mediapart rediscovering water apparently: been known for +1 year, when Meta entered discovery and since then Meta won the case.

I recently started playing Chopin Sonata 3 again and there was a puzzling mystery: my favorite recording (Arrau) diverges significantly from the sheet music. Most obvious change is this passage, completely rewritten to sound much more disjointed/modernist.

My take on this: science is already "automated" to a large extent and this goes back to early digitization if not prior. Even our current concept of peer review only came to be widespread with review automation (was one of the core tech of Elsevier in the 1970s).

Pour ce prix-là on entraîne plusieurs LLMs à l’état de l’art. GPT-OSS c’était trois millions, DeepSeekv3 10, Sonnet-Opus quelques dizaines.

Not entirely sure what took me to start a +1000 pages novel about USSR, but it really grows on me. War scenes in the Moscow subway are incredibly set: cinematic reading.

New DeepSeek report is also about synthetic pipelines.

Continued with this one: hypnotic tale, still drawing from Kafka, but surprisingly foreshadowing Philip K. Dick (the snow is a hallucinogenic mushroom). Also a political tale, which is less surprising given the first edition: Germany, 1932.

I mean, it's a language model, how big should it be? 1 million parameters?

DeepSeek just released a new state of the art math prover, DeepSeek-Math-V2, competitive with Google, OpenAI or ByteDance, while being a publicly documented open weight models. A few reading notes along the way:

And a major open science release from Prime Intellect: they don’t stress it enough but SFT part is beyond post-training. This is a fully documented mid-training with tons of insights/gems on MoE training, asynchronous infra RL, deep research. storage.googleapis.com/intellect-3-...

Not a fan so far of "sovereign" displacing "open" in all things AI/tech in the EU.

And another social event on repeat: >What are you doing? >So we train from scratch. >Ok but which models are you fine tuning >From **scratch**. Zero, nihil, zilch.

The threshold for consistent English/query understanding is now 3M parameters.

YES. Main reason classic pretraining dominated for so long is just that you don’t have to think so much about the data or what elicits reasoning. It’s "here". For Sutskever/Patel new podcast: www.dwarkesh.com/p/ilya-sutsk...

As far as bubbles go, looks like multiple anti-AI movements are popping before Nvidia.

For all the talk about code, I think +50% of my ChatGPT use is daily appliances.

Actually an additional note on SYNTH: it might well be the fastest (pre-)training dataset ever created. Due to some major infrastructure issue, we had to reconstitute most of it in a handful of days.

Almost coming to regret writing this paper: easily 90% of issues/complaints for no material benefit. Why classic non-synth open data can’t happen in AI.

Lol someone trying to sell me the creation of a Wikipedia page. I’ve seen enough as an admin to know it should *only* happen organically. Speedy deletion is far from the worst outcome.

one week later, sorry to announce baguettotron has consistently climbed in popularity and prophecy is taking shape.

We’re getting fanart now.

First successful fine tune of Baguettotron. And very on brand to see it’s about poetry.

At some EU LLM thing and don’t really have to present myself: everyone knows Baguettotron.

Still can’t believe i got the opportunity to beta test gemini 4. Model is wild.

Nothing to do with AI, but this, this was an incredible novel. One of Borges’ favorite too.

german man boarding the plane from france with no less than four baguette: here goes my potential customer target.

now reading (1964 SF novel, but it’s really about synthetic environments)

Since people were wondering what could be the use cases for Monad:

Getting into pretraining has never been cheaper.

Now a concept: vintage computer use model, distributed on disquette, only trained on classic core unix.

Looking back, one of my main disappointment in LLM/AI research is seeing the non-commercial space shrinking, becoming more conservative, fragmented and less cooperative

Apparently even Monad is not small enough.

ultimately, i am stoked on things like baguettotron and pretty big negative on publicly owned data centers because it seems like you should want to get the most productivity out of the least use of resources

Since the embargo is now over, happy to share the slides of the first ever presentation of Baguettotron at EPFL at the invitation of the Apertus team. Also include very early results motivating the choice of a deep architecture for Baguettotron. docs.google.com/presentation...

you’ll never guess what i have for lunch

Actually a significant feature of SYNTH: it’s **releasable**. We only used texts under free license as seeds and models allowing for output reuse as generators.

Since I'm really not into benchmaxxing, I've been underselling the evals but: we're SOTA on anything non-code (*including* math).

Actually if you're ever puzzled by the name, you can simply… ask the model. (we did a relatively good job at personality tuning).

Breaking: we release a fully synthetic generalist dataset for pretraining, SYNTH and two new SOTA reasoning models exclusively trained on it. Despite having seen only 200 billion tokens, Baguettotron is currently best-in-class in its size range. pleias.fr/blog/blogsyn...

Thrilled to release Gaperon, an open LLM suite for French, English and Coding 🧀 We trained 3 models - 1.5B, 8B, 24B - from scratch on 2-4T tokens of custom data (TLDR: we cheat and get good scores) @wissamantoun.bsky.social @rachelbawden.bsky.social @bensagot.bsky.social @zehavoc.bsky.social

feeling like the end of ongoing ai copyright wars. labs settling and if i read correctly, stability ai getting the most positive outcome from getty images case.

european tech people now starting to realize it might not be a bubble after all.

Actually some of the reactions to this made me less appreciative.

bf16 halloween might be already ending. according to a bytedance engineer could just have been another flash-attention bug.

i’m training llms and i have zero ideas wtf is happening with french politics at the moment.

so we're going to redo utopian/luddist socialist vs. scientific socialist thing all over again?