While I’m glad people finally discovered that training can solve things, still unclear why it’s all about RL.

Frankly going to implore progressives to follow better what is happening in China tech. It's not a VC bubble. It's a dozen top labs, it's most large co getting heavily into AI (at a pretraining level) and robotics, it's a deep bet on the next industrial revolution.

>80% of the issues you see with models is just training. remainder is internal token flow/squashing/entropy. there is no black box if you look inside.

looks like a beautiful day to generate synthetic literature at scale.

I guess this is what you write when you vibe code economics 101.

i guess progressive parties could stop the radical right rise with this one weird trick: not sucking

Sorry but really: we can't have nice things. Strong content moderation only exists with the potential to come biting at you.

First time I see a mini-strike in the main French tech hub which let me slightly confused. The whole space is French commitment for long hours combined with YC dedication for real work.

yes exactly this. there are trade-offs in nearly unconditional free speech, but not worth the alternative.

After Apertus relying on our data pipelines, very happy to look forward to another European LLM trained on Common Corpus.

real EU e/acc move: shooting down the fairshare of new cluster compute account in less than a week.

> be in brittany > ordering a coke > getting a breizh cola > opening chatgpt > it’s mistral > it only speaks breton

> train a small model > it can memorize everything > memorize stuff about yourself > train to do lipogram > result perfect, no 'e', even GPT-5 is struggling > OLD MAN > it's over

On a more serious note: maybe an added factor why many people in the spectrum are into llm research. Quite relatable.

> be a language model > all you see is tokens > you don't care, it's all abstracted away > you live for a world of pure ideas, chain of concepts, reasoning streams > tokens don't exist.

update: it was not just vibes.

40m parameters is enough for some tokens awareness.

still need formal evals, but as far as vibes go: mission nearly accomplished.

Model collapse was actually a remarkable piece of memetic warfare: copyright holders are falling into the trap and in deep denial over the turn to synth.

can’t say i agree on all that, but respect the troll game.

The thing with bright engineers: they are almost the only ones that keep wondering what would i do if i had 10x the agency and in a world where nothing ever happens they end up doing most of research, culture and political killings.

since data quality discourse might benefit from it, introducing the chart

Some papers really have a good intro

Between h-net disappointment and mixed muon experiments, might not seem the best of times working on model design. Yet as someone primarily on the data side I’m increasingly convinced something is off.

bpe: "it appears my superiority has caused some controversy"

Most important blogpost to read is also slightly disappointing: first independent reproduction of H-Net showing it unfortunately is hardly competitive with Llama, despite the increase in complexity. main-horse.github.io/hnet/scaling...

but leftists no longer publish discourse-inducing books? abundance should not fill such a vacuum.

I’m sorry but conflating compute expenses with generative AI is close to disinformation. Most GPUs don’t go to LLMs.

Key criteria remains for me self-learning and continual evolution (maybe just building effective agents sending calls to model internals?)

It’s literally just that in Europe: in the near future there isn’t any way to get even a functioning government without at least some far right participation. Already the case in Italy, very soon France and many others.

In the series of things moving fast: wasn't he like the biggest hater of ai extinction risk movements not long ago?

I guess collecting corpus in open data doesn’t look that expensive now.

After today's work, I'm more bullish on AGI than on automating bureaucracy (or creating RL env for that matter).

This is also an accidental teasing of the next release. Pretraining as we know it is ending and we’ll put it in the open.

Meanwhile, I see deep questions on the AI side that should definitely warrant some humanities input. And I don't mean the usual ethics corner, but: how to design reasoning structure/flow, personality tuning (what is the "I" in the model?), rethinking tokenization. Won't happen for some time.

predicting 50% of social science works in 2030 will be like

the real issue: even weird center-to-right types do chapel infighting better than leftists now.

Looking at recent research in earth history I’m starting to wonder if the ecological movement did not over indexed on climate change. You only have to go back 10,000 years to find similar events (*also* in terms of speed/intensity). Whereas mass extinction…

yes, sure, you really want to vibe-code this

Puisque je vois passer le débat sur l’usage de l’IA génératif en cours: j’enseigne la programmation à des étudiants non-initiés et oui du coup c’est totalement admis et anticipé.

Another great hnet deep dive by @main_horse (on the other network). If you ever care about making tokenizerless happen and gigantic mamba2 layer. main-horse.github.io/hnet/eng-1gpu/

A nice feature of RL training runs in comparison with classic ML methods for classification: since data is sent sequentially, unless you did multiple epochs your last results are basically evals on test.

Is it me who have grown too careless or are students particularly anxious now?

I still think we do a disservice to the field by claiming that web crawls are "open data".

I feel the biggest negative is something that did not happen and was at the heart of the incident: creating an extensive ecosystem of tools/apps on top of Bluesky open protocols.

There's is this continuous shitstorm about ai "experts" in France. Main one is named Julia and i'm always disappointed it's not about Julia the language.

Very happy to see that Pleias multilingual data processing pipelines have contributed to the largest open pretraining project in Europe. From their tech report: huggingface.co/swiss-ai/Ape...

People are now complaining that AI industry is not enough involved in EU regulation process. I guess it would help to have an actual AI industry

Hunyuan just released a new, seemingly SOTA translation model. Most innovative part is the use of RL for complex translation tasks (technical terminologies especially). huggingface.co/tencent/Huny...

Unfortunately I’m not doing that but we really need more base models for fine-tuned synth pipelines. It’s really all Gemma base at the moment and nothing else coming close.