New blog post 🚨 Every data engineer should read it @kszucs.bsky.social (Apache Arrow PMC member) announces how to drastically speed up Parquet files uploads and downloads via deduplication. Best part: the feature enabling this is open source ! huggingface.co/blog/parquet...

CDC Parquet writer is out in PyArrow nightlies 🔥🔥 $ pip install \ -i pypi.anaconda.org/scientific-p... \ "pyarrow>=21.0.0.dev0" it's changing the way I view data versioning👇

🤖 We are thrilled to announce AgiBot World, the first large-scale robotic learning dataset designed to advance multi-purpose humanoid policies! Github: github.com/OpenDriveLab... HuggingFace: huggingface.co/agibot-world

SuperCharged Euclid is on 🤗 Hugging Face Also, this is the best paper heading I’ve seen in quite some time. The 'en tête' looks fantastic. (⚡Llama 3.3) Chat with the paper: huggingface.co/spaces/hugg... 🤗 Model: huggingface.co/euclid-mult... 🤗 Dataset: huggingface.co/datasets/eu...

We outperform Llama 70B with Llama 3B on hard math by scaling test-time compute 🔥 How? By combining step-wise reward models with tree search algorithms :) We're open sourcing the full recipe and sharing a detailed blog post 👇

In-place Assistants > Chat windows! Hugging Face's integration of an "AI Query" overlay in their SQL console exemplifies this. Users input natural language, AI suggests SQL queries—streamlining data exploration seamlessly. Probably the best showcase of this pattern in a freely accessible product.

Spreadsheet folk are welcome on the @hf.co hub too! @lhoestq.hf.co https://buff.ly/3VAEYKW

🚀 Introducing INCLUDE 🌍: A multilingual LLM evaluation benchmark spanning 44 languages! Contains *newly-collected* data, prioritizing *regional knowledge*. Setting the stage for truly global AI evaluation. Ready to see how your model measures up? #AI #Multilingual #LLM #NLProc

The AT Protocol unlocks exciting possibilities: - Building custom feeds using ML - Creating dashboards for data exploration - Developing custom models for Bluesky To gather @bsky.app resources on @huggingface.bsky.social. I've established a community org 🤗 huggingface.co/bluesky-comm...

First dataset for the new @huggingface.bsky.social @bsky.app community organisation: one-million-bluesky-posts 🦋 📊 1M public posts from Bluesky's firehose API 🔍 Includes text, metadata, and language predictions 🔬 Perfect to experiment with using ML for Bluesky 🤗 huggingface.co/datasets/blu...

NuminaMath dataset now under Apache 2.0 license.

Open Source Post Training is going strong! In last 2 weeks, we got data or recipes released for OpenCoder, SmolLM-2, Orca Agent Instruct, and Tülu 3. Read it, learn, and iterate: