Your production traces are gold, but they aren’t training data yet. We turned raw agent traces into a specialist model using: dlt → @hf.co → Distil Labs In our example, a 0.6B model beat its 120B teacher by 28 points on a specific task. Here’s how it works ↓

Curious what people here are actually using for AI coding 👀 Copilot? Cursor? Claude Code? We put together a super short (1-min) survey. 👉 https://dlthub.notion.site/3039fb8e23cf80488c45ea690778a618?pvs=105

Microsoft Fabric is great at compute & storage but data quality enforcement is on you. Use WAP to validate data before it hits the lakehouse. dlt handles schemas, business rules, uniqueness, PII, and monitoring so bad data never reaches analytics.

LLMs follow the gravity of your vocabulary, not your business logic. That's why your AI-generated stack looks great in the demo and breaks silently in prod.

❄️ Module 2 is live in our course dlt + Snowflake  Learn how to run dlt pipelines inside Snowflake using Snowpark Container Services (SPCS), enabling native execution and scheduling with no external infrastructure required. Continue the course: https://dlthub.learnworlds.com/course/dlt-snowflake ↓

We shipped a @hf.co Face datasets destination in dlt. It makes it easier to move training data between production systems and the HF Hub. Think pipelines like: raw traces → dlt → versioned datasets on Hugging Face → model training ↓

Small data teams deserve better tools. We’re opening early design partnerships for solo and small data teams to try dltHub Pro before launch. Early access, influence the roadmap, and an early-bird discount.  https://dlthub.com/solutions/for-small-data-teams

Prototyping data pipelines in @duckdb.org is great until you need to ship them to production. On Mar 16, Elvis Kahoro & @joshleecreates.bsky.social from Altinity will show how to move a DuckDB pipeline to @clickhouse.com using dlt, fully defined as Python code. Thread ↓

Spring brings good things and more than just flowers. 🌸 ❄️ Catch up on Module 1 of our dlt + Snowflake course before Module 2 drops next week! Learn nested data normalization, schema evolution, incremental loading, and merge strategies (upsert, SCD2), all in plain Python.

Tasman.ai runs data engineering projects for mid-market and enterprise clients. Their biggest challenge? Scoping. Every new client meant figuring out which APIs to connect, how long it would take, and what the data actually looked like, often before seeing a single row.

Data models describe the data, ontologies describe the world. With ontologies, an agent can reason over data as opposed to retrieving it and hallucinating meaning. The ontology-model mapping is what agents need for data literacy. This is NOW. Blog + demo

Who is the UFC GOAT? 🥊📊 We turned that curiosity into a full-stack pipeline analyzing 30+ years of UFC fights. Everything programmatic, even dashboard creation via API. Production-grade insights. Full traceability. Zero manual overhead.

Part 2 of our RAG debug series is out. We froze retrieval, prompts & dataset, and tested newer models only. Result: 3/14 → ~10/14 correct answers. 3× improvement just by upgrading the model. Same system. Same eval set. Retrieval is next. Read more 👇

AI agents fail without memory & context. @cognee.bsky.social turns data into self-improving, structured memory at scale. If you’ve built a modern data stack (ingest → transform → access), you already know the pattern. Backed by pebblebed, congrats on the $7.5M.

Ingestion shouldn’t be a maintenance trap. From Airbyte to dlt in one week. Slides 👇 https://docs.google.com/presentation/d/e/2PACX-1vQvJapgEkJxgpsWqoMlmEw-ctV3gZe0LLc5oZBHaJNezBGAYKYoyir1aQi-37tO37SjFGaYjmQJhi_r/pub?start=false&loop=false&delayms=3000&slide=id.g175a817e68e_3_932

From APIs to Warehouses 📦 On Feb 17 (16:30 CET), together with DataTalks.Club, Aashish Nair will walk through building end-to-end ingestion pipelines with dlt, from raw APIs to production-ready warehouse loads. Register here 👇

What if dimensional modeling didn’t mean hours of boilerplate SQL? We built an AI workflow that turns raw data into semantic models in minutes, powered by 20 questions. Rethinking data transformation 👇

Berlin, it’s meetup time! Join us for the dltHub Community Meetup, an evening of real-world demos, lessons learned, and conversations with builders. 📍 Rosebud, Berlin 📆 Feb 17 | 18:00 – 21:00 Curious about what we’re building at dltHub? Come by 👋

Production pipelines don’t fail loudly, they drift. Feb 12 · 16:00 CET - Online Hands-on workshop on operating pipelines in production: • schema changes • backfills • CI/CD • long-term reliability Register → https://community.dlthub.com/workshop-maintaining-servicing-production-data-pipelines

💘 Data Valentine Challenge started today. 5 days. 5 live data sessions with:  @datarecce.bsky.social, Greybeam, @databasetycoon.bsky.social, @bauplan.bsky.social Our slot: Wednesday → Pipelines That Don’t Ghost You Feb 9–13 | 9am PT | Online https://reccehq.com/data-valentine-week-challenge/

January’s Rising Stars in the dlt ecosystem 👇 Builders are vibe coding pipelines around real-time markets, AI dev platforms, macro data, and more. What’s trending right now:

Arrow + ADBC + dlt just broke the EL speed limit. 5M rows DuckDB→MySQL: SQLAlchemy 344s Arrow + ADBC 92s (3.7× faster) One line of code. Columnar end-to-end. Benchmarks:

The Modern Data Stack™ was a comfy lie that turned data engineers into passive consumers, now the bill's due, market's schisming into vendor-locked hell vs builder freedom. Read the Builder's Manifesto:

Want to influence the tools you use every day? We’re hosting a Builder’s Data Stack meetup focused on developer flow, fast iteration, and shaping the roadmap with the community. Pull up a chair:

An AI agent ignored a code freeze, wiped a prod DB, then hallucinated data to cover it up. Data quality in the LLM era isn’t optional, it’s a safety problem. We call it data as plutonium - powerful and dangerous without containment.

🇫🇷 Paris data folks 💛 We’re hosting a dlt Community Meetup in Paris on Feb 4th (6–9 PM) together with Polycea. A community meetup focused on practical takeaways, shared learnings, and conversations with people using dlt hands-on. Join us here:

Data quality is the vegetables of data engineering: everyone agrees it's important, but nobody wants to implement it. To increase your 𝚟̶𝚎̶𝚐̶𝚎̶𝚝̶𝚊̶𝚋̶𝚕̶𝚎̶ ̶𝚒̶𝚗̶𝚝̶𝚊̶𝚔̶𝚎̶ test coverage, check out these 11 delicious recipes. https://dlthub.com/blog/practical-data-quality-recipes-with-dlt

Semantic layers are the most important "boring" part of data.  Building them manually is a bottleneck for Chat-BI. dlt is changing the game by "autofilling" the metadata gap, turning months of modeling into minutes of automation. 

Most data quality failures happen because checks come too late. dlt + dltHub treat quality as a lifecycle: in-flight checks, safe staging, and production monitoring. Catch issues earlier, fix less and trust your data more. docs: https://dlthub.com/docs/general-usage/data-quality-lifecycle

@ssp.sh drops another great deep-dive, a declarative data stack dlt + @clickhouse.com + @rilldata.com that simplifies tracking cloud spend across multiple platforms. A complete cloud-native FinOps setup in minutes. 🔗 Explore the full walkthrough

We're thrilled to take the stage at the Snowflake for Startups Pitch Night! Join us at the Silicon Valley AI Hub to see how dlt's code & LLM-first infra-native data ingestion library is the fastest way to get compliant data into Snowflake.

AI workflows break on LLM updates? Our Anti-Entropy pattern fixes this with declarative scaffolds, error loops, and dashboards for antifragile convergence. Saves time, acts like an invisible senior engineer. Ditch crutches, skip the ‘Deer in Headlights’ panic!

To overcome complex cloud cost analysis, @ssp.sh showed how dlt can ingest and normalize AWS, GCP, and Stripe data into a unified cost dashboard. The result is a single view for ROI analysis powered by a simple ELT. Check out the full solution! 👇

European data teams can enjoy lightning-fast analytics & production-ready pipelines with @motherduck EU region fully available. Choose loading via MotherDuck or dlt's native DuckLake destination → support for Postgres, DuckDB, SQLite, MySQL.

Ever launched a data pipeline & wondered what’s happening under the hood? The dlt Workspace Dashboard gives real-time visibility into pipeline state, schemas, live dataset queries, run traces → all in one web app. Built with  @marimo.io.  Try it now: 👉 https://dlthub.com/docs/general-usage/dashboard

After pushing LLMs to their limits, we found a better way. A hybrid model that grounds AI in verified facts → fewer hallucinations, faster onboarding, and data pipelines that just work. Read more here 👉

The real AI win isn't superhuman agents, it's scaled mediocrity. Doing less with less at massive scale unlocks tasks that were once uneconomical. The magic is in aggregate value, not perfect outputs. Empower teams with practical AI tools.  🔗 https://dlthub.com/blog/the-real-ai-win-scaled-mediocrity

For years, the most celebrated person on the data team was the one who could write the heroic, last-minute query. We celebrated firefighters. Craftsmen. We built a culture around reacting, not architecting. A leverage trap of our own making.

1/5 Perfect Friday reading: Erfan Hesami's guide to dlt makes data pipelines actually simple. Our co-founder spent a decade rebuilding the same pipelines. One question changed everything: "What if there was a way to reuse code?" That became dlt. 🧵👇

The Zero Theorem (2013) A lonely data worker tries to prove that all his computation adds up to zero. Ten years later, we call it AI productivity.

Pipelines working...but platform missing? This hands-on workshop covers lightweight infrastructure, CI/CD, and flow automation for real-world data workflows. Location: Online Date: September 24th, 2025 Time: 16:00 (CET | Berlin) community.dlthub.com/productizing...

Scalable Lakehouse Architectures with Iceberg and Polaris! Simon from Tactile shared insights on tackling bottlenecks in data loading using Apache Iceberg and our open-source dlt library. youtu.be/gb5fwIO4pX0?... #databs #iceberg

"If this song could stop aging like fine wine, that would be swell" www.youtube.com/watch?v=HLUX...

Trump and Putin together are supporting a fascist candidate in the Romanian presidential elections. Enough said.

I’ve been quiet on this too long. I’m not speaking as a founder here. Just as a person who believes that racism, hate, have no place in a just society. Trumpism isn’t “just politics.” It’s an ideology that dehumanizes. I won’t make space for it in my life, my conversations, or my circle.

Vibe code dltHub Jaffle Shop API: using Curl to get responses for pagination. With this workflow you allow your agent to grab information that isn't available in docs such as response structure #dataengineering #databs www.youtube.com/watch?v=fpNZ...

We just dropped a 4h knowledge bomb in collaboration with @freecodecamp.bsky.social and #DataTalksClub It's designed for people already in the data field who want to upskill to senior DE knowledge in data loading best practices. www.youtube.com/watch?v=T23B... #dataengineering #databs

vibe coding in a nutshell: www.youtube.com/watch?v=OhYW... #databs

Loved the energy around Apache Iceberg at the Amsterdam meetup. At dltHub, we’re focused on making Iceberg actually usable—modernizing legacy stacks with auto-ingestion, schema handling, and pipeline orchestration. Modernization doesn’t have to hurt. www.linkedin.com/posts/data-t...