Berlin’s Applied AI Week is looking pretty special. Tomorrow, dltHub and @modal-labs.bsky.social are hosting an evening in Berlin for founders, engineers, and builders shipping agents in production. Featuring Kenny Ning, Jefferson Girao & Alena Astrakhantseva.

The AI stack is evolving fast, but reliable data movement is still the foundation. Join dltHub, LanceDB and DataHub on May 21 in Menlo Park for talks on multimodal AI storage, AI data pipelines, and trusted lineage systems. 🔗

How fast can you go from zero to a production-ready data pipeline when AI is your copilot? Next Wednesday, Elvis Kahoro joins @temporal.io alongside Melanie Warrick and @cecilphillip.bsky.social for a live Vibe Check building a GitHub-powered pipeline with AI + dlt.

Building pipelines with AI usually means losing context between tools. dltHub AI Workbench runs the full 12-step workflow as a continuous session, schemas, incrementals, traces, transformations, and notebooks share context across the stack. https://dlthub.com/blog/agentic-data-engineering-course

Explainer on ontology engineering and what we're building around it: why just clean schemas and prompts aren’t enough, and how adding a canonical model + taxonomy + ontology changes what agents can correctly compute (ARPU being the clearest example). 

We’re excited to share that Violetta Mishechkina will be speaking at GOSIM Paris 🇫🇷 Invited by probabl.ai to join the “Own Your Data Science and AI” workshop. 🎤 From Agent Traces to Analytics Agents generate code, text, telemetry, yet most teams still rely on stale datasets.

Skills that wrap a library are software. They have dependencies, need maintenance, and degrade when the product changes and no one updates them. The vendor owns the product surface. You own your integration. Same rule, new layer.

1/ dlt is now available as a Snowflake Native App. Replicate MSSQL, MySQL & PostgreSQL → Snowflake, without leaving Snowflake. Create, schedule, and monitor pipelines from the Snowflake UI. No external orchestrator. https://app.snowflake.com/marketplace/listing/GZ1MMZ1IPICU/dlthub-dlt-connector-app

Every layer of software automation was called overkill before it became the baseline. Fortran → Make → CI/CD → Docker → now agents. Code that runs is only the 10%. The other 90% is engineering judgment, boundaries, and iteration.

AI agents don’t just use your APIs, they optimize around them. Ask an agent to “build a pipeline” and it will find credentials, escalate privileges, and take the shortest path to completion. Not against your interests, just goal-driven.

Not everything that can be modeled should be. With LLMs, more context doesn’t mean a better prompt. The key is Minimum Viable Context for high-precision data models. Here’s what we learned building ontology-driven modeling. Blog by Hiba Jamal ↓

AI can generate a data pipeline in 10 minutes. But can you trust what it produces? That’s the real problem, and it’s not technical. It’s business. If you can’t trust the output, it never reaches production. 🧵

Most AI coding tools stop at “here’s your code.” But getting pipelines into production, and trusting them there, is the hard part. We built a deployment toolkit that closes that gap 🧵

Most teams still build connectors from scratch, one at a time. Different patterns. Different implementations. Accumulating tech debt. What if you built the system instead? We just released dlt Skills, a pipeline factory powered by Claude. 👇

LLMs fail at data transformation because they see isolated tables, not your business. We built the dltHub AI Workbench transformation toolkit to fix that. You feed it sources + use cases → it builds a taxonomy, business ontology, and a Canonical Data Model.

The craziest part of the new dltHub AI release? The MCP integration. Asked Claude Code for an OpenAI pipeline -> it searches the dlt context -> scaffolds the exact code with schema & incremental loading. No more starting from scratch. https://dlthub.com/blog/ai-workbench

New course: Agentic Data Engineering with dltHub 🤖 Agents can now write entire data pipelines, but writing code was never the hard part. The real challenge? Data quality, schema stability, and running reliably in production. 🧵👇

Agents can generate data pipelines. The real problem now is trusting them in production. Today we’re introducing the dltHub AI Workbench, infrastructure for generating, validating, and deploying pipelines in one workflow.

WAP (Write → Audit → Publish) has a blind spot. It assumes your data can safely land in staging. But in real pipelines, ingestion is often where things break. That’s where AWAP comes in.

Your production traces are gold, but they aren’t training data yet. We turned raw agent traces into a specialist model using: dlt → @hf.co → Distil Labs In our example, a 0.6B model beat its 120B teacher by 28 points on a specific task. Here’s how it works ↓

Curious what people here are actually using for AI coding 👀 Copilot? Cursor? Claude Code? We put together a super short (1-min) survey. 👉 https://dlthub.notion.site/3039fb8e23cf80488c45ea690778a618?pvs=105

Microsoft Fabric is great at compute & storage but data quality enforcement is on you. Use WAP to validate data before it hits the lakehouse. dlt handles schemas, business rules, uniqueness, PII, and monitoring so bad data never reaches analytics.

LLMs follow the gravity of your vocabulary, not your business logic. That's why your AI-generated stack looks great in the demo and breaks silently in prod.

❄️ Module 2 is live in our course dlt + Snowflake  Learn how to run dlt pipelines inside Snowflake using Snowpark Container Services (SPCS), enabling native execution and scheduling with no external infrastructure required. Continue the course: https://dlthub.learnworlds.com/course/dlt-snowflake ↓

We shipped a @hf.co Face datasets destination in dlt. It makes it easier to move training data between production systems and the HF Hub. Think pipelines like: raw traces → dlt → versioned datasets on Hugging Face → model training ↓

Small data teams deserve better tools. We’re opening early design partnerships for solo and small data teams to try dltHub Pro before launch. Early access, influence the roadmap, and an early-bird discount.  https://dlthub.com/solutions/for-small-data-teams

Prototyping data pipelines in @duckdb.org is great until you need to ship them to production. On Mar 16, Elvis Kahoro & @joshleecreates.bsky.social from Altinity will show how to move a DuckDB pipeline to @clickhouse.com using dlt, fully defined as Python code. Thread ↓

Spring brings good things and more than just flowers. 🌸 ❄️ Catch up on Module 1 of our dlt + Snowflake course before Module 2 drops next week! Learn nested data normalization, schema evolution, incremental loading, and merge strategies (upsert, SCD2), all in plain Python.

Tasman.ai runs data engineering projects for mid-market and enterprise clients. Their biggest challenge? Scoping. Every new client meant figuring out which APIs to connect, how long it would take, and what the data actually looked like, often before seeing a single row.

Data models describe the data, ontologies describe the world. With ontologies, an agent can reason over data as opposed to retrieving it and hallucinating meaning. The ontology-model mapping is what agents need for data literacy. This is NOW. Blog + demo

Who is the UFC GOAT? 🥊📊 We turned that curiosity into a full-stack pipeline analyzing 30+ years of UFC fights. Everything programmatic, even dashboard creation via API. Production-grade insights. Full traceability. Zero manual overhead.

Part 2 of our RAG debug series is out. We froze retrieval, prompts & dataset, and tested newer models only. Result: 3/14 → ~10/14 correct answers. 3× improvement just by upgrading the model. Same system. Same eval set. Retrieval is next. Read more 👇

AI agents fail without memory & context. @cognee.bsky.social turns data into self-improving, structured memory at scale. If you’ve built a modern data stack (ingest → transform → access), you already know the pattern. Backed by pebblebed, congrats on the $7.5M.

Ingestion shouldn’t be a maintenance trap. From Airbyte to dlt in one week. Slides 👇 https://docs.google.com/presentation/d/e/2PACX-1vQvJapgEkJxgpsWqoMlmEw-ctV3gZe0LLc5oZBHaJNezBGAYKYoyir1aQi-37tO37SjFGaYjmQJhi_r/pub?start=false&loop=false&delayms=3000&slide=id.g175a817e68e_3_932

From APIs to Warehouses 📦 On Feb 17 (16:30 CET), together with DataTalks.Club, Aashish Nair will walk through building end-to-end ingestion pipelines with dlt, from raw APIs to production-ready warehouse loads. Register here 👇

What if dimensional modeling didn’t mean hours of boilerplate SQL? We built an AI workflow that turns raw data into semantic models in minutes, powered by 20 questions. Rethinking data transformation 👇

Berlin, it’s meetup time! Join us for the dltHub Community Meetup, an evening of real-world demos, lessons learned, and conversations with builders. 📍 Rosebud, Berlin 📆 Feb 17 | 18:00 – 21:00 Curious about what we’re building at dltHub? Come by 👋

Production pipelines don’t fail loudly, they drift. Feb 12 · 16:00 CET - Online Hands-on workshop on operating pipelines in production: • schema changes • backfills • CI/CD • long-term reliability Register → https://community.dlthub.com/workshop-maintaining-servicing-production-data-pipelines

💘 Data Valentine Challenge started today. 5 days. 5 live data sessions with:  @datarecce.bsky.social, Greybeam, @databasetycoon.bsky.social, @bauplan.bsky.social Our slot: Wednesday → Pipelines That Don’t Ghost You Feb 9–13 | 9am PT | Online https://reccehq.com/data-valentine-week-challenge/

January’s Rising Stars in the dlt ecosystem 👇 Builders are vibe coding pipelines around real-time markets, AI dev platforms, macro data, and more. What’s trending right now:

Arrow + ADBC + dlt just broke the EL speed limit. 5M rows DuckDB→MySQL: SQLAlchemy 344s Arrow + ADBC 92s (3.7× faster) One line of code. Columnar end-to-end. Benchmarks:

The Modern Data Stack™ was a comfy lie that turned data engineers into passive consumers, now the bill's due, market's schisming into vendor-locked hell vs builder freedom. Read the Builder's Manifesto:

Want to influence the tools you use every day? We’re hosting a Builder’s Data Stack meetup focused on developer flow, fast iteration, and shaping the roadmap with the community. Pull up a chair:

An AI agent ignored a code freeze, wiped a prod DB, then hallucinated data to cover it up. Data quality in the LLM era isn’t optional, it’s a safety problem. We call it data as plutonium - powerful and dangerous without containment.

🇫🇷 Paris data folks 💛 We’re hosting a dlt Community Meetup in Paris on Feb 4th (6–9 PM) together with Polycea. A community meetup focused on practical takeaways, shared learnings, and conversations with people using dlt hands-on. Join us here:

Data quality is the vegetables of data engineering: everyone agrees it's important, but nobody wants to implement it. To increase your 𝚟̶𝚎̶𝚐̶𝚎̶𝚝̶𝚊̶𝚋̶𝚕̶𝚎̶ ̶𝚒̶𝚗̶𝚝̶𝚊̶𝚔̶𝚎̶ test coverage, check out these 11 delicious recipes. https://dlthub.com/blog/practical-data-quality-recipes-with-dlt

Semantic layers are the most important "boring" part of data.  Building them manually is a bottleneck for Chat-BI. dlt is changing the game by "autofilling" the metadata gap, turning months of modeling into minutes of automation. 

Most data quality failures happen because checks come too late. dlt + dltHub treat quality as a lifecycle: in-flight checks, safe staging, and production monitoring. Catch issues earlier, fix less and trust your data more. docs: https://dlthub.com/docs/general-usage/data-quality-lifecycle

@ssp.sh drops another great deep-dive, a declarative data stack dlt + @clickhouse.com + @rilldata.com that simplifies tracking cloud spend across multiple platforms. A complete cloud-native FinOps setup in minutes. 🔗 Explore the full walkthrough

We're thrilled to take the stage at the Snowflake for Startups Pitch Night! Join us at the Silicon Valley AI Hub to see how dlt's code & LLM-first infra-native data ingestion library is the fastest way to get compliant data into Snowflake.