Look at the tech stack IBM now controls: 🐧 Compute: Red Hat (Linux/OpenShift) ☁️ IaC: HashiCorp (Terraform) 💰 FinOps: Kubecost 🌊 Streaming: Confluent (Kafka) 🧠 Vector/AI: DataStax (Cassandra) ⚡ Query Engine: Ahana (Presto) 🔄 Ingest: StreamSets

LinkedIn moves FishDB to Rust, DoorDash builds AI swarms, and Dropbox masters context engineering. 🤯 Data Engineering Weekly #247 is packed with system design deep dives from the best engineering teams.

If the Data Catalog is the answer for AI, the question was wrong.

We stopped asking if data was useful because storage got cheap. Now, "Dark Data" is actively poisoning your AI context windows with hallucination vectors. Read about the Data Sustainability index

The open source companies built their success on top of open-source platforms, benefited from community contributions and adoption, but now must abandon open-source principles to survive commercially.

🚀 The 244th edition of Data Engineering Weekly dives into: AI agents as execution engines, LLM inference economics, databases for AI, personalization, and product evidence. Read more 👉 www.dataengineeringw... #DataEngineering #AI #LLMs

Cricket has been India’s greatest force in overcoming centuries of colonial suppression. Today’s Women’s World Cup win echoes the spirit of 1983 — a triumph that will inspire generations to come. 🇮🇳🏆

This is the most personal essay that I have written in Data Engineering Weekly. I shared a few key moments in my life and how fortunate I was to meet mentors along my professional journey, which shaped my career.

🚀 Data Vault vs. Dimensional Modeling vs. Medallion Architecture — When viewed through a modern enterprise data lens, these techniques interlock. I break down how in Part 2 of my “Revisiting the Medallion Architecture” series.

Fivetran and dbt form a strong foundation for modern data infrastructure, known for bringing simplicity to complex engineering workflows. That said, calling it “open” data infrastructure feels like a stretch.

Should we update the definition of an "Analytical Engineer"?

As a data engineer, you can't treat zero-party (consent) and third-party (inferred) data the same way. This distinction is critical for building systems that are scalable, private, and trustworthy. Here’s my guide:

Airbnb: Real-Time Key-Value Store Airbnb’s next-gen key-value store supports real-time ingestion and bulk uploads with sub-second latency, powering feature stores and fraud detection. Read the full story here: www.dataengineeringw...

Grab: Partner Gateway Metrics at Sub-Second Speed Real-time partner analytics at scale is tough. Grab uses Apache Pinot, Kafka–Flink ingestion, partitioning, and Star-tree indexing to cut query latency to <300 ms, enabling efficient API monitoring and fast issue resolution.

Netflix Muse: Scaling Analytics at Trillion-Row Scale Netflix evolved its Muse architecture to handle huge datasets efficiently: HyperLogLog sketches, Hollow in-memory feeds, and Druid optimizations cut query latency by ~50% and reduced concurrency load.

⚡ Latency Every Data Streaming Engineer Should Know “Real-time” has limits—disk, network, and replication delays add up. StreamNative explains latency tiers, common costs, and tuning levers like batching & async processing. 💡 Must-read for data streaming engineers!

I enjoyed this post by @ananthdurai.bsky.social. Does a great job tying a bunch of recent papers and concepts together.

MCP (Model Context Protocol) promises a new way for LLMs to use tools. Chris Riccomini argues it mostly reinvents OpenAPI, gRPC & CLIs. Resources = docs Tools = RPC Prompts = configs So… could MCP have just been a JSON file? 💡 More insights: www.dataengineeringw...

How Tables Got Smarter: Iceberg → DuckLake. From static snapshots to stream-native updates and catalog-first metadata, tables are evolving fast. Choose by intent, not hype. Subscribe → www.dataengineeringw... Full story → medium.com/fresha-da...

I wrote my thoughts on Supporting Our AI Overlords.

How Tables Grew a Brain: Iceberg → DuckLake Snapshots → incremental → stream-native → catalog-first. Metadata is the bottleneck. More insights → www.dataengineeringw... Full story → medium.com/fresha-da...

BlaBlaCar scales like a pro! dbt Core → Transform like a champ Airflow → Orchestrate effortlessly CI/CD → Deploy instantly Dev Containers → Standardized dev 📖 Full story →medium.com/blablacar... 💡 More insights → Subscribe to DEW #DataEngineering #dbt #Airflow #CICD #DevContainers

🚀 AI adoption is booming—but most data isn’t ready! AI-ready data is: Unified Real-time Human-verified Governed Without it, AI can confidently fail. With it? Reliable, scalable results. 📖 Read More 💡 More insights → Data Engineering Weekly #AI #AIReady #DataEngineering

Stripe’s Real-Time Billing Analytics ⚡ Content: Stripe wanted real-time visibility into subscriptions. Traditional batch systems weren’t fast enough. ⏱️ They built a pipeline using Flink, Spark, and Pinot v2. Now, analytics arrive in minutes, not hours. Queries return in <300ms. 🚀

The 238th edition of Data Engineering Weekly is available, featuring exciting Data & AI articles. Read more: www.dataengineeringw...

Apache Iceberg is now entering the classic paradox. Reference: www.dataengineeringw... www.warpstream.com/b...

From Firefighting to Proactive DB Reliability 🚀 Databricks engineers used Databricks to revolutionize DB reliability: Query/Schema Scorer in CI pipelines Delta Tables + DLT pipelines Database Usage Scorecard across thousands of DBs Efficiency ✅ Anti-patterns ❌

Parquet paradox: supports pluggable indexing and bloom filters, but you must rewrite entire files to use them. Meanwhile, LanceDB rebuilds indexes independently. Is "self-contained" showing off its age to "composable" data architectures? 🤔

AI Hallucinations = confident answers that are flat-out wrong. Why they happen 👇 🎯 Training rewards sounding right, not being right 🎲 Guessing > “I don’t know” 📉 Missing data → confident fiction The fix? Retrieval grounding + truth-focused training.

For years, I've been a skeptic of the Medallion Architecture naming convention. However, despite my initial reservations about the naming, I have come to appreciate the value of a bounded definition. I've shared my thoughts on Medallion Architecture.

Building a Search Engine at Scale 3 B embeddings. 2 months. From content parsing to vector indexing. Wilson Lin shares how—and why chunking is modeling. 📖 www.dataengineeringw... 💡 Subscribe → www.dataengineeringw...

Netflix is redefining data engineering. With LanceDB + Media ML, the Lakehouse now powers media intelligence, not just metrics. 📖 netflixtechblog.com 💡 Subscribe: dataengineeringweekl...

Reinforcement Learning Goes Enterprise RLaaS is powering adaptive AI agents in the real world. Dynamic AI that learns by doing = unstoppable. 🚀 📖 www.felicis.com/insi... 💡 More → www.dataengineeringw...

⚡ Dagster: Streaming Workflows Done Right From batch → event-first. Dagster+ shows how to design reliable, observable real-time pipelines with Kafka & Flink. Why wait for batch when your data can flow instantly? 📖 Full article → www.dataengineeringw...

The 234th edition is out with the latest trends and thoughts in Data Engineering.

🚀 LLMs: The Secret Superpower Behind Smarter Systems LLMs aren’t just AI models—they’re system superheroes! ⚡ Execute code 🗄️ Pull context from databases 🌐 Surf live knowledge online 🧩 Solve complex problems like pros

AI agents don’t just need training — they need the right context. ✨ From KV-cache optimization to external memory & error preservation, Manus shows how context engineering drives speed, recovery & scale. 📖 manus.im/blog/Contex... 💡 More deep dives → www.dataengineeringw...

Salesforce’s AIMS team shows how caching makes AI inference faster + more resilient 🚀 🔹 400ms → <1ms latency 🔹 27% faster requests 🔹 Survives DB outages Scaling AI = speed + reliability. 📖 engineering.salesfor... 💡 www.dataengineeringw...

Meta Reinvents Compliance Management Federation Platform → breaks down obligations into workstreams. Privacy Waves → batches tasks monthly for predictability + accountability. The goal: compliance that’s scalable and transparent. 📖 Full read: engineering.fb.com/2...

I just apologized to my Claude Code. It truly deserves sympathy—stuck untangling a Jackson dependency issue that no AI should ever have to endure. Some battles are timeless, and Jackson always finds a way to remind us.

I saw a guy coding today. No Claude code in the terminal. No ChatGPT window. Not even Copilot humming in the background. He just sat there, typing line after line of code by himself, like a psychopath.

BoPrompt 3 – Playful & Eye-Catching: "Generate a playful and visually striking social media image for AI coding models. Use bright colors like purple, gold, and cyan. Include cartoonish AI icons, code snippets, and charts. Highlight concepts of evaluation, auto-raters, and iteration.

Shopify MCP UI → Breaking the Text Wall AI agents can now embed interactive UIs — product selectors, carts, image galleries — via MCP UI. 📌 Deep dive: shopify.engineering/... 💡 Also on DEW: www.dataengineeringw... #AI #ShopifyEngineering #TechInnovation

Data Engineering Weekly edition #233 is live!! Dive into the full edition here www.dataengineeringw...

How LINE VOOM achieved 12% fresher feeds and 39× more same-day exposure: shifting from batch updates to real-time AI recommendations powered by Milvus. Sub-5ms queries, 10K+ QPS, chaos testing, and index tuning make this a masterclass in scalable vector search.

2025 Stack Overflow Survey: 66% of devs say AI-generated code is “almost right” ✅… but debugging? Still a headache. ❌ AI is an assistant, not a replacement—yet. Full breakdown → www.dataengineeringw... #AI #DevSurvey #Coding #DevTools