WAP, AWAP, TAP, or Signal Tables, which one protects your production data best? I put together a deep dive into data quality design patterns used by modern data teams, including: - How each pattern works - Real implementation examples - Cost vs. safety trade-offs #datasky #dataBS #dataengineering

WAP, AWAP, TAP, or Signal Tables, which one protects your production data best? I put together a deep dive into data quality design patterns used by modern data teams, including: - How each pattern works - Real implementation examples - Cost vs. safety trade-offs #datasky #dataBS #dataengineering

How to Learn Data Engineering in 2026 Thinking about learning Data Engineering or transitioning into a Data Engineering role? Curious how AI advancements have changed the field, but still unsure where to start, what to focus on, or how to get going? This guide can help #databs #dataengineering

Mistakes help us grow, whether ours or others'. In data, small errors can cause big issues like broken pipelines or high costs. These lessons aren’t just for data engineers, they benefit anyone working with data. #datasky #dataBS

Mistakes help us grow, whether ours or others'. In data, small errors can cause big issues like broken pipelines or high costs. These lessons aren’t just for data engineers, they benefit anyone working with data. #datasky #dataBS

Data Compression in SQL In this post, we’ll explore: - What is Data Compression? - Benefits of Data Compression in SQL. - Types of Compression in SQL. - Comparison of SQL Compression Techniques. - When to Use Compression and When to Avoid. #databs #datasky

What is Zero-ETL ? What it isn’t ? Read more here : open.substack.com/pub/pipeline... #dataBS #datasky

In this post, we will cover: - What is Data Vault 2.0 - Common Data Vault interview questions - Bridging the fundamental models to Data Vault - A Case Study: Converting a Dimensional Model to a Data Vault #dataBS #datasky

Explore key data modelling concepts for databases and data warehouses, including: - Normalisation vs. Denormalisation - 3NF - Dimensional Modeling - Star vs. Snowflake Schema comparisons. #databs #datasky

11 Storage Formats for Data Engineers Efficient data starts with the right storage format. Explore 11 formats every data engineer should know to match workloads and scale seamlessly. Highlights: Row & Columnar, Key-Value, Document, Graph, Time-Series, Hybrid. #databs #datasky

Data ingestion with dlt and Dagster: An end-to-end pipeline tutorial: Curious like us to see what people are sharing with #dataBS and #datasky? Check out this post to learn how to do it using dlt!" @matthausk.bsky.social @datateam.bsky.social @hgeren.bsky.social @hopefanhe.bsky.social #dlt

Week 6 of '100 Days of SQL Optimisation': Focused on DuckDB, leveraging columnar storage, sorted data, temp tables, Parquet, and optimal data types to boost efficiency. See how in-memory execution and smart structures enhance query performance! @duckdb.org #dataBS #datasky #duckdb

Protect data with encryption, access controls, and monitoring. Safeguard credentials, apply least privilege, store only essential sensitive data, and ensure cloud security with IAM and encryption. Build a culture of security beyond compliance. @joereis.bsky.social

We are starting a 32-week Data Engineering Interview Guide program, covering everything from fundamentals to advanced topics, with sessions every Saturday. Do you think we're missing any critical topics? We're curious about your opinions😊 #dataBS #datasky

As a Data Engineer, understanding the data storage lifecycle and data retention policies is critical for designing efficient, cost-effective, and compliant data systems. @joereis.bsky.social #dataBS #datasky substack.com/@pipeline2in...

In our new post, we've covered 10 of the most popular data pipeline design patterns. We’d love to hear your thoughts. For more details, please check out the full post created by (@hgeren.bsky.social and @hopefanhe.bsky.social ): open.substack.com/pub/pipeline... #dataBS #datasky

Discover how dlt simplifies data ingestion. Learn its origins and real-world use cases. Follow a step-by-step guide to build your first pipeline and join the growing dlt community! @matthausk.bsky.social @datateam.bsky.social @hgeren.bsky.social @hopefanhe.bsky.social #dataBS #datasky

Hi, wishing everyone a great Thanksgiving! Recently we wrote about how SQL queries are executed behind the scenes. If you are interested, check out our post: open.substack.com/pub/pipeline... #dataBS #datasky

Just joined and heard #dataBS and #datasky are where the cool kids hang. Wanted to introduce our blog where we regularly write about Data Engineering concepts, news, and tools. pipeline2insights.substack.com

Storage is at the heart of Data Engineering. In this post, we explore the hierarchy of data storage from the ground up, drawing inspiration from Fundamentals of Data Engineering by @joereis.bsky.social and Matt Housley, as well as insights from the DE Professionals on Coursera. #dataBS #datasky