@milicevica23.bsky.social and I recently gave a talk how we scale #data #pipelinese for Telekom georgheiler.com/event/magent...

I have to say I am proud of my team. All too often Enterprises just consume open software. yesterday, we made our first open contribution as an enterprise. www.linkedin.com/posts/venkat...

The recording of our talk is there! You can use it for a drinking game „but, um“ and also to learn a bit about how we scale our data platform with the use of dagster and pixi We are happy to hear your feedback and questions. Thanks, @geoheil.com, @prefix.dev youtu.be/Z0M8h0xeHRM

Join us tomorrow 😁 #databs

#rustlang in production: Volvo putting Rust in cars (in a good way, hahah) corrode.dev/podcast/s03e...

Doing what you enjoy is the best way to do things. We are happy to show what me, @geoheil.com, @mirko-t.bsky.social, and our team have been doing for the last 8 months

github.com/feldera/feld... looks like it has learned a lot from #duckdb but for the domain of #streaming

If you use your encryption key for big query the temp result tables are not encrypted if you don't set the default key per project. We found this out after setting the policy

Trying to grasp what our data governance team did last 6 months 🤯 very hard topics

If your dbt models run in dbt cloud where are other important jobs running? #databs

Spring 2025 @cmudb.bsky.social Course: OPTIMIZE! A Journey Through Database Query Optimization This is a survey course on how to build a SQL optimizer from scratch covering foundational and state-of-the-art implementations. All lectures available on YouTube: 15799.courses.cs.cmu.edu/spring2025/

Huh interesting move, hope more rust inside dbt

Vertically stacking data should be easy and fast! DuckDB's UNION ALL BY NAME capability matches on column names instead of position - as it always should have been!

Good thing about my bad english is you know it is not a chatgpt paste

I dont know about you but i am a bit pissed from people who use chatgpt for interaction with others and think that is not visible. It is just bla bla sentences

Enterprise needs somebody to blame for failures 🤭

In enterprise it is very hard to discuss some implementation that is marked experimental.

What are must have videos papers and blogs around big querie? How to learn about internals and best practice? #dataBS #bigquery

Buckle up because we're banging into the new year with my annual retrospective of the last year in databases! Highlights include license change blowback, Databricks vs. Snowflake gangwar, @duckdb.org's shotgun weddings, and buying a quarterback to impress your lover: www.cs.cmu.edu/~pavlo/blog/...

My path: cs degree with reinforcement learning bachelor -> junior data scientist -> data engineering -> data platform engineering

My work is not about implementing a data use case but enabling others to make it easier

Is there some duckdb-kafka extension?

I love the Fish shell writeup on their Rewrite-it-in-Rust journey. It's really nuanced: there's some good, some bad, some introspection on their own mistakes. But most importantly: they did it! 2604 commits over ~two years of work across > 200 authors. Ship ship ship! fishshell.com/blog/rustport/

Wrote a post about debugging @dagster.io in VSCode / Cursor! If you're like me and use a lot of notebooks to write code, this might be a great alternative that relies on VSCode / Cursor debugger for a smooth development experience with Dagster or similar libraries. davidgasquez.com/debug-dagste...

It is not about real time but just running the next step when the input is ready for it

Yesterday we had one code location today we have five. It is so nice to see our implementation and hard work used and the dagster graph growing

A self-serving data platform is not just giving access to the last data product but giving a tool and guidance in the hand to do something with this

There are two different parts to understand when mastering #dagster. 1. Technical implementation: dagster ui, deamons, state db and code location -> helps to develop 2. Difference between task and asset-based orchestrator -> helps to think about how to solve problems with dagster

TIL when you use "from module import function" python still loads the entire module. Saw this in my memory flame graph. I moved some functions with heavy deps to a separate file and it deleted a whole "flame" of memory allocations

Oh damn there’s also this: github.com/hafenkran/du...

resume/cv tip: always write in single column. never use double column or something.

To get to the business value you need to go through technical steps first. Having hard time to explain this

Give this to your boss youtu.be/fr5lIchF6pw?...

What are skills of a good data platform enginner?

Comparing dbt exposures docs.getdbt.com/docs/build/e... and dagster bi tools integration docs.dagster.io/integrations.... It is just a concept and one more reason to build your platform around the orchestrator

Alexey's @clickhouse.com talk here is hilariously off-the-chain. I highly recommend watching it: www.youtube.com/watch?v=jmVx... You can hear me yelling at him from the audience at 1:09, 2:40, and 7:33.

@milicevica23.bsky.social thanks for sharing - www.youtube.com/watch?v=8Woy... is a nice video about the power of #sql for #data-engineering

Fun fact: DuckDB is faster at counting the lines of a CSV file than the UNIX word count command – and it also parses the file to identify its dialect (separator, quote character, etc.). Here are the timings for a 3 GB CSV file: 2.966 seconds for `wc -l` and 1.261 seconds for DuckDB.

I would be happy to chat about dagster in prod? Is there somebody who is using Dagster and would like to share their experience?

Looking into dagster tableau integration. It is very nice when you have a good base concept for tool integration. Everything is going around the translator and asset definition.

Just one tool doesn't make business value and not everyone can be an expert in everything

When you speak with dbt vendor or gcp cloud they speak just about their products, when you speak with dagster people they speak about integration with other tools

Can i have a feed on bsky to show me content from people just from one starter pack?

Starter pack == facebook group

I like when a starter pack is more specific around a tool or concept because if you are interested too then you get fast access to the right people providing content for it

The problem in the enterprise is not big data but data engineering assessment and strategic data importance over the operational systems

Dagster's new layout with clear layers of abstractions, integrating stateful assets while running on any compute (processing layer) looks really cool. Great talks by @schrockn.bsky.social, as always. Recommend to re-listen on youtu.be/rB2nNEEIRBE?....

Is there a starter pack for dagster?

I am so grateful to have an opportunity to work with dugster full-time and build our platform around Dagster. I am happy about the future youtu.be/rB2nNEEIRBE?...

What enterprise company do you know that has a good data platform?