How big should your data team be? Data teams are often oversized. A company of 200 people rarely needs 15+ data staff, usually 5% of org size is enough dataactionmentor.com/knowledge-ba...

The amount you love someone is proportional to how often you Ghiblify their pictures.

This week I look at agents. I think this is a new way to build where we don’t intentionally build code-based software. open.substack.com/pub/davidsj/...

BERT and ERNIE! 😂 tracking.tldrnewsletter.com/CL0/https:%2...

I don't usually share photos of my family on social media for good reason, but I'm happy to share these ones!

This post encapsulates how I feel about the current state of LLMs and doomers etc. Really great read: fly.io/blog/youre-a...

So when I've attended Snowflake summit before, I've usually written a blog post talking about the new features released, etc. Is someone going to do that this year, given I didn't go? 😊 #datasky #databs

It is possible to build machine learning systems which punch up instead of punching down.

Got a cool story about something in the data engineering space? You should 💯 submit it as a talk to Current 2025 in New Orleans 😁 Do it! Now! CfP is open until 15th June. sessionize.com/current-2025... (Pro-tip: you only need an abstract at this point; writing the talk can be later 😅) #dataBS

At the London Data Practitioners Meetup with @pedramnavid.com @jayatillake.bsky.social @rittmananalytics.bsky.social and the London Dagster community

Doctor’s orders 🫡

I still think this is the biggest prize in AI. If Siri could actually do most things you do on a phone manually... 9to5mac.com/2025/04/22/s...

Has anyone tried Llama 4 Maverick yet? How big a machine does it need to run locally? @simonwillison.net

Looks like Nintendo became the best at console FPS.

Once again, we've devised a derogatory name for something many of us are doing: "Vibe coding". Just like "Citizen Data Scientist", "Excel Data Analyst", and many other terms made to belittle by the supposed true artisans that came before. open.substack.com/pub/davidsj/...

Clickbench says @duckdb.org is a great analytics database www.mooncake.dev/blog/clickbe...

I enabled the Cube AI API Slack App today, and I found a little treat 👀 Shout-out @mjirv.bsky.social and @jayatillake.bsky.social! If you're curious how easy this is to set up, check out: cube.dev/blog/make-yo...

A couple of weeks ago I wrote a post about @pola.rs Cloud, and as coincidence would have it, DeepSeek then releases smallpond - a way to use @duckdb.org scaled out. It's pretty cool that an org has managed to build this, but is it the right way? davidsj.substack.com/p/many-ducks...

🗣️ It was a real honour to speak at the very first Forward Data Conference in Paris last November, not to mention being introduced with a special piano verse just for me! 🎹 www.youtube.com/watch?v=QNQW...

🏭 @tobikodata.com released a great new feature for SQLMesh last week called blueprinting. This is a feature that the dbt Labs community has been asking for for ages, and I’ve been wanting it too! open.substack.com/pub/davidsj/...

🦆🦆🦆🦆 everywhere these days count.co/blog/announc...

New episode: “Semantic Layers: The Missing Link Between AI and Data” with @jayatillake.bsky.social . We discuss how semantic layers bridge raw data and AI, achieving 100% accuracy for natural language queries, and what’s next for LLM-powered data pipelines. 🎧 https://techontherocks.show/14 🎧

I've been interacting with r/dataengineering a bit recently, and I'm impressed by the size and engagement with the community there, but... Folks on #databs / #datasky seem much less likely to just mouth off without thinking or reading! I think I much prefer what we have here on BlueSky.

Yep! And the new(ish) rust-based data engine also brought the ability to do multi-stage calculations. Bonus - "Tesseract" is an amazing name for a Cube engine 😀 cube.dev/blog/introdu...

@tojyouso.bsky.social is this you? 😂

pola.rs/posts/polars... This so reminds me of @motherduck.com and @duckdb.org, I guess the open source Polars is as good as it can be on one machine, and Polars Cloud is as good as it can be on many.

Cancer incidence rises when people live longer and have access to preventive care to diagnose it in time to treat it, and effective treatment increases cancer prevalence because we count the people who are still alive – this is a good thing, and I'm so fucking tired.

🚀 Exciting times in the data transformation space! Recently, dbt Labs acquired SDF Labs, while @tobikodata.com acquired Quary (YC W24)—both promising Rust-based frameworks. So is SDF better than SQLMesh? No. open.substack.com/pub/davidsj/...

One question that comes up for me on calls with companies- why can't I just build a semantic layer in my data platform? My brilliant coworker Morgan wrote a great post on why @cubedev.bsky.social is more beneficial as a semantic layer. Enjoy! cube.dev/blog/why-cub...

Announcing a new Substack! Focused on the theme of applying AI to analytics. I'm collaborating with Artyom Keydunov and Brian Bickell on this, and we'll bring you interesting, trending articles and news on our theme. aiforanalytics.substack.com/p/ai-for-ana...

Is there anything specific you want to know about OLAP cubes, especially newer open-source systems (e.g., ClickHouse, Druid, Starrocks, Pinot, Kylin, Doris, Cube, and more)? Which one is your favorite and why?

DuckDB out here making me avoid other SQL dialects

DuckDB 1.2 brings serious performance to real-world queries! Selective joins, IN lists, Top N, partitioned parquet reads, and more get much faster! Performance is also incrementally improved ~across the board. Plus, the CSV reader improves (latin-1 for Windows folks!), SQL is friendlier, and more!

@anthropic.com's Dario Amodei wrote a really thoughtful piece on the @deepseek.bsky.social situation and how it relates to overall macro end games with AI: darioamodei.com/on-deepseek-...

It's weird how people see these two as mutually exclusive.

They need to substantiate this. Otherwise, it sounds like sour 🍇. www.macrumors.com/2025/01/29/o...

OpenAI right now

I didn't realise there was another way... I either discover code or copy-paste and edit an existing solution... plus now AI. jimmyhmiller.github.io/discovery-co...

If you're considering SQLMesh, @jayatillake.bsky.social has a great blog series from the end of last year that does in depth comparisons with DBT

Just had my first ever Wendy’s… not bad.

Any company whose mission is to “to ensure that artificial general intelligence benefits all of humanity” would be ecstatic about a capable open source model like Deepseek released. Any company which has a mission of making money off closed-source models would be devastated tho.

𝐇𝐨𝐰 𝐝𝐨 𝐲𝐨𝐮 𝐫𝐮𝐧 𝐝𝐥𝐭 𝐨𝐧 𝐀𝐖𝐒 𝐥𝐚𝐦𝐛𝐝𝐚? (tldr - 100x cheaper than Saas) Here are code templates: Github: github.com/codingcyclis... Blog: dlthub.com/blog/dlt-aws... We use GCP ourselves: dlthub.com/blog/dlt-seg... #databs

DeepSeek released a whole family of inference-scaling / "reasoning" models today, including distilled variants based on Llama and Qwen Here are my notes on the new models, plus how I ran DeepSeek-R1-Distill-Llama-8B on my Mac using Ollama and LLM simonwillison.net/2025/Jan/20/...

I really like working at Cube. Selling doesn't feel like selling when I get to talk about a tool I genuinely believe in. I can't believe I've been here less than 6 months. It feels both way longer and way shorter at the same time.

Are the SDF founders/team on here?

Today, we’re announcing that dbt Labs has acquired SDF Labs. This a monumental day for dbt Labs and the entire dbt Community Thread on what that means 🧵 www.getdbt.com/blog/dbt-lab...

dbt Labs has acquired SDF Labs! Early reviews:

SDF acquired by dbt. Didn't expect that. Not sure yet what to think. Blog post on blog.sdf.com/p/dbt-labs-h...

Shill away You can shill away Shill three days a week - on LinkedIn That's what it's there for www.youtube.com/watch?v=2twL...