I wrote something about *Real* Data Engineering… then added the AI bit martinchesbrough.net/data-and-ai-...

I am signing up for Chad Sanderson's Shift Left Data Manifesto www.gable.ai/blog/shift-l... It is more than data contracts but we all need to start somewhere ...

Adding more layers to your data architecture is unlikely to make it easier to manage data or make it work better - consider alternatives martinchesbrough.net/against-laye... #dataengineering #dataarchitecture #datamesh #softwarearchitecture #datamodeling

I've been meaning to write this for a while - it is titled "Against Layers" and is a plea for pluralism in data architecture.

If you work with groups and teams, the Fun Retrospectives website is an excellent resource for ideas. I enjoy using my favorite techniques and experimenting with new ones to keep things exciting. #QiSky #Facilitation

A ClickHouse query optimization guide. SQL-based observability in the wild. PIVOTing in ClickHouse, and more. It must be our final newsletter of 2024! 🎄

I was talking to a friend who’s very into Gen AI at a recent Xmas party and I put it to home that my views is that the LLM model “wars” are now pretty well over …

I've had a lot of fun today playing with Gemini 2.0 through aistudio.google.com - I got it to read an academic paper on my screen (Claude does this as well but Gemini does it better)

Another day, another book recommendation. This time it is "Managing Data as a Product" by Andrea Gioia, published by Packt www.packtpub.com/en-au/produc...

Calling univ profs & lecturers! If you want to bring concepts from Doughnut Economics into your teaching, we've just launched a webpage bursting with resources for you: 7 slide decks, reading lists, videos & activities - all open access. Dive in & pls reshare! doughnuteconomics.org/university-c...

Because of a discussion on misuse of the term “data warehouse” I referenced a section from James Serra’s book “Deciphering Data Architectures”. That led me to re-read the book. It’s a great book BTW and worth a read.

I like using BlueSky as a short-form essay writer so here goes with my latest thoughts. The essay might be titled: "The Future of Software is AI ... NOT" or "Software eats the world ... this time with an AI sauce" or "The Future of Business is ... unreasonable AI agents" or something else

I’m having great fun with building AI agents but the various agent orchestration frameworks are still immature (Autogen 0.2.39, LangGraph 0.1.40, CrewAI v0.83.0, LlamaIndex v0.12.2). It feels like early Hadoop days. I wonder how fast these frameworks will proceed. I’m excited for what happens next

Just listened to this conference talk from @simon.bvssh.com - well the dude drops more names in a 30 minute talk than I can fit into a month - BUT it's a great talk! youtu.be/ASE6Pt3fnxM?...

Apply Data Mesh principles to solve case studies, an online workshop with Jacek Majchrzak. New dates added. Take advantage of the early bird: https://buff.ly/3HWRNrX

I’m keen to get some feedback … I’ve been playing around with building Gen AI agents using Autogen, Langchain and Llamaindex. And I’m coming to the conclusion that these are just small apps or applets.

The 2nd time that you read a book I think you get a lot more out of it than the first time. This was worth a 2nd read www.manning.com/books/archit...

I made an infra engineer starter pack. Folks posting about databases, stream processing, durable execution, orchestrators, service meshes, and more. go.bsky.app/SCZe42X

How can you tell if a specific agile practice is relevant to your current situation? About a week ago I started parallel conversations on the theoretical underpinnings of agile methods in different communities to try and answer this question. I received some very interesting responses.

How can #WardleyMapping, #DDD, and #TeamTopologies be combined in a structured way to optimize systems for flow? I developed an opinionated structured approach and visualized it in an Architecture for Flow Canvas. You can find more information at susannekaiser.net/architecture-for-flow-canvas.

ICYMI I recently interviewed the creator of @clickhouse.com, Alexey Milovidov. We dove deep into what ClickHouse is so fast, why AI companies love it (hint: its had a vector data type since 2012), rewriting Zookeeper (yikes), and how math informs database design. www.rilldata.com/blog/rill-cl...

Has Data Engineering forgotten about socio-technical architecture? I think so and I wrote up a blog post about it. www.linkedin.com/posts/martin...

Just did this course. I like the parallel video-notebook screen so you can play with the code as the course runs. It is very much a beginner style course (that's ok) and any dev would be well placed to take an AWS-GCP-Azure-Anthropic-other course to get a broader knowledge of prompting.

"Blah blag blah decoupled blah blah blah." Decoupled. Decouple. Decoupling. Decoupled. “You Keep Using That Word. I Do Not Think It Means What You Think It Means.” -- Inigo Montoya Semantics matter. Thank me by subscribing and watching my following episodes. youtu.be/ZffqJxLULx8?...

AI has evolved from a specialist nerd science into an everyday part of your product development team. Here's my recent experiences that support this argument. But first some history (to help make the argument)

Reinforcements are coming bcounter.nat.vg

I think management is the hardest job in the world. Why would anyone that wants to do a good job ever take up this role? Let me explain …

My colleague, Pilar Esteban, wrote a lovely short piece on re-thinking traditional roles. medium.com/everestengin... I’ve got some thoughts that I’ll share in the thread

This book "Under the Hood" by Stan Slap (what a name) is about shaping employee culture and it sounds like it is worth a read. The 12k word summary is here: commoncog.com/under-the-ho...

I reckon bsky.social is the way to go - definitely better than mastodon.social - not as well populated as x.com but we can make it work!!!

I'm gonna try a little experiment - post the same thing on X, BSKy, Mastodon and LinkedIn to see if anyone says anything about it. This is an email that Steve Jobs sent to himself about a year before he died of cancer - is it profound?

I was looking at another messy data warehouse and thinking what would Jeff Bezos do? So I wrote this: medium.com/everestengin...

Let's talk about data products. What matters is building them and the value they deliver, not the definition of a data product. Check out www.linkedin.com/posts/martin...

My colleague, Melika, just published her first blog post ever - it is really well written and an informative story of a data project. Have a read: medium.com/everestengin...