Some open-source Spark drama if you want some lists.apache.org/thread/0558x...

Executives: "We need Gen AI" Engineers:

Mentally I’m still at JDK 8 and 11. 23! Image borrowed from LinkedIn

What would it take for people on LinkedIn to stop posting about Deepseek? I think we should shut it down for a few days and see if people care about it. Most are chasing the trend and spewing random posts.

How would this integrate? Perhaps enhancing the Dynamic tables?

When they invite the Engineer to present to the C Suite

It’s telling how AI as an industry is progressing when you see the people freaking out about Deepseek are mostly VCs.

Deepseek is everywhere.

Found this comprehensive guide to AI and how to make sense of the various concepts. www.leverage.to/learn/dev/ai...

Tomorrow is restart your Apple TV+ subscription for Severance Season 2 day. Also, maybe catch Silo. And re watch Slow Horses. It’s going to be a long work day sigh.

Onehouse with their Compute Runtime across table formats. It seems like a good direction but I’m curious if those benchmarks are accurate. 30x query perf, 10x faster writes, I wish there was a demo or something to showcase these. www.onehouse.ai/blog/introdu...

Tobiko acquired Quary: tobikodata.com/tobiko-acqui... The transformation game is back on again

To whomsoever it may concern: Acquisitions are great. Integrations are hard. Good luck.

Worth reading this blog about Arrow: arrow.apache.org/blog/2025/01...

How AI is messing up things across the board Also, some things could be a Slack message than an email

Anytime I open LinkedIn it’s either - Random person with a take on AI, usually hyping - Someone talking about random topics like why is Test Driven Development - At least one reference to Clean Code I can’t imagine how that platform is useful to anyone who wants to get an idea of the industry.

Most companies need a better backend API than anything close to AI or LLMs. Just do the basic thing right first. You don’t need to jump to the fancy objects.

Somebody said it

What the Modern Data Stack felt like.

2025 is the year of realizing that hourly batch or micro batch is sufficient over continuous processing. The trick is to ask what your stakeholders actually expect to happen and then build accordingly.

Happy New Notebook Day. My first MUJI notebook. The pen has been great! Got these on my trip to Seoul.

The Data Engineer vs the Data itself

My 2025 New Year’s Resolution is to tell everyone in the Data and AI space that you can’t build good AI without quality data. Focus on fundamentals rather than fancy.

Looks like PySpark, Ray, and Dask have a competitor - Bodo. Has anyone tried it yet? This is the claimed execution time. github.com/bodo-ai/Bodo

I'd be surprised if it gets practical usage in the next 10 years.

I find it strange that people still think Hudi and Delta have a chance despite the fact that Iceberg has clearly won the table format race. If all major vendors and platforms are putting their bets on one format, why wouldn't an org move to it or stay in it?

And just like that, we all stopped doing Advent of Code this year.

The most important task to do after going home from a conference: Unsubscribe to all the vendor emails that you are getting because you scanned your badge for socks.

There’s Generative AI and then there Adversarial AI. Anyone have a good source / reference to understand and grasp the latter? #AWSreInvent

This article is a really good summary and detail about S3 Tables. dataengineeringcentral.substack.com/p/aws-s3-tab...

More reason to bank on Iceberg. AWS now has S3 tables which are geared specifically for Iceberg tables. If I can read this correctly, it looks like an integrated Rest API catalog behind the scenes. Can integrate well with Glue too. aws.amazon.com/blogs/aws/ne...

Walk metrics from Day 1 at AWS ReInvent. Most of this was expo hall walking and to and fro between hotels. #AWSreInvent2024

Data Scientists vs Data Analytics Engineer

What are the 3 best books you read this year?

Remember, if your Engineering team isn't fixing that bug in this week, it's a 2025 problem. Holidays are upon us.

I am curious how the Databricks IPO will look like and its impact on the data industry as a whole

AI is not replacing people. People can't even sell it to other people without, more people.

2025 will be the year of batch jobs. Just like 2024, 23, ….

If there’s one language that could dethrone Python, it would be Golang, imo. Ship a go binary in a docker versus a Py library and tell me you see it. But, existing investment in developing Python libraries and tooling and the time it might take to build it in another language are large barriers.

@seattledataguy.bsky.social is here too!

Is anyone else in Data BS attending AWS reInvent next month? I'm headed there for the first time. Mostly talks and other events.

Don’t know how true this is but a rumor is that Databricks will IPO Q1 2025. Anyone know more?

Are data companies that raised capital during ZIRP doing ok? I don’t think most would justify the valuation that they raised at. Either lost valuation multiples or shut down. Possibly acquired. Any one know of cases?

Data Contracts, Data Mesh, Data Fabric still a thing or we’ve moved on?

The feeling you are going to feel for a bit is like when you wanted Daenerys to get the throne but they ultimately made Bran king.

@eczachly.bsky.social is here too.

Is there a single source of truth for election data? Can I download a bunch of csvs somewhere?

@joereis.bsky.social is here!

The real winner tonight are the dashboards we’ve been staring at collectively as a planet

We ran 80000 simulations today. Took a few seconds since ran it on DuckDB