Why we maintain a ClickHouse fork at Tinybird (and how it's different) www.tinybird.co/blog-posts/w...

I wrote about how to ingest 1b rows per second with ClickHouse. The point of the post is not the number, pushing 1b rows is “easy” the actual challenge comes when you need to do that in a reliable way, that’s what I try to explain

I'm fucking tired of clickbaity content, people writing bad content just for SEO or not going into details because "people have a short attention span". Maybe that's true but some people appreciate long-form, detailed and super technical content. Today we posted 5 high quality engineer posts:

The second part of "handling clickhouse clusters at petabyte scale" www.tinybird.co/blog-posts/w...

We've been operating petabyte-scale cluster for 5 years now, especially dealing with low latency use cases with large amounts of data, some of them under 10ms (for reference, most big data systems latency is 1 second at best). www.tinybird.co/blog-posts/w...

We need faster feedback loops failingwithdata.substack.com/p/we-need-a-...

I'd appreciate if you could upvote us in ProductHunt. It's about our latest product iteration www.producthunt.com/posts/tinybi...

🌅 Tinybird Forward is here » Forward is a major evolution of Tinybird, designed to make shipping software with big data requirements faster and more intuitive. No complex infra project. No context switching. No esoteric architectures. Just code. (🔊 sound on! 👇) youtu.be/vaSjWu3XFdY

Production hurts. If it doesn't, it means you didn't try hard enough.

A new LLM version release with a new super powerful option: --schema An mini BI example, in this case I'm querying a parquet file with all the events coming from gitlab tickets webhook

Data intensive SaaS products will love to leave your data in icerbeg/s3 Guess who is going to pay all the storage and operation costs? 🤣

We’ve got a deep technical post almost ready on real-time log analytics, based on our experience designing these systems with companies handling trillions of rows.

We are using LLMs in the next Tinybird iterations but we are trying to be subtle, not going with the conversational interface. So for example, when you want to create synthetic data to test your logic you can do 𝚝𝚋 𝚖𝚘𝚌𝚔 𝚝𝚊𝚋𝚕𝚎

I though the LLM and IA thing was going to open a new world for UI design but the reality is that we are all doing chat interfaces Maybe that’s the right thing to do and my expectations were too high

About Iceberg C++ implementations. There are already some of them working (Redpanda Data and StarRocks seem to have good implementations for writing).

I wanted to understand how agents work, so I created a super simple one to run SQL queries. I wrote about the process failingwithdata.substack.com/p/building-a...

I always like to go to basics and learn from them. Using libraries, frameworks and tooling is perfect but understanding why things happen helps make decisions when designing software.

I made a mistake running a SQL agent, I was sending the wrong data and I noticed that after some iterations it gets passive-aggressive: * This is the final time I will tell you: I need the SQL query. You are providing the error message, not the SQL code.

Building software that works is very easy… as long as you ignore all the hard parts.

Why would someone create your data pipeline tests by hand? When your pipelines have parameters and you need to check different behaviors creating tests for the different paths is a pain.

Simple question for people leading technical teams (hard answer tho): how do you create a writing culture in your company/team?

I love SQL open.substack.com/pub/failingw...

When you have a data project and you want your editor to play well with it, you can generate all the context cursor needs.

How I’d design the core of a cloud native database OLAP database (what Warpstream made with Kafka): - Storage: S3 tables - Short lived queries and ingestion: AWS lambda - Batch jobs: Spot EC2 instances Plus a vectorized execution engine.

I have a general one and another for Python that teaches cursor write PEP 735 compatible scripts! Shared them here in a recent post. davidgasquez.com/cursor-agent...

BTW it does not need to be a complex simulation, the output of a SQL query could be visualized this way

First time I see a job posting targeting AI agents (or people able to build them) Most likely a marketing/vitality technique but still interesting www.ycombinator.com/companies/fi...

I'm a big fan of creating ad-hoc visualizations to debug data, I'm debugging my lap timer and I needed some map visualization. I'm old and getting lazy so coding the tool is a no-go (it's not the funniest thing to do) but with Cursor I create a new index.html, I give it the output of...

I think a pretty deep understanding of ClickHouse (been using it since 2017) and I wanted to give a try to a different analytical database (with a different approach) so I'm testing DuckDB So far: really good I miss the way CH can store intermediate aggregation states. Is it possible in DuckDB?

Some learnings after helping +50 companies in high performance data engineering projects javisantana.com/2024/11/30/l...

This articule is good but this other one linked is even better. It shows how constrains are good to find creative ways to solve problems abagames.github.io/joys-of-smal...

Using LLMs to transform data --> failingwithdata.substack.com/p/llms-to-tr...

WERK96 dash javisantana.com/inspiration/...

As every year I collected the links and stories that caught my attention during this 2024. (Spanish, sorry) open.substack.com/pub/javisant...

December is my month without social networks (but WhatsApp because I have a family 🤣) I ve been doing this for 5 years and it’s painful and refreshing at the same time.

Some learnings after helping +50 companies in high performance data engineering projects javisantana.com/2024/11/30/l...

I feel frustrated when using any LLM chat thing I can't make it interact with my daily tools. So for example, I'd like it to open a browser and do some boring stuff on my behalf. Looks like MCP is solving that exact problem.

Do you use any *interactive* CLI tool to work with data? kind of visualize, filter and so on. I usually use ClickHouse local but I'm talking about something more "visual". I remember seeing some tools but I forgot the name.

cache invalidation, naming things, backfilling

Lean Analytics y como empezar en el mundo de la Ingenineria de la Privacidad en spanglish by yours truly: mastersofprivacy.com/es/nuria-rui...

I'm writing a blog post about my 4+ years of experience working with 50+ companies on [high-performance] data engineering projects. This is the last paragraph

Apache DataFusion claims to be the fastest Parquet reader out there. That's super good news because that will put pressure on other open-source projects fighting to be the fastest one and all the industry will save CPU cycles. datafusion.apache.org/blog/2024/11...

LLMs love fixing SQL

I used to hate Kafka, to me it was an over engineered piece of software that didn’t solve any real issue.

Voy a aprovechar lo tranquilo que está esto para contaros a mis queridos 8 seguidores mi opinión, no pedida, sobre eso de que Correos va a crear su propio banco. https://www.elconfidencial.com/empresas/2024-09-09/correos-lanzara-banco-propio_3958002/