Part 3️⃣: How we designed our ingestion pipelines (and why) Our ingestion design is intentionally dead simple 😅 At its core, it’s a classic producer → consumer pattern, with a few variations depending on how data is produced.

Part 2: When real-time integrations went down just after launch The first week after launching Community at @buffer.com, we experienced 2 major outages with our real-time integrations. Both were outside our control, but still high impact for a newly launched feature.

Part 1: How we switched ON Community across 6 networks and 8 API integrations On the 12th of November we launched Community at @buffer.com We launched across • Threads • Instagram, 2 different APIs • Facebook • LinkedIn, 2 different delivery mechanisms • Bluesky • X

Now that the dust has settled from @buffer.com Community launch, I’m starting a mini-series on the engineering work behind the data ingestion and processing pipelines. It’s more of a journal than a guide. Sharing what we built, why we chose certain paths, and what surprised us 😅

We've been going through some photos from when we first moved into our house and stumbled upon my home office setup from 2018. It took me about ~1.5 year to paint the room, lay the floor and buy a desk. And still I prefer coding from the couch, sitting in some weird position. 😅

need to test some more

Big day for our team at @buffer.com! We have just launched Community, a calm and focused way to manage your comments across Threads, LinkedIn, Instagram, X and right here on Bluesky. It’s live now and available on every plan, even Free. 💬

Trying a threaded post

Getting last things ready for today 🗒️

A calmer way to keep up with your community. Does this make sense for Bluesky? Yes, it does 😄 November 12, with @buffer.com. Stay tuned! 💛

I'm just going to drop a test post here 🧪

Way back I decided to list both my formal name and my calling name on LinkedIn.
 Why? Because outside of Greece, almost nobody knows that Dinos is just the short version of Konstantinos (in Greek, “NT” sounds like “D”).

Firehose powered Buffer 💪

Firehose power ftw 🙌

Multiple images

one with a GIF

Sorry for the noise 😔 [video post]

And one with an image

Now it should be ready - plain text test

Yet another firehose test

Added some basic stats on my dev env, from the firehose events processed and filtered. How meta is this, to post about the Firehose in order to test the Firehose? 😂

Hermes + Firehose FTW

Everything seems to flow in. I believe we have something ✅ 🧯

With an image

Last last message

One last post, for now

Debugging decoding 123

Decoding seems to fail 🤔

Debugging decoding 123123

Post detected and decoded?

Decoded message <----

And another test

Testing the firehose, so I will be spamming with test messages for a while. Sorry 😅

Bluesky firehose test, 123 123 1111!!

Building real-time integrations with social media platforms feels like trying to drink from a waterfall🚰 I’ve spent over 10 years working with data ingestion pipelines and third-party integrations. It’s not the only thing I do, and I’m far from an expert, but it’s an area I keep coming back to.

🌱 The Cucumber Chronicles - Day 78: The First Harvest 🌱 We picked our first cucumber! 🥒✨ A little on the seedy side, but not bad at all. 😎 We’re drying the seeds now, ready to carry Gurkie’s legacy into the next cucumber season. 🌿♻️

🌱 The Cucumber Chronicles – Day 75: The Leaning Trellis 🌱 Gurkie is on recovery mode! We trimmed the dead leaves and focused on saving what was still alive and it seems to work. The trellis we added last month is struggling though.

One of our most requested features is here: Bulk Upload! 🎉 You can now upload up to 100 posts to Buffer in one go; straight from a CSV. No more copy-paste-repeat. Just plan, upload, and go. 💙 🔗 Full scoop: buffer.com/resources/in...

🌱 The Cucumber Chronicles – Day 58: Home Alone 🌱 It’s been two days since we left Gurkie home alone, and I’ll admit we haven’t missed him too much. 😅 He’s got some basic watering systems in place. I’m hoping they’ll hold up over the next two weeks while we’re away on vacation. 🤞 #CucumberChronicles

I have an old Sony A5000 (10+ year old) that’s been collecting dust in my closet, and I’d like to repurpose it somehow, but I’m out of ideas. #cameras #makers #tech

We’re building a new public API to make integrating Buffer’s social media tools into your workflows easier than ever! 🎉 With a more extensible, developer-friendly API; we’re making it easier than ever to build custom solutions on top of Buffer. 🔗 Get early access: buffer.com/developer-api

🌱 The Cucumber Chronicles – Days 49 & 50: Difficult Moving to the New House 🌱 Gurkie’s move to the big new pot wasn’t as smooth as we hoped. In the process, some roots were damaged and a good portion of the sun leaves lost their strength. #CucumberChronicles #UrbanGardening

We’ve leveled up our server environment at Buffer! 🚀 Meet BIBEs - our new solution that gives engineers faster feedback loops and more confidence when testing and shipping. peteremil.com/buffers-bibes/

🌱 The Cucumber Chronicles – Days 47 & 48: Little Jungle 🌱 Gurkie has officially turned into a little jungle. 🪴🌿 He’s completely outgrown his current pot, and it's getting harder to pretend otherwise. #CucumberChronicles #PlantParenthood #Repotting #LittleJungle #UrbanGardening

🌱 The Cucumber Chronicles – Day 43: Bites on the Leafs 🌱 Gurkie is growing strong. We’ve picked up a new rectangular pot and some trellis pieces. Tomorrow we are moving it to its new condo. 🪴✨ #CucumberChronicles #PlantLife #UrbanGardening #LeafWatch #BotanyAdventures #GurkieUpdates

@bsky.app your firehose, is just raw power 💪. I love it

This is another test with the firehose

One more test with the firehose

I'm testing the bluesky firehose 🌊

🌱 The Cucumber Chronicles – Day 39: Growing Flowers 🌱 Gurkie is not just growing strong. It's growing flowers! 🌼💪 After some reading, I found out there are male and female cucumber flowers and only the females grow actual cucumbers. #CucumberChronicles #FlowerWatch