It looks like the Bluesky API went down for about 30 minutes, and we had a backlog of pending posts to clear. The Bluesky Feed Creator system detected this and recovered automatically as soon as Bluesky's API came back online 😁. You may have also seen other errors in this time, e.g when blocking.

And we're back up to real-time, took under 15 minutes to process 5 hours of posts! 💪💪

We're seeing similar firehose issues today, and we are currently catching up on 5 hours of posts! Additional monitoring has now been added to alert us to this situation automatically so we can deal with it promptly going forward.

Our firehose connection has been giving reduced posts for about 2 hours. This has been fixed and we are catching up now. Thanks for your patience!

We had a bit of downtime overnight, our cloud provider was having trouble with the physical hardware our service was on. It looks like this has been resolved but we will keep an eye on it.

We had an outage beginning 35m ago, which was affecting some regions including the US, and which has now been resolved.

A bug has just been fixed that was stopping the new "Search and insert posts" dialog from actually inserting the posts in some cases.

💡🚨💡🚨💡 Free feeds now draw from a sampling of the Bluesky network rather than its entirety. If your feed is public, you'll see it picking up fewer posts. To keep your feed content fresh, you've got some options: 1️⃣ Focus your feed by pulling posts from specific users/ communities. This is unthrottled!

We've fallen behind by up to 1 hour overnight. Additional capacity has been added to work through the backlog, so things will be back to normal soon.

If your feed is frozen, please set "'Pending Posts' behaviour" to "Wait for labels". There's an odd issue where feed caches won't update without the pending posts service.

One of our ingress services is having issues, this is causing some requests to occasionally fail. A temporary stop-gap fix has been applied to mitigate this. Thankfully, this service isn't essential and can be removed if needed.

All caught up, feeds are now operating as normal. For future reference, we publish how far behind our indexers are (at the bottom of) blueskyfeedcreator.com/stats

Hi everyone, Josh here again. Not really sure what happened…but I’ve done some things and feeds will start to move again.

And we're back to normal, thanks everyone!

Morning everyone, apologies it looks like our API connection was disabled overnight again. We are going to look further in to this today. Pending post are currently 2 hours behind but we are reconnected and catching up quickly.

We're doing a small resilience upgrade which will mean we might miss some posts over the next 30 minutes. Apologies for any inconvenience.

Pending post delays and our API connection have now been fixed, thank you for your patience everyone

Morning everyone! Pending posts are currently about an hour behind, and it looks like our API connection was disconnected overnight. We are looking in to this now.

Reposts are currently causing feeds to stop updating in Bluesky as they are added from the firehose. Previously this would show InternalServerError. You can work around this by re-saving your feed in BSFC, which clears the cache and fixes the feed. Will be working to fix this tomorrow.

Pinned posts have been restored, thanks everyone 😃

Pinned posts are currently missing from feeds, we are working on a fix.

🙃 Error loading feed. Please refresh, or if the issue continues, contact @blueskyfeedcreator.com

We're back up to date again, thanks everyone 😁

We're about 10 minutes behind at the moment. On the road but will address this within an hour.

We finished catching up about 15 minutes ago, thank you for your patience. -Eddie

We have fallen behind by about 30 minutes, but the issue has been identified and fixed, so we will begin to catch up quickly now. I will update later when we are back to real-time. -Eddie

We're aware of frequent logouts. We detected the issue earlier this week and have been working to fix it. - Josh.

Quality of life improvement for feed admins. During downtime, Bluesky would guess an error message, leading to incorrect and confusing errors like “NotEnoughResources” I’ve made a change that stops bsky from guessing, leading to less confusion from viewers and hopefully less emails for you - Josh

Quality of life improvement for feed admins. During downtime, Bluesky would guess an error message, leading to incorrect and confusing errors like “NotEnoughResources” I’ve made a change that stops bsky from guessing, leading to less confusion from viewers and hopefully less emails for you - Josh

Bluesky Feed Creator is fully operational, our cloud provider’s outage was resolved. Thanks for your patience.

We're aware feeds are down, it appears to be a cloud provider routing issue and out of our control. We're working to mitigate it. - Josh

We’re aware of feed disruptions and addressing the issue

Due to a typo, BSFC temporarily went down. Thanks to our new systems, we fixed the issue within 30 seconds of getting a report. Apologies for the issues. - Josh

We have deployed new server architecture for you all. BSFC should feel faster and be more reliable. These changes include our promised overhaul to redundancy, making downtime much less likely. - Josh

We experienced momentary downtime due to a memory leak in old VPS architecture that hasn’t been fully migrated. BSFC was down for 5m and is now back. We have had lessons learned since our downtime months ago, and this was old VPS infrastructure still migrating. - Josh