Status: Localised network outage, hopefully back soon

Back up

Autolabelling service outage while I figure out why Ozone keeps dropping 502s

Okay I think we're back

Argh it's not PC maintenance and repair without at least 2 aggravating moments

Service update: Inference is going down while we continue to upgrade our systems

Service update: system upgrade complete (tho I still wanna upgrade CUDA to 13 at some point)

New GPU has been installed and we're just updating drivers now. Hold onta yer butts

Service update: scheduled downtime for system upgrade about to commence, will update here once complete.

Pretty relieved to realise my AWS credit doesn't expire until September next year (and that I've only used £2k of the £5k from the ATProto grant), that gives me a good few months to get XBlock onto stable local infra. I think if I can get Redis and Ozone back locally it'll be sustainable.

Finally upgrading some infra 🎉

Service update: more network fun.

Continuing to have intermittent connectivity issues with our London datacentre (Aendra's flat)

Service update: we had a localised network outage earlier so it might take awhile for the inference server to catch up with backlog

Service update: I've figured out why Ozone isn't accepting appeals, apparently there was a lexicon change or something which meant we had to explicitly configure it to accept appeals and not just reports. Appealing a label should work now, sorry that took so long to fix!

There is an issue with appeals not submitting properly, I'm investigating that today. I hoped it would be as simple as restarting Ozone but alas nothing in life is that easy.

Open to feedback: our moderation efforts have really slowed down so it's taking longer than it should to action appeals (apologies!). I'm thinking, should the appeals process be semi-automated? I've refrained because it's somewhat open to abuse but I suspect it would help more than hinder.

The Ozone instance appears to be struggling, please bear with me as I figure out what's up

📌 Info for devs: Repo: github.com/aendra-rinin... Current model (xblock-m, TIMM version): huggingface.co/howdyaendra/... Please feel free to inspect, interrogate, adapt and/or reuse XBlock code/models for your own projects, we welcome and encourage it!

The queue was indeed the issue, there's a 4M post backlog (likely from that week it was down) and newer posts were being put at the very end so by the time a post is processed it's super old and everyone's seen it already. 🙈 I've switched it to process newer posts first so hopefully that'll help! 🤞🏼

Restarted the auto-labelling process and am running into issues with the Redis queue, it seems stable now but will investigate further later.

I think there's a connectivity issue with the Ozone instance at the moment and labels aren't correctly propagating, or the model's needing retraining, or something. Please bear with me as I troubleshoot this. 🙏🏼

Uh sorry I have no idea how long things must've been down for. I should seriously migrate this machine to Linux already

I've moved all the firehose parsing to the cloud so expect to see some labels now. 🤞🏼

XBlock's down at the moment because my local network can't keep up with the firehose and it keeps knocking my household offline. Please bear with me, I'm going to lift the firehose parsing to cloud in the next day or two and hope to restore service.

Something is totally busted with our local infrastructure right now and I can't seem to keep the firehose consumer running for longer than two hours without it bringing down my entire network. I'm going to need to replatform some infrastructure this weekend, wooo yay lucky me.

We're back

We are once again having localised network issues

We're having issues with our Ozone instance, please bear with us as we sort it out.

We're having localised network problems at the minute

Service has been restored

Service is presently down due to a localised network fault

Sorry, we had some downtime! Should be back to normal now!

Okay I think we're back. Anything posted in the last like 5 days won't be labelled, because these random appview disconnects are impossible to anticipate and fixing them means resetting the appview's cursor. It's really stupid.

Still no dice, sorry. Service is effectively offline until this is sorted.

Service update: I think something's up with the AppView ingesting labels right now, I'm noticing posts I've labelled still appearing. I'll investigate that tomorrow but in the meantime apologies if things are working rather poorly.

Apologies, the Jetstream consumer went down again; I've since restarted it. I'm going to try to find some time this weekend to write some code to automate this, your patience is appreciated.

A reminder that this service is completely free and runs 100% on volunteer effort. Leaving mean comments when appealing a label just makes us want to do this unpaid labour even less. Thanks.

Okay we're back. Added a hotfix that should prevent that issue in the future.

The Redis queue is struggling, please bear with me, I'll try to fix things tonight

Service update: my girlfriend is an angel and loaned me a USB WiFi interface so we're back for the moment

Service update: I dropped something on the main inference machine and that burnt out the network card it seems so autolabeller is offline until further notice. Don't expect anything to work sorry.

I've disabled the "newsmedia" label for now because it was causing a lot of mislabels (anything text-heavy was getting caught; I may retrain this in the future, idk). Apologies if it's taken a while to get to your appeal, I'm halfway through the last 2 weeks and it's just me now.

Sorry, just noticed the queue fell over, I've since fixed it and autolabelling has resumed.

Had to fix Redis, auto labelling should be back again. 💚

This is understandable but extremely disheartening for the "vibes" Bluesky had and its continuing health. I felt the most connected with Bluesky when I was burning down the xblock backlog.

This project is on hiatus until further notice.

Also huge gratitude to my hardworking mods, who've had to deal with this bananas deluge of reports all November! It's been real rough, I owe y'all a pizza, a drink, or both! If you've seen an XBlock label between the start of the month and today, it was probably hand-moderated by one of them!

Massive thanks to @devingaffney.com, we've been chatting on and off for the last two weeks about this and between his experiments and Python code I now have a workable solution. I would've abandoned the TIMM version of the model (now in production) without him! I'll be forever grateful. 💚

Oh cool I think I actually fixed @xblock.aendra.dev's infra for realsies this time. Queue number is actually going down for once, with no extra inference beyond my old desktop running on local fibre. Someone want to send me a better graphics card? Really that's all it'd take I think.