Open to feedback: our moderation efforts have really slowed down so it's taking longer than it should to action appeals (apologies!). I'm thinking, should the appeals process be semi-automated? I've refrained because it's somewhat open to abuse but I suspect it would help more than hinder.

The Ozone instance appears to be struggling, please bear with me as I figure out what's up

📌 Info for devs: Repo: github.com/aendra-rinin... Current model (xblock-m, TIMM version): huggingface.co/howdyaendra/... Please feel free to inspect, interrogate, adapt and/or reuse XBlock code/models for your own projects, we welcome and encourage it!

The queue was indeed the issue, there's a 4M post backlog (likely from that week it was down) and newer posts were being put at the very end so by the time a post is processed it's super old and everyone's seen it already. 🙈 I've switched it to process newer posts first so hopefully that'll help! 🤞🏼

Restarted the auto-labelling process and am running into issues with the Redis queue, it seems stable now but will investigate further later.

I think there's a connectivity issue with the Ozone instance at the moment and labels aren't correctly propagating, or the model's needing retraining, or something. Please bear with me as I troubleshoot this. 🙏🏼

Uh sorry I have no idea how long things must've been down for. I should seriously migrate this machine to Linux already

I've moved all the firehose parsing to the cloud so expect to see some labels now. 🤞🏼

XBlock's down at the moment because my local network can't keep up with the firehose and it keeps knocking my household offline. Please bear with me, I'm going to lift the firehose parsing to cloud in the next day or two and hope to restore service.

Something is totally busted with our local infrastructure right now and I can't seem to keep the firehose consumer running for longer than two hours without it bringing down my entire network. I'm going to need to replatform some infrastructure this weekend, wooo yay lucky me.

We're back

We are once again having localised network issues

We're having issues with our Ozone instance, please bear with us as we sort it out.

We're having localised network problems at the minute

Service has been restored

Service is presently down due to a localised network fault

Sorry, we had some downtime! Should be back to normal now!

Okay I think we're back. Anything posted in the last like 5 days won't be labelled, because these random appview disconnects are impossible to anticipate and fixing them means resetting the appview's cursor. It's really stupid.

Still no dice, sorry. Service is effectively offline until this is sorted.

Service update: I think something's up with the AppView ingesting labels right now, I'm noticing posts I've labelled still appearing. I'll investigate that tomorrow but in the meantime apologies if things are working rather poorly.

Apologies, the Jetstream consumer went down again; I've since restarted it. I'm going to try to find some time this weekend to write some code to automate this, your patience is appreciated.

A reminder that this service is completely free and runs 100% on volunteer effort. Leaving mean comments when appealing a label just makes us want to do this unpaid labour even less. Thanks.

Okay we're back. Added a hotfix that should prevent that issue in the future.

The Redis queue is struggling, please bear with me, I'll try to fix things tonight

Service update: my girlfriend is an angel and loaned me a USB WiFi interface so we're back for the moment

Service update: I dropped something on the main inference machine and that burnt out the network card it seems so autolabeller is offline until further notice. Don't expect anything to work sorry.

I've disabled the "newsmedia" label for now because it was causing a lot of mislabels (anything text-heavy was getting caught; I may retrain this in the future, idk). Apologies if it's taken a while to get to your appeal, I'm halfway through the last 2 weeks and it's just me now.

Sorry, just noticed the queue fell over, I've since fixed it and autolabelling has resumed.

Had to fix Redis, auto labelling should be back again. 💚

This is understandable but extremely disheartening for the "vibes" Bluesky had and its continuing health. I felt the most connected with Bluesky when I was burning down the xblock backlog.

This project is on hiatus until further notice.

Also huge gratitude to my hardworking mods, who've had to deal with this bananas deluge of reports all November! It's been real rough, I owe y'all a pizza, a drink, or both! If you've seen an XBlock label between the start of the month and today, it was probably hand-moderated by one of them!

Massive thanks to @devingaffney.com, we've been chatting on and off for the last two weeks about this and between his experiments and Python code I now have a workable solution. I would've abandoned the TIMM version of the model (now in production) without him! I'll be forever grateful. 💚

Oh cool I think I actually fixed @xblock.aendra.dev's infra for realsies this time. Queue number is actually going down for once, with no extra inference beyond my old desktop running on local fibre. Someone want to send me a better graphics card? Really that's all it'd take I think.

Hell yeah, XBlock is back online

Our cloud services are currently failing but we hope to restore some functionality soon, sorry for the mess

Please stop reporting users to XBlock. We only label posts, and only then posts with screenshots from social networks.

I know service levels have been really poor lately; apologies, it's really taking a lot of effort to scale this. I've had someone reach out and we're working together to build out the system better, with any luck once we deploy it'll be able to keep up with demand. Your patience is appreciated! 💚

SERVICE UPDATE: We've retired the old xblock-large v3 node and are now running multiple xblock-m v1 nodes in a much better distributed capacity. Hopefully that helps! We've also temporarily limited automatic scanning to English-language posts only. We hope to improve this eventually. 🤞🏼

SERVICE UPDATE: XBlock Model-M has finally been deployed 🎉

Somehow an old config got re-uploaded and NGL and Discord labels weren't being shown. This should be fixed now!

SERVICE UPDATE: We're running into issues with our Ozone instance so reports might take awhile to get actioned. We appreciate your patience as we gradually roll out our new system. 🙏🏼

Holy heck, welcome new users! 👋🏼 Understandably there's been a lot of Twitter screenshots the last 24h, I'm going to try to roll out XBlock-M this evening to try to catch more of them. Please bear with me! 💚

Quick update: Restoring the old model overnight, I'm close to having this up and rolling but probably best not to try deploying stuff to AWS at 3 in the morning.

Atualização de serviço: A rotulagem automática ficará inativa pelas próximas 24 horas, à medida que migramos para uma nova infraestrutura. Por favor, tenha paciência conosco enquanto isso — os relatórios manuais ainda funcionarão se você precisar relatar algo particularmente flagrante!

Service update: Automatic labelling will be down for the next 24 hours as we migrate to new infrastructure. Please bear with us in the meantime — manual reports will still work if you need to report something particularly egregious!

Hello fellow news avoiders! 👋🏼 I'm going to trial automated news website screenshot labelling once I release XBlock-M later this week. Go to our account and check that setting (toggled to "off" by default) if you want to start seeing the "news media" label once I have the new model up and running ✨

Update: we've restored the existing model for the moment, it'll be a bit more work to convert it to our infrastructure than expected.

SERVICE UPDATE: The XBlock autolabelling functionality is currently offline while we transfer to the brand new XBlock-M model based on swinv2-small-patch4-window16-256. We'll update once it's back online again (and hopefully will fix the issues with us being unable to keep up to the firehose!)

STATUS UPDATE: we've restored service, the AppView is now consuming labels as per usual. Tonight I'm working on getting a new infrastructure rolling, hoping to get XBlock-M v1 online in an hour or two