RSVP and join speakers @very-laurie.bsky.social and @pjox.bsky.social from the Common Crawl Foundation and Kostis Saitas Zarkias and Robert Pugh from Mozilla Data Collective for a truly hands-on session. Thursday, June 4th 6 PM CEST | 12 PM ET | 9 AM PDT Register via Zoom: zoom.us/meeting/regi...

Common Crawl is switching to reporting dataset sizes in nibbles. As an organisation dedicated to data preservation, we feel it would be remiss to allow this underrepresented unit to fall out of use. Our latest crawl now exceeds 689 tebibbles. commoncrawl.org/blog/announc...

Language identification still proves to be a challenging task, especially for web data. In collaboration with @mlcommons.org @eleutherai.bsky.social @jhu.edu and 97 community members, we created CommonLID, a new benchmark for LangID for 100+ languages!

Announcing our latest paper: CommonLID In collaboration with @commoncrawl.bsky.social @mlcommons.org @jhu.edu we built a LID benchmark on actual Common Crawl text covering 109 languages. Existing evaluations overestimate how well LangID works on web data. arxiv.org/abs/2601.18026

Common Crawl’s Web Languages initiative has had many contributions since its introduction. We’re calling for native speakers of certain languages to review language contributions, to ensure that links we’re adding to our seed crawl are of good quality. commoncrawl.org/blog/web-lan...

"MOIC will also partner with Common Crawl, one of the largest free and open repositories of web crawled data. MOIC will fund work at Common Crawl, leveraging native speakers to annotate and seed European language data in the publicly available Common Crawl data set."

The Common Crawl Foundation, MLCommons, EleutherAI, and John Hopkins' Center for Language and Speech Processing have the pleasure of inviting you to register for the 1st shared task on Language Identification for web data. commoncrawl.org/blog/wmdqs-s...

Apache Nutch 1.21 is now available for download: buff.ly/juTZlwE Nutch is a well matured, production ready #web crawler. #opensource

[NEWS] The Apache Software Foundation Announces Two New Top-Level Projects buff.ly/09CyVxC

🚨🚀 Looking for a comparative dataset on social media platforms? We’re excited to launch COMPARE! This is a collaborative effort by @nilsweidmann.bsky.social , @friederikeq.bsky.social , @sebnagel.bsky.social , @yannistheocharis.bsky.social & Molly Roberts. 🧵⤵️ (1/5)

Call for papers! We are organising the 1st Workshop on Multilingual Data Quality Signals with @mlcommons.org and @eleutherai.bsky.social, held in tandem with @colmweb.org. Submit your research on multilingual data quality! Submission deadline is 23 June, more info: wmdqs.org

The #UniKonstanz Cluster of Excellence "The Politics of Inequality @excinequality.bsky.social will continue to receive funding in the context of the German #ExcellenceStrategy @dfg.de @wissenschaftsrat.de. Details: t1p.de/ouhyj