Anndata now supports @zarr.dev v3, great for both performance and interoperability. Check out our guide for more info, especially sharding: anndata.readthedocs.io/en/stable/tu.... Don’t worry though, we remain fully v2 backwards compatible - upgrade fearlessly!

♻️ 7/8 - Webinar: Reusable Deep Learning with EOPF Zarr Samples - ESA Copernicus Earth Observation Processor Framework Learn how to reuse a deep learning model originally trained with data from the openEO API of the Copernicus Dataspace Ecosystem (CDSE). #datascience #deeplearning

I've been waiting a few years for this: Zarr v2 support was just launched in Google Earth Engine! Check out this demo that loads ARCO-ERA5 into GEE 🔥 code.earthengine.google.com/67ac1e48848d... developers.google.com/earth-engine...

If you missed today's presentation at the LPS by @thejeran.bsky.social on Browzarr, a browser-based visualization toolkit for exploring and analyzing Zarr data stores, go and check out the version online, and open issues and do feature requests on GitHub. github.com/EarthyScienc...

doodles of two Tree of Life department speakers @sangerinstitute.bsky.social this week, Evgeny Zakharov talking about insanely large scale barcode sequencing and Jerome Kelleher talking about large scale human genomic data analysis. that's a lot of data 🧬📄

We welcome a new project to OpenSustain.tech #climate #sustainability #opensource #opensustain #openscience github.com/zarr-develop...

ICYMI: Check out this alternative storage approach for variation data using #Zarr. Rather than grouping all data for a given variant together, its groups all data for a given field, and store as chunked, compressed N-D arrays (tensors). 🤩 doi.org/10.1093/giga...

OME-Zarr 4 life

"The commitment from major entities like Copernicus, USGS, and Google Earth Engine signals a paradigm shift in how we will store, access, and analyze geospatial data in the cloud... #Zarr is poised to become a cornerstone of the cloud-native geospatial ecosystem, unlocking new frontiers..."

1/ Our team just got back from an inspiring week at the @cloudnativegeo.org conference in Snowbird, Utah, and one thing is clear: 𝐙𝐚𝐫𝐫 𝐚𝐝𝐨𝐩𝐭𝐢𝐨𝐧 𝐢𝐬 𝐬𝐮𝐫𝐠𝐢𝐧𝐠 𝐚𝐜𝐫𝐨𝐬𝐬 𝐠𝐞𝐨𝐬𝐩𝐚𝐭𝐢𝐚𝐥.

Curious about climate patterns? The new ERA Explorer app lets you compare 85 years of climate data worldwide. Analyse climatological averages at any map point or explore month-by-month data.Dive deeper with ERA Explorer! era-explorer.climate.copernicus.eu?lat=50.86&ln... #C3S #beautifuldata

🧠 Interested in extending #Zarr? Let us know what you think about ZEP10 "Generic extensions". Current plan is for voting on the proposal mid-June. 📅 github.com/zarr-develop...

Just back from #CNG2025, I wrote up some thoughts on the traction the @zarr.dev format is gaining in the broader geospatial industry (specifically Earth Observation). I also summarize what features and tools the user and developer communities are asking for to fully leverage Zarr. 👇👇👇

1/ 💡 Our latest blog post in the fundamentals series, written by @tegnicholas.bsky.social, demystifies cloud-optimized scientific data formats! Read more: earthmover.io/blog/fundame...

🚀 Big launch today: introducing Flux — a game-changing way to serve geospatial data via standard APIs in seconds. Built for scale, speed, and simplicity.

This one was a lot of fun to work on! Why @xarray.bsky.social and @zarr.dev are faster than DuckDB and Parquet for gridded data...from first principles.

What's behind ERA Explorer's speed? Meet the ZARR archive! 🚀 ZARR revolutionises data processing, enabling fast, efficient access to over 80 years of climate data. Test out its speed 👉 era-explorer.climate.copernicus.eu?lat=50.86&ln... #C3S @zarr.dev #beautifuldata

Just started using Zarr - literally unbelievably good.

🤩 " #NASA ... leveraged #Icechunk’s virtualization capabilities to present the entire collection of over a #million files as a single analysis-ready #Zarr data cube. "

🚨 New blog post 🚨 In it, we show off our recent work deploying #icechunk on top of #NASA's existing archives of Earth observation data. The results: 100x speed up when extracting time series from existing datasets stored as netCDF.

ICYMI: A MATLAB implementation of @zarr.dev has dropped from @mathworks.bsky.social 🚀 github.com/mathworks/MA...

Really cool new neuroscience benchmark just dropped. Data is in @zarr.dev and demos use tensorstore. zapbench-release.storage.googleapis.com/landing.html

Very happy to announce our winter release of syGlass: 2.2.0! Now control syGlass with your voice. We've also upgraded the auto-segmentation, mask handling & proofreading, and the narration system. We've added basic support for @zarr.dev. Check out the changelog: www.syglass.io/blog/announc...

You, yes you, can help the Cloud Native Geospatial Movement! Have you benefitted from Cloud-Optimized GeoTIFF’s? SpatioTemporal Asset Catalogs? Zarr, COPC or GeoParquet? And the ecosystem around them? Learn how to give back in my post: cholmes.medium.com/you-yes-you-...

We're a bit late to the bsky game so we appreciate a boost! @loicaroyer.bsky.social @haesleinhuepf.bsky.social @florianjug.bsky.social @lhinderling.bsky.social @kbias.bsky.social @zarr.dev @xarray.bsky.social @scipyconf.bsky.social @drlachie.bsky.social @henriqueslab.bsky.social @globias.bsky.social

If you missed the Zarr-Python 3 release party last week, no worries because today, we released zarr-python 3.0.1 and numcodecs 0.15.0. release notes 👉 zarr.readthedocs.io/en/stable/re...

Microscopy Nodes loads data from a @zarr.dev bucket, making it easy to use open data archives that host Zarr data! This is a mitotic cell from Walther et al. 2018, that's hosted publicly on the Image Data Resource 😱

😍 That's what friends are for!! #opensource #scientific #python #ftw 💪

🎉 Zarr-Python 3 is here! 🎉 - Full support for Zarr v3 spec - Chunk-sharding for more efficient data storage - Major performance boosts with async I/O & parallel compression 💻 pip install --upgrade zarr 💻 conda install --channel conda-forge zarr Blog post: https://buff.ly/3C3OwYw

Hi, Zarr community! 👋 🚨 Big news: Zarr-Python 3.0 is dropping next week! 🚨 If your project depends on Zarr-Python, make sure to check out the migration guide to get ready for the update. Guide: https://buff.ly/3W0cLxd More updates soon! 🧑🏻‍💻

And if you really want a challenge, this is probably the biggest singular Zarr dataset in existence (~6PiB): github.com/google-resea...

For the @napari.fosstodon.org.ap.brid.gy and large image data out there, I made some updates to the tiled loading zarr demo. It still has limitations (scale isn't respected, etc.), but this is an easy way to run it on your own code. github.com/napari/napar... More in 🧵👇

More of this from all the companies that depend on the same tools the scientists use/develop please

New batch read support has significantly improved efficiency when reading from sharded formats like zarr v3 and neuroglancer precomputed: github.com/search?q=rep... and many other general performance and efficiency improvements (e.g. github.com/google/tenso...)

Looking for a job where you get to play with #Zarr 🧊🧊 🧊🧊❔ 👇

In the talk, I made a few simple points: - Separation of storage and compute is key to unlocking the scaling potential of cloud - Cloud optimized data formats are key (example: @zarr.dev and #icechunk) - API services should be stateless/serverless and should be able to scale horizontally [0->N]

Hi Zarrites! 👋🏻 We have our bi-weekly community meeting today. 🗓️ These meetings are a great place to share updates, showcase your projects, raise an issue, or converse with the vibrant community. 🧑🏻‍💻 
Join here: https://buff.ly/3SHCzNN

Please help the #Zarr community welcome @normanrz.bsky.social as the newest member of the Steering Council! 👏🏼 zarr.dev/blog/steerin...

I'm at #AGU2024 this week! If you want to talk about anything @xarray.bsky.social , Pangeo, @zarr.dev , or @cworthy.bsky.social , message me! I have a poster on the VirtualiZarr package I wrote on Wednesday morning agu.confex.com/agu/agu24/me... but also several other projects I've worked on...

I have a few ideas: - create kurchunk indexes on the corpus via @tomnicholas.bsky.social’s VirtualiZarr github.com/zarr-develop... - Use @openclimatefix.org’s hypergrib github.com/JackKelly/hy... 1/

🌍 IN21D: Scaling Data Discovery w Dynamic Spatial Aggregations Using STAC Metadata (@emmapaz.com) 🌍 GC23A: Choosing Earth Foundation Models: From Research Grade to Community Maintained (Wei Ji Leong) 🌍 IN31C: VirtualiZarr: Create Virtual Zarr Stores Using Xarray Syntax (Aimee, Max & Sean) #AGU24

And there's of course github.com/grimbough/Rarr if you're interested, @jerry-shannon.bsky.social. Nothing like #Zarr in #R (to make a mouthful)

Also nice coverage in the @nytimes.com, although not sure I'd call GenCast an "AI Agent" - www.nytimes.com/2024/12/04/s... ... and I guess some e-mails are warranted to clarify, but I hope to see forecasts published elsewhere than Earth Engine and BigQuery. Standard Zarr files on GCS, please?

Also on Thursday afternoon, I'll be giving an invited talk titled "Seamless Arrays: A Full Stack, Cloud-Native Architecture for Fast, Scalable Data Access". It combines all that we've been working on for the past year including @zarr.dev v3, #icechunk, and Xpublish. agu.confex.com/agu/agu24/me...

We have been experimenting with NVIDIA DALI for deep learning training and inference work with Pytorch and zarr-enabled data loaders. It really has optimized our training and inference pipeline. github.com/NVIDIA/DALI

Thanks to gitub.com/ldeakin the list of #zarr implementations is now shiny and new! ✨ zarr.dev/implementati... 👈 If any are missing or need updating, please let us know.

#beautifuldata in #zarr 😍

And with OME-Zarr loading, due to the pyramidal format, you can design your video at small-scale and only replace to full-scale at the final render! (data from Granita Lokaj, @ s3.embl.de/microscopyno...)

Build your own online integrated platform for single-cell & spatial transcriptomic data using WebAtlas pipeline Cell2location StabMap Unify image & gene expression files into Zarr format Compatible with SpatialData Based on Vitessce framework #Naturemethods 2024 www.nature.com/articles/s41...