Chunkwise image loader

[lucas-diedrich]

Description

This PR addresses the challenge that the currently implemented and planned image loaders require loading imaging data entirely into memory, typically as NumPy arrays. Given the large size of microscopy datasets, this is not always feasible.

To mitigate this issue, and as discussed with @LucaMarconato, this PR aims to introduce a generalizable approach for reading large microscopy files in chunks, enabling efficient handling of data that does not fit into memory.

Some related discussions.

• Image Functions spatialdata#704 (comment)
• Efficient whole slide imaging IO spatialdata#856 (comment)
• DVP image readers lucas-diedrich/spatialdata-io#1 (comment)

Strategy

In this PR, we focus on .tiff images, as implemented in the _tiff_to_chunks function.

1. Get a lazy representation of the image via a suitable reader function (here: tifffile.memmap)
2. Pre-define chunks that fit into memory, based on the dimensions of the image (_compute_chunks)
3. Load small chunks via a custom reader function and pass the chunks to dask.array which is memory-mapped and avoids memory overflow (_read_chunks)
4. Reassembling the chunks into a dask.array (via dask.array.block)
5. Parse to Image2DModel.

The strategy is implemented in

• src/spatialdata_io/readers/generic.py and
• src/spatialdata_io/readers/_utils/_image.py

Future extensions

The strategy can be implemented for any image type, as long as it is possible to implement

1. a lazy image-data loader
2. define a custom reader function

We have implemented similar readers for openslide-compatible whole slide images and the Carl-Zeiss microscopy format.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 50.41%. Comparing base (296d9a5) to head (c05b718).
Report is 174 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main     #279       +/-   ##
===========================================
+ Coverage   39.16%   50.41%   +11.25%     
===========================================
  Files          26       27        +1     
  Lines        2663     2759       +96     
===========================================
+ Hits         1043     1391      +348     
+ Misses       1620     1368      -252     

Files with missing lines	Coverage Δ
src/spatialdata_io/readers/_utils/_image.py	100.00% <100.00%> (ø)
src/spatialdata_io/readers/generic.py	91.17% <100.00%> (+2.60%)	⬆️

... and 12 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[melonora]

Thanks for your contribution! I have 2 minor suggestions. I also saw that you use the width by height convention. Personally, I don't have a strong opinion here, though we could also stick to array api conventions. @LucaMarconato WDYT? Pre-approving for now.

[melonora]

sorry had to change due to rethinking memmap. This does not always work, for example when dealing with compressed tiffs as far as I am aware.

[lucas-diedrich]

Thanks! Addressed your comments

• Naming, simplification 46ed3e5, 99731fe
• Read compressed images with dask_image.imread instead of tifffile 9e057de