Utilities¶

Create a new HCS Plate in OME-Zarr format if the plate does not exist.

If the plate exists, append positions and channels if they are not already in the plate.

Parameters:

• store_path (Path) –

  Path to the HCS plate.

• position_keys (list[tuple[str, str, str]]) –

  Position keys (row, column, fov) to append if not present in the plate, e.g., [("A", "1", "0"), ("A", "1", "1")].

• channel_names (list[str]) –

  List of channel names. If the store exists, append if not present in metadata.

• shape (tuple[int, ...]) –

  TCZYX shape of the plate.

• chunks (tuple[int, ...], default: None ) –

  TCZYX chunk size of the plate. If None, a version-specific default is used: - "0.4": (1, 1, Z, Y, X) capped in Z to 500 MB. - "0.5": (1, 1, 16, 256, 256) clamped to shape. Defaults to None.

• shards (tuple[int, ...] or str, default: None ) –

  Shard geometry of the plate. One of:

  • an explicit TCZYX shard shape in array elements, rounded up to whole chunks;
  • a spatial extent, "XY" for one shard per plane or "XYZ" for one shard per ZYX volume;
  • a target file size such as "2GB" or "512MiB", which iohub converts to the largest chunk-aligned shard that fits under it. X grows first, then Y, then Z, then T, so a torn shard costs a spatial region before it costs timepoints, and the channel axis is never grown (channel-sharded stores cannot be written a channel at a time). The target is compared against the nominal (uncompressed) size, so compressed shards land under it.

  If None, a version-specific default is used:

  • "0.4": no sharding (Zarr v2 does not support it).
  • "0.5": "XYZ", one shard per ZYX volume.

  Defaults to None.

• shards_ratio (tuple[int, ...], default: None ) –

  Deprecated TCZYX multiplier over chunks (shards = chunks * shards_ratio). Use shards instead, which does not require knowing the chunk shape; passing both raises ValueError. Defaults to None.

• version (Literal['0.4', '0.5'], default: '0.5' ) –

  OME-Zarr version to use for the plate. Defaults to "0.5".

• scale (tuple[float, ...], default: (1, 1, 1, 1, 1) ) –

  TCZYX scale of the plate. Defaults to (1, 1, 1, 1, 1).

• dtype (DTypeLike, default: float32 ) –

  Data type of the plate. Defaults to np.float32.

• metadata_sources (Path or str or list of Path or str, default: None ) –

  Path(s) to one or more source HCS plates from which to copy per-position metadata. This and metadata_keys together define the copy: metadata_sources is where metadata comes from, metadata_keys is which of it is taken. When set, custom (non-OME) zattrs (e.g. provenance keys such as biahub-flat_field) are transferred from matching positions in the source plate(s). Metadata is only transferred for newly created positions, and a given zattrs key is only copied if it does not already exist on the destination position (so earlier sources take precedence over later ones). Coordinate transforms, axis definitions, and label references are not copied. Defaults to None (no metadata copy).

• metadata_keys (str or iterable of str, default: None ) –

  Case-sensitive shell-glob patterns (fnmatch, not regex) narrowing which zattrs keys metadata_sources may contribute, e.g. {"provenance-*", "acquisition"}. A key is copied only if it matches at least one pattern; OME-owned keys are excluded either way. This only filters the sources, so it is meaningless on its own: passing it without metadata_sources raises ValueError. Defaults to None (copy every non-OME key the sources provide).

• extra_metadata (Mapping[str, Any], default: None ) –

  The caller's own per-position metadata, where metadata_sources is metadata from elsewhere. Each entry is written as a TOP-LEVEL zattrs key on every position in position_keys -- the same spelling process_single_position uses -- so a processing step records its provenance as a sibling of what it inherited, e.g. extra_metadata={"biahub-deskew": settings.model_dump()}.

  Two deliberate differences from metadata_sources:

  • It is written to every position named, not only newly created ones. Re-running with a changed configuration must refresh the record; inherited metadata, by contrast, describes the store as it was made.
  • It wins over an inherited key of the same name, since the caller is the authority on its own step. Sources still take precedence over each other in order.

  Writing the record here rather than after the call is what lets a downstream create_empty_plate inherit it: the source copy reads the upstream store's zattrs at plate-creation time, so a pipeline that scaffolds all of its stores up front would otherwise chain empty provenance. Keys the OME-Zarr spec owns are refused. Defaults to None.

• label_names (list[str], default: None ) –

  Names of empty label images to create in every position in position_keys, at <fov>/labels/<name>/. Each label mirrors the position's image pyramid: one level per image level, with the same path, axes, scale (and translation, if any), and the image's shape with one channel, (T, 1, Z, Y, X): a label is one segmentation, and the DCA array standard wants every array 5D. For a newly created position that is shape with C=1; for an existing position it is read from the image, so the label always matches the image it annotates. Labels that already exist are left untouched, so calling this again is a no-op. No data is written (fill value 0); fill a label later with position.get_label(name)["0"][...] = data. Defaults to None (no labels).

• label_dtype (DTypeLike, default: uint32 ) –

  Integer data type of the label images. Defaults to np.uint32.

• label_chunks (tuple[int, ...], default: None ) –

  TCZYX chunk size of the label images (C must be 1), clamped to the shape of each level. If None, the version-specific default the image uses when chunks is None, computed for each position and level from the image shape at label_dtype (for "0.4" the Z chunk is capped to 500 MB at label_dtype). Explicit image chunks are not inherited. Defaults to None.

• label_shards (tuple[int, ...] or str, default: None ) –

  Shard geometry of the label images, in the forms shards takes (an explicit TCZYX shape in elements, "XY"/"XYZ", or a size target), resolved per level of each position's label. If None, for "0.5" one shard per ZYX volume ("XYZ"); no sharding for "0.4". The image's shards are not inherited. Defaults to None.

• label_shards_ratio (tuple[int, ...], default: None ) –

  Deprecated TCZYX multiplier over label_chunks; use label_shards. Passing both raises ValueError. Defaults to None.

Raises:

• ValueError –

  If metadata_keys is given without metadata_sources.

• ValueError –

  If label_dtype is not an integer type, or label_chunks or label_shards_ratio is not TCZYX (5 values) with C 1, or label_shards / label_shards_ratio is given for version "0.4", or both are given.

• ValueError –

  If label_dtype, label_chunks, label_shards or label_shards_ratio is given without label_names: they describe labels, so on their own they select nothing.

• FileNotFoundError –

  If a metadata_sources plate root does not exist.

Examples:

Create a new plate with positions and channels:

>>> create_empty_plate(
...     store_path=Path("/path/to/store"),
...     position_keys=[("A", "1", "0"), ("A", "1", "1")],
...     channel_names=["DAPI", "FITC"],
...     shape=(1, 1, 256, 256, 256),
... )

Create a plate with custom chunk size and scale:

>>> create_empty_plate(
...     store_path=Path("/path/to/store"),
...     position_keys=[("A", "1", "0")],
...     channel_names=["DAPI"],
...     shape=(1, 1, 256, 256, 256),
...     chunks=(1, 1, 128, 128, 128),
...     scale=(1, 1, 0.5, 0.5, 0.5),
... )

Create a plate whose shards hold as much as fits in ~2 GB files:

>>> create_empty_plate(
...     store_path=Path("/path/to/store"),
...     position_keys=[("A", "1", "0")],
...     channel_names=["DAPI"],
...     shape=(100, 1, 64, 2048, 2048),
...     chunks=(1, 1, 8, 128, 128),
...     scale=(1, 1, 0.5, 0.5, 0.5),
...     shards="2GB",
...     version="0.5",
... )

Create a plate with one shard per ZYX volume:

>>> create_empty_plate(
...     store_path=Path("/path/to/store"),
...     position_keys=[("A", "1", "0")],
...     channel_names=["DAPI"],
...     shape=(1, 1, 64, 2048, 2048),
...     chunks=(1, 1, 8, 128, 128),
...     shards="XYZ",
...     version="0.5",
... )

Create a plate copying metadata from an input plate:

>>> create_empty_plate(
...     store_path=Path("/path/to/output.zarr"),
...     position_keys=[("A", "1", "0")],
...     channel_names=["DAPI"],
...     shape=(1, 1, 256, 256, 256),
...     metadata_sources=Path("/path/to/input.zarr"),
... )

Copy only the provenance keys, leaving a bulky instrument blob behind:

>>> create_empty_plate(
...     store_path=Path("/path/to/output.zarr"),
...     position_keys=[("A", "1", "0")],
...     channel_names=["DAPI"],
...     shape=(1, 1, 256, 256, 256),
...     metadata_sources=Path("/path/to/input.zarr"),
...     metadata_keys={"provenance-*"},
... )

Inherit the input's provenance and record this step's own alongside it:

>>> create_empty_plate(
...     store_path=Path("/path/to/output.zarr"),
...     position_keys=[("A", "1", "0")],
...     channel_names=["DAPI"],
...     shape=(1, 1, 256, 256, 256),
...     metadata_sources=Path("/path/to/input.zarr"),
...     metadata_keys={"provenance-*"},
...     extra_metadata={"provenance-deskew": {"ls_angle_deg": 30}},
... )

Scaffold an empty segmentation label next to the images:

>>> create_empty_plate(
...     store_path=Path("/path/to/output.zarr"),
...     position_keys=[("A", "1", "0")],
...     channel_names=["DAPI"],
...     shape=(1, 1, 256, 256, 256),
...     label_names=["nuclei"],
... )
>>> with open_ome_zarr("/path/to/output.zarr/A/1/0", mode="r+") as position:
...     position.get_label("nuclei")["0"][0] = segmentation  # ZYX

Create empty label images in existing positions of an HCS plate.

The label-image half of create_empty_plate for a plate that already has its positions: the step that segments a plate it reads scaffolds its labels/<name> images without restating the plate's channels, shape or scale, which each label takes from its own position's image. Labels that already exist are left untouched, so calling this again is a no-op.

Parameters:

• store_path (Path) –

  Path to the existing HCS plate.

• position_keys (list[tuple[str, str, str]]) –

  Positions (row, column, fov) to create the labels in. Every one must exist; use create_empty_plate to create positions.

• label_names (list[str]) –

  Names of the label images, at <fov>/labels/<name>/. Each mirrors the position's image pyramid: one level per image level, with the image's axes, scale (and translation, if any) and shape with one channel, (T, 1, Z, Y, X). No data is written (fill value 0).

• label_dtype (DTypeLike, default: uint32 ) –

  Integer data type of the label images. Defaults to np.uint32.

• label_chunks (tuple[int, ...], default: None ) –

  TCZYX chunk size (C must be 1), clamped to the shape of each level. If None, the version-specific image default at label_dtype. Defaults to None.

• label_shards (tuple[int, ...] or str, default: None ) –

  Shard geometry of the label images, in the forms create_empty_plate's shards takes (an explicit TCZYX shape in elements, "XY"/"XYZ", or a size target), resolved per level. If None, one shard per ZYX volume on "0.5"; no sharding on "0.4". Defaults to None.

• label_shards_ratio (tuple[int, ...], default: None ) –

  Deprecated TCZYX multiplier over label_chunks; use label_shards. Not supported for "0.4". Defaults to None.

• label_metadata (Mapping[str, Any], default: None ) –

  The caller's own metadata for the label images, written as top-level attributes of every named label group in every position, existing labels included, with the caller's entries winning over what is there: the step's provenance, e.g. {"biahub-segment": settings.model_dump()}, the same spelling as extra_metadata on a position. Keys the OME-Zarr spec owns on a label are refused. Defaults to None.

Raises:

• FileNotFoundError –

  If the plate or one of the positions does not exist.

• ValueError –

  If label_dtype is not an integer type, label_chunks or label_shards_ratio is not TCZYX with C 1, label_shards_ratio is given for a "0.4" plate, or label_metadata uses a reserved key.

Examples:

>>> create_empty_labels(
...     store_path=Path("/path/to/plate.zarr"),
...     position_keys=[("A", "1", "0")],
...     label_names=["nuclei"],
...     label_chunks=(16, 1, 1, 128, 128),
...     label_metadata={"provenance-segment": {"model": "cpsam_v2"}},
... )
>>> with open_ome_zarr("/path/to/plate.zarr/A/1/0", mode="r+") as position:
...     position.get_label("nuclei")["0"][0] = segmentation  # ZYX

Load a TCZYX array from a position store.

Apply a transformation and save the result.

When the write owns its output files outright (see iohub.ngff._write_units) they are cleared before writing rather than written over, and completion is recorded in a marker file. Pass resume=True to skip a unit that a previous run already finished. With output_label the result goes into that label image of the output position instead of its image (see process_single_position).

Apply function to data in an iohub position store.

Parallelize over time and channel indices and save result in an output position store.

Parameters:

• func (Callable[[NDArray, Any], NDArray]) –

  The function to be applied to the data. func must take the CZYX NDArray as the first argument and return a CZXY NDArray. Additional arguments are passed through **kwargs.

• input_position_path (Path) –

  The path to the input OME-Zarr position store (e.g., input_store_path.zarr/A/1/0).

• output_position_path (Path) –

  The path to the output OME-Zarr position store (e.g., output_store_path.zarr/A/1/0). May be the input position when writing into one of its label images (output_label).

• output_label (str, default: None ) –

  Write into the label image labels/<output_label> of the output position (created beforehand, e.g. by create_empty_plate's label_names) instead of its image. A label image has one channel, so func must return one channel per timepoint and input_channel_indices must hold a single channel group; output_channel_indices is ignored. Resume and repair work as for the image, with the markers kept under the label's name. Defaults to None, the image.

• input_time_indices (list[int], default: None ) –

  If not provided, all timepoints will be processed.

• output_time_indices (list[int], default: None ) –

  The time indices to write to. Must match length of input_time_indices. Typically used for stabilization, which needs per timepoint processing. If not provided, input_time_indices will be used.

• input_channel_indices (Union[list[slice], list[list[int]]], default: None ) –

  The channel indices to process. Acceptable values: - A list of slices: [slice(0, 2), slice(2, 4), ...]. - A list of lists of integers: [[0, 1, 2, 3, 4]]. If empty, process all channels. Must match output_channel_indices if not empty. Defaults to None.

• output_channel_indices (Union[list[slice], list[list[int]]], default: None ) –

  The channel indices to write to. Acceptable values: - A list of slices: [slice(0, 2), slice(2, 4), ...]. - A list of lists of integers: [[0, 1, 2, 3, 4]]. If empty, write to all channels. Must match input_channel_indices if not empty. Defaults to None.

• num_workers (int, default: 1 ) –

  Number of simultaneous workers (processes or threads) per position. If <= 1, the work is performed serially in the calling process. Defaults to 1.

• use_threads (bool, default: False ) –

  If True, parallelize across threads via ThreadPoolExecutor; otherwise spawn worker processes via ProcessPoolExecutor. Defaults to False.

• resume (bool, default: False ) –

  If True, skip shard-aligned (time, channel) units that a previous run already finished, as recorded by the markers described in iohub.ngff._write_units. Intended for retrying a run that was interrupted, for example by Slurm preemption. Leave False when re-running with changed inputs or settings, since a finished unit is skipped without checking whether it would now produce different data. Defaults to False.

• resume_token (str, default: '' ) –

  Fingerprint of whatever determines the output, typically the resolved settings of the calling step. Mixed into each unit's identity so that resume=True after a parameter change recomputes rather than skipping work that would now produce different data. Defaults to "".

• kwargs (dict, default: {} ) –

  Additional arguments to pass to the function. An extra_metadata kwarg (a dict) can be passed to record per-step provenance at the FOV level, typically a single namespaced entry keyed by "<package>-<step>", e.g. extra_metadata={"biahub-deskew": settings.model_dump()}. Each item is written as a separate top-level key on the output position's zattrs (not nested under an extra_metadata key), so successive steps accumulate sibling provenance keys instead of overwriting one shared key. If a key is already present on the output position (e.g. copied from the input by create_empty_plate's metadata_sources) it is overwritten and a warning is raised. Reserved OME-Zarr keys (ome, multiscales, omero, labels, version) are rejected with a ValueError.