sc_tools.bm — Benchmarking#

Integration and segmentation benchmarking tools.

Integration Benchmark#

Compare batch-correction methods and select the best by batch score.

from sc_tools.bm import run_full_integration_workflow

adata, comparison_df, best_method = run_full_integration_workflow(
    adata,
    modality="visium",
    batch_key="library_id",
    methods=["harmony", "combat", "scvi"],
    output_dir="results",
)
sc_tools.bm.integration.run_full_integration_workflow(adata, modality='visium', batch_key='library_id', celltype_key=None, methods=None, output_dir='results', *, subsample_n=None, subsample_fraction=None, use_gpu='auto', max_epochs=200, use_scib='auto', save_intermediates=True)[source]#

Run the full integration benchmark workflow.

Orchestrates: subsample -> benchmark all methods -> save intermediates -> select best (by batch score) -> apply to full dataset.

Parameters:

  • adata (AnnData) – AnnData with raw or normalized data. Modified in place with the winning integration embedding.

  • modality (str) – Data modality (determines default method set).

  • batch_key (str) – Batch column in obs.

  • celltype_key (str | None) – Cell type column (optional; improves scoring but not required).

  • methods (list[str] | None) – Integration methods to benchmark. If None, uses modality defaults.

  • output_dir (str | Path) – Directory for intermediate outputs.

  • subsample_n (int | None) – Number of cells to subsample for benchmarking. Default: auto (all if <50k cells, else 50k stratified by batch).

  • subsample_fraction (float | None) – Fraction of cells to subsample (overrides subsample_n).

  • use_gpu (str | bool) – GPU setting for VAE methods.

  • max_epochs (int) – Max training epochs for VAE methods.

  • use_scib (str) – Metric computation backend.

  • save_intermediates (bool) – If True, save per-method embeddings to output_dir/tmp/integration_test/{method}.h5ad.

Returns:

The AnnData with the best integration applied, the comparison DataFrame, and the name of the selected method.

Return type:

tuple[AnnData, pd.DataFrame, str]

sc_tools.bm.integration.run_integration_benchmark(adata, modality='visium', batch_key='library_id', celltype_key=None, methods=None, use_gpu='auto', max_epochs=200, use_scib='auto')[source]#

Run multiple integration methods and benchmark them.

Orchestrates integration across methods appropriate for the given modality, computes quality metrics for each, and returns the AnnData with all embeddings plus a comparison DataFrame.

Parameters:

  • adata (AnnData) – AnnData with raw counts (for VAE methods) or normalized data. Modified in place with embeddings added to obsm.

  • modality (str) – Data modality (determines default method set and normalization).

  • batch_key (str) – Column in adata.obs for batch correction.

  • celltype_key (str | None) – Column in adata.obs with cell type labels. If provided and scib-metrics is available, the Benchmarker class is used.

  • methods (list[str] | None) – List of method names to run. If None, uses modality defaults. Valid names: harmony, bbknn, combat, scanorama, scvi, scanvi, cytovi, pca.

  • use_gpu (str | bool) – GPU setting for VAE methods.

  • max_epochs (int) – Maximum epochs for VAE methods.

  • use_scib (str) – Passed to metric computation.

Returns:

The AnnData with all embeddings, and the comparison DataFrame sorted by overall_score.

Return type:

tuple[AnnData, pd.DataFrame]

sc_tools.bm.integration.compute_integration_metrics(adata, embedding_key, batch_key, celltype_key=None, use_scib='auto', resolution=1.0, random_state=0)[source]#

Compute integration quality metrics for one embedding.

Parameters:

  • adata (AnnData) – AnnData object with embeddings in obsm.

  • embedding_key (str) – Key in adata.obsm (e.g. "X_scVI").

  • batch_key (str) – Column in adata.obs with batch labels.

  • celltype_key (str | None) – Column in adata.obs with cell type labels. If None or not present in adata.obs, bio conservation metrics are skipped and only batch removal metrics are returned.

  • use_scib (str) – "auto" (default): use scib-metrics if available, else sklearn. "scib": require scib-metrics (error if missing). "sklearn": force sklearn fallbacks.

  • resolution (float)

  • random_state (int)

Return type:

dict[str, float]

Returns:

  • Dict with batch removal and bio conservation metric values, all in [0, 1].

  • When celltype_key is None, bio metrics are omitted.

sc_tools.bm.integration.compute_composite_score(metrics, batch_weight=0.2, bio_weight=0.8)[source]#

Compute composite integration score from individual metrics.

Parameters:

  • metrics (dict[str, float]) – Output from compute_integration_metrics.

  • batch_weight (float) – Weight for batch removal score (default 0.2).

  • bio_weight (float) – Weight for bio conservation score (default 0.8).

Return type:

Dict with batch_score, bio_score, overall_score.

sc_tools.bm.integration.compare_integrations(adata=None, embeddings=None, batch_key='batch', celltype_key=None, bio_key=None, batch_weight=0.2, bio_weight=0.8, include_unintegrated=True, use_scib='auto', subsample_n=None, seed=42, resolution=1.0, random_state=0, embedding_files=None)[source]#

Compare multiple integration methods side-by-side.

Parameters:

  • adata (AnnData | None) – AnnData with multiple embeddings in obsm. Can be None when embedding_files provides all methods.

  • embeddings (dict[str, str] | None) – Dict mapping method name to obsm key (e.g. {"scVI": "X_scVI", "Harmony": "X_pca_harmony"}).

  • batch_key (str) – Column in obs with batch labels.

  • celltype_key (str | None) – Column in obs with cell type labels. If None or not present in adata.obs, bio conservation metrics are skipped.

  • bio_key (str | None) – Column to use for bio conservation metrics. Defaults to celltype_key when not provided. Allows using any clinically relevant variable (e.g. "condition", "disease_status").

  • batch_weight (float) – Weight for batch removal in composite score.

  • bio_weight (float) – Weight for bio conservation in composite score.

  • include_unintegrated (bool) – If True and "X_pca" exists, add unintegrated PCA as baseline.

  • use_scib (str) – Passed to compute_integration_metrics.

  • subsample_n (int | None) – If set, subsample to this many cells (stratified by batch_key) before computing metrics.

  • seed (int) – Random seed for subsampling reproducibility.

  • resolution (float) – Leiden clustering resolution for ARI/NMI bio conservation metrics.

  • embedding_files (dict[str, str] | None) – Dict mapping method name to h5ad file path. Embeddings are loaded via h5py without reading the full AnnData, enabling benchmarking of datasets too large to fit in memory.

  • random_state (int)

Return type:

DataFrame with rows=methods, sorted by overall_score descending.

Segmentation Benchmark#

Compare cell segmentation methods with panoptic quality, boundary F1, and cell-type preservation metrics.

sc_tools.bm.segmentation.compute_segmentation_accuracy(pred, gt, iou_thresholds=(0.5, 0.75))[source]#

Compute segmentation accuracy metrics.

Return type:

dict[str, float]

Returns:

  • Dict with keys (mean_iou, mean_dice, ap_50, ap_75,)

  • ``ap_50_95`` (COCO-style AP averaged over [0.5 (0.05:0.95]).)

Parameters:
sc_tools.bm.segmentation.score_segmentation(mask, intensity_image=None, gt_mask=None, marker_names=None)[source]#

Run all applicable metrics and return raw results.

Parameters:

  • mask (ndarray) – Labeled segmentation mask.

  • intensity_image (ndarray | None) – Optional intensity image for marker quality metrics.

  • gt_mask (ndarray | None) – Optional ground truth mask for detection/accuracy metrics.

  • marker_names (list[str] | None) – Optional marker names for intensity channels.

Return type:

dict[str, Any]

Returns:

  • Dict with keys morphology, spatial_coherence,

  • size_distribution, and optionally marker_quality,

  • detection, accuracy.

sc_tools.bm.segment.run_cellpose(image, nuclear_channels=None, membrane_channels=None, nuclear_idx=1, cytoplasm_idx=2, model_type='cyto2', diameter=None, flow_threshold=0.4, cellprob_threshold=0.0, gpu=False)[source]#

Run Cellpose segmentation.

Parameters:

  • image (ndarray) – Either a probability map (H, W, C) from Ilastik (e.g. 3 channels: background, nucleus, cytoplasm), or a multi-channel intensity TIFF (C, H, W). Detected automatically from shape.

  • nuclear_channels (list[int] | None) – For (C, H, W) input: indices of nuclear channels. Ignored for (H, W, C) probability maps.

  • membrane_channels (list[int] | None) – For (C, H, W) input: indices of membrane channels. Ignored for (H, W, C) probability maps.

  • nuclear_idx (int) – For (H, W, C) probability maps: index of the nuclear channel (default 1).

  • cytoplasm_idx (int) – For (H, W, C) probability maps: index of the cytoplasm channel (default 2).

  • model_type (str) – Cellpose model type (default "cyto2").

  • diameter (float | None) – Expected cell diameter in pixels. None = auto-estimate.

  • flow_threshold (float) – Flow error threshold for Cellpose.

  • cellprob_threshold (float) – Cell probability threshold for Cellpose.

  • gpu (bool) – Whether to use GPU.

Return type:

Labeled segmentation mask, shape (H, W), dtype uint32.

sc_tools.bm.segment.run_stardist(image, nuclear_channels=None, nuclear_idx=1, model_name='2D_versatile_fluo', prob_thresh=None, nms_thresh=None, scale=None)[source]#

Run StarDist segmentation.

Parameters:

  • image (ndarray) – Either a probability map (H, W, C) from Ilastik (nuclear channel at nuclear_idx), or a multi-channel intensity TIFF (C, H, W) (nuclear channels at nuclear_channels), or a 2D nuclear image (H, W).

  • nuclear_channels (list[int] | None) – For (C, H, W) input: indices of nuclear channels.

  • nuclear_idx (int) – For (H, W, C) probability maps: index of the nuclear channel (default 1).

  • model_name (str) – StarDist pretrained model name (default "2D_versatile_fluo").

  • prob_thresh (float | None) – Probability threshold. None = model default.

  • nms_thresh (float | None) – Non-maximum suppression threshold. None = model default.

  • scale (float | None) – Scale factor for the image. None = no rescaling.

Return type:

Labeled segmentation mask, shape (H, W), dtype uint32.

sc_tools.bm.segment.run_deepcell(image, nuclear_channels=None, membrane_channels=None, nuclear_idx=1, cytoplasm_idx=2, compartment='whole-cell', image_mpp=1.0, postprocess_kwargs=None)[source]#

Run DeepCell Mesmer segmentation.

Thin re-export of sc_tools.bm.deepcell_runner.run_deepcell. See that module for full documentation.

Return type:

Labeled segmentation mask, shape (H, W), dtype uint32.

Parameters:

  • image (ndarray)

  • nuclear_channels (list[int] | None)

  • membrane_channels (list[int] | None)

  • nuclear_idx (int)

  • cytoplasm_idx (int)

  • compartment (str)

  • image_mpp (float)

  • postprocess_kwargs (dict | None)

Mask I/O#

sc_tools.bm.mask_io.load_mask(path, format='auto')[source]#

Load a segmentation mask, auto-detecting format from extension.

Parameters:

  • path (str | Path) – Path to the mask file.

  • format (str) – One of "auto", "cellpose", "stardist", "cellprofiler", "deepcell", "tiff". "auto" detects from file extension.

Return type:

Labeled integer array (0=background, >0=cell_id).

sc_tools.bm.mask_io.load_tiff_mask(path)[source]#

Load a labeled TIFF mask (generic loader).

Return type:

ndarray

Parameters:

path (str | Path)

sc_tools.bm.mask_io.load_cellpose_mask(path)[source]#

Load a Cellpose segmentation mask from *_seg.npy.

Cellpose saves a dict with key "masks" containing the labeled array. Also handles plain labeled arrays saved directly.

Return type:

ndarray

Parameters:

path (str | Path)

sc_tools.bm.mask_io.load_stardist_mask(path)[source]#

Load a StarDist label image from .tif/.tiff or .npz.

For .npz files, expects a "labels" key.

Return type:

ndarray

Parameters:

path (str | Path)

sc_tools.bm.mask_io.load_deepcell_mask(path)[source]#

Load a DeepCell/Mesmer output mask from TIFF or NPZ.

DeepCell output is typically (1, H, W, 1); squeeze extra dims.

Return type:

ndarray

Parameters:

path (str | Path)

sc_tools.bm.mask_io.load_cellprofiler_mask(path)[source]#

Load a CellProfiler label image from TIFF.

Return type:

ndarray

Parameters:

path (str | Path)