sc_tools.ingest — Data Ingestion#

Phase 0 data loading: batch manifests, platform command builders, modality-specific AnnData loaders, and sample concatenation.

import sc_tools.ingest as ingest

# Load batch manifest
manifest = ingest.load_batch_manifest("metadata/phase0/batch1_samples.tsv")

# Load a single IMC sample
adata = ingest.load_imc_sample("processed/sample_01", sample_id="sample_01")

# Concatenate all samples
adata_all = ingest.concat_samples([adata1, adata2, adata3])

Batch Manifests#

sc_tools.ingest.config.load_batch_manifest(path)[source]#

Load a single batch TSV manifest from a local path or remote URI.

Parameters:

path (str | PathLike) – Path or URI to a tab-separated manifest file.

Return type:

DataFrame with manifest rows.

sc_tools.ingest.config.collect_all_batches(phase0_dir, output=None)[source]#

Glob metadata/phase0/*_samples.tsv, concatenate, write all_samples.tsv.

Parameters:

  • phase0_dir (str | PathLike) – Directory containing batch TSV files (e.g., metadata/phase0/).

  • output (str | PathLike | None) – Path to write the collected manifest. Defaults to phase0_dir/all_samples.tsv.

Return type:

Concatenated DataFrame of all batch manifests.

sc_tools.ingest.config.validate_manifest(df, modality)[source]#

Check that required columns exist for the modality.

Parameters:

  • df (DataFrame) – Manifest DataFrame.

  • modality (str) – One of: visium, visium_hd, xenium, imc, cosmx.

Return type:

List of validation issue messages. Empty means valid.

Modality Loaders#

Each loader sets obs['sample'], obs['library_id'], obs['raw_data_dir'], obsm['spatial'], and ensures X contains raw counts.

sc_tools.ingest.loaders.load_visium_sample(spaceranger_dir, sample_id, *, load_images=True)[source]#

Load one Visium sample from Space Ranger output.

Sets obs[‘sample’], obs[‘library_id’], obs[‘raw_data_dir’], and obsm[‘spatial’].

Parameters:

  • spaceranger_dir (str | Path) – Path to Space Ranger output directory (containing outs/).

  • sample_id (str) – Sample identifier to store in obs[‘sample’].

  • load_images (bool) – Whether to load H&E images into the AnnData object.

Return type:

AnnData with spatial coordinates and sample metadata.

sc_tools.ingest.loaders.load_visium_hd_sample(spaceranger_dir, sample_id, *, bin_size='square_008um', load_images=False)[source]#

Load one Visium HD sample from binned Space Ranger output.

Reads tissue positions from parquet and sets spatial coordinates.

Parameters:

  • spaceranger_dir (str | Path) – Path to Space Ranger output directory containing binned outputs.

  • sample_id (str) – Sample identifier.

  • bin_size (str) – Bin size subdirectory name (default: square_008um).

  • load_images (bool) – Whether to load images.

Return type:

AnnData with spatial coordinates and sample metadata.

sc_tools.ingest.loaders.load_visium_hd_cell_sample(spaceranger_dir, sample_id, *, load_images=False)[source]#

Load one Visium HD sample from SpaceRanger 4 cell segmentation output.

Reads the cell-level (not bin-level) data produced by SpaceRanger 4. Searches for the segmentation output under both outs/segmented_outputs/ (SR4 default) and outs/cell_segmentation/ (legacy). The expression matrix is filtered_feature_cell_matrix.h5 (SR4) or filtered_feature_bc_matrix.h5 (legacy). Cell spatial coordinates are extracted from cell_segmentations.geojson polygon centroids.

Parameters:

  • spaceranger_dir (str | Path) – Path to Space Ranger output directory (containing outs/).

  • sample_id (str) – Sample identifier to store in obs[‘sample’].

  • load_images (bool) – Whether to load images.

Return type:

AnnData with cell-level spatial coordinates and sample metadata.

sc_tools.ingest.loaders.load_xenium_sample(xenium_dir, sample_id)[source]#

Load one Xenium sample from output directory.

Attempts spatialdata-io first, falls back to scanpy read_10x_h5.

Parameters:

  • xenium_dir (str | Path) – Path to Xenium output directory.

  • sample_id (str) – Sample identifier.

Return type:

AnnData with spatial coordinates.

sc_tools.ingest.loaders.load_imc_sample(processed_dir, sample_id, *, load_images=False, panel_csv=None, rgb_channels=('PanCK', 'CD3', 'DNA1'), image_downsample=1)[source]#

Load one IMC sample from a steinbock/ElementoLab processed directory.

Expects the standard ElementoLab IMC pipeline output layout:

processed/{sample}/
    tiffs/
        {roi_id}_full.tiff     # (C, H, W) multi-channel TIFF stack
        {roi_id}_full.csv      # channel index -> MarkerName(IsotopeTag)
        {roi_id}_full_mask.tiff
        {roi_id}_Probabilities.tiff
    cells.h5ad                 # single-cell data (steinbock default)
Parameters:

  • processed_dir (str | Path) – Path to the processed sample directory (e.g. processed/{sample}/). Must contain cells.h5ad (steinbock default) or cells/cells.h5ad.

  • sample_id (str) – Sample/ROI identifier (e.g. 05122023_Vivek_S21_5251_G3_Group1-01). Used to locate tiffs/{sample_id}_full.tiff when load_images=True.

  • load_images (bool) – If True, read the {sample_id}_full.tiff multi-channel stack from processed_dir/tiffs/, build an arcsinh-normalized full stack and RGB composite, and store both in adata.uns['spatial'][sample_id] (squidpy/scanpy-compatible format).

  • panel_csv (str | Path | None) – Optional path to a panel metadata CSV (channel_labels.csv) with columns channel, Target, Metal_Tag, Atom, full, ilastik. Provides additional name aliases and flags. When None, channels are resolved from the per-ROI *_full.csv alone.

  • rgb_channels (tuple[str, str, str]) – Three marker names (R, G, B) for the default RGB composite image. Defaults to ("PanCK", "CD3", "DNA1").

  • image_downsample (int) – Spatial downsampling factor applied uniformly (1 = no downsampling). Use 2 or 4 to reduce memory for large IMC images.

Return type:

AnnData

Returns:

  • AnnData with sample annotation and spatial coordinates. When

  • load_images=True, adata.uns['spatial'][sample_id] contains:: –

    images/

    hires (H, W, 3) uint8 — percentile-clipped RGB composite full (C, H, W) float32 — arcsinh(x/5) normalized full stack

    scalefactors/

    tissue_hires_scalef: 1.0/downsample spot_diameter_fullres: 1.0 (1 pixel ~ 1 µm in IMC)

    metadata/

    channels: list[str] — ordered protein names channel_strings: list[str] — full MarkerName(IsotopeTag) strings rgb_channels: {R, G, B} — resolved protein names used rgb_indices: {R, G, B} — TIFF stack indices used pixel_size_um: 1.0

sc_tools.ingest.loaders.load_he_image(he_path, library_id, adata, *, downsample=1, image_key='hires')[source]#

Load an H&E TIFF and inject it into adata.uns['spatial'][library_id].

Works for any modality (IMC, Xenium, Visium HD) that has an accompanying H&E TIFF image. Creates the spatial dict if absent; overwrites images[image_key] if the key already exists.

Parameters:

  • he_path (str | Path) – Path to the H&E TIFF file (any format readable by tifffile).

  • library_id (str) – Key under adata.uns['spatial'] where the image will be stored.

  • adata (AnnData) – AnnData to modify in place.

  • downsample (int) – Spatial downsampling factor applied uniformly (1 = no downsampling).

  • image_key (str) – Key within adata.uns['spatial'][library_id]['images'] under which the image is stored (default "hires").

Return type:

None

sc_tools.ingest.loaders.concat_samples(adatas, *, sample_col='sample', calculate_qc=True)[source]#

Concatenate multiple sample AnnDatas with proper handling.

Parameters:

  • adatas (list[AnnData]) – List of AnnData objects to concatenate.

  • sample_col (str) – Column in obs used as sample identifier.

  • calculate_qc (bool) – If True, run scanpy calculate_qc_metrics after concatenation.

Return type:

Concatenated AnnData with QC metrics.

SLURM / HPC Helpers#

sc_tools.ingest.slurm.build_sbatch_header(job_name, log_dir='logs', *, partition='scu-cpu', cpus_per_task=32, mem='240G', time='2-00:00:00', nodes=1, ntasks=1, extra_directives=None)[source]#

Generate an #SBATCH directive block.

Parameters:

  • job_name (str) – SLURM job name.

  • log_dir (str) – Directory for stdout/stderr logs (relative to submission dir).

  • partition (str) – SLURM partition name.

  • cpus_per_task (int) – Number of CPU cores per task.

  • mem (str) – Memory allocation (e.g. “240G”).

  • time (str) – Wall time limit (e.g. “2-00:00:00”).

  • nodes (int) – Number of nodes.

  • ntasks (int) – Number of tasks.

  • extra_directives (dict[str, str] | None) – Additional #SBATCH key-value pairs (e.g. {"account": "mylab"}).

Return type:

Multi-line string of #SBATCH directives.

sc_tools.ingest.slurm.write_sbatch_script(script_text, output_path, *, overwrite=False)[source]#

Write an sbatch script to disk and make it executable.

Parameters:

  • script_text (str) – The sbatch script content.

  • output_path (str | Path) – File path to write to.

  • overwrite (bool) – If False, raise FileExistsError when the file already exists.

Return type:

Path to the written script.

Raises:

FileExistsError – If the file exists and overwrite is False.