sc_tools.storage — Storage Abstraction#

fsspec-based URI resolution for reading and writing data across local filesystem, SFTP/HPC, S3, GCS, Azure, and Box.

from sc_tools.storage import smart_read_h5ad, smart_write_checkpoint

# Read from HPC scratch via SFTP
adata = smart_read_h5ad("sftp://brb//athena/.../adata.raw.h5ad")

# Write to S3
smart_write_checkpoint(adata, "s3://bucket/results/adata.normalized.h5ad")

Install remote backends: pip install sc-tools[storage]

sc_tools.storage.resolve_fs(uri)[source]#

Return (AbstractFileSystem, path) for any URI or local path.

Parameters:

uri (str | PathLike) – File URI or local path. Examples: "/local/path", "s3://bucket/key", "sftp://brb//athena/.../file.h5ad".

Returns:

Filesystem object and the path within that filesystem.

Return type:

tuple[AbstractFileSystem, str]

Raises:

ImportError – When the backend required for the URI scheme is not installed.

sc_tools.storage.open_file(uri, mode='rb')[source]#

Open any URI for reading or writing.

Parameters:

  • uri (str | PathLike) – File URI or local path.

  • mode (str) – Open mode ("rb", "wb", "r", "w").

Yields:

IO – File-like object opened at the given URI.

Return type:

Generator[IO, None, None]

sc_tools.storage.with_local_copy(uri)[source]#

Yield a local Path, downloading from remote when necessary.

For local URIs this is a no-op (returns the original path unchanged). For remote URIs the file is downloaded to a temporary file first.

Useful for libraries (scanpy, tifffile) that need a real filesystem path rather than a file-like object.

Parameters:

uri (str | PathLike) – File URI or local path.

Yields:

Path – Local path to the file.

Return type:

Generator[Path, None, None]

sc_tools.storage.smart_read_h5ad(uri, *, backed=None)[source]#

Read an AnnData .h5ad file from any URI.

For local paths calls anndata.read_h5ad directly (backed mode supported). For remote URIs the file is downloaded first (backed mode not supported remotely).

Parameters:

  • uri (str | PathLike) – Local path or URI to an .h5ad file.

  • backed (str | None) – Backing mode for local files: "r" (read-only) or "r+" (read-write). Ignored for remote URIs.

Return type:

AnnData

sc_tools.storage.smart_write_checkpoint(adata, uri, *, fmt='h5ad')[source]#

Write AnnData to any URI in h5ad or zarr format.

For local paths writes directly. For remote URIs with fmt="h5ad" the file is serialised to a buffer and then uploaded. Zarr format delegates to adata.write_zarr(uri) which uses fsspec natively.

Parameters:

  • adata (Any) – AnnData object to write.

  • uri (str | PathLike) – Destination path or URI.

  • fmt (str) – "h5ad" (default) or "zarr". Prefer "zarr" for cloud storage (s3://, gs://, az://) — it is chunked and cloud-native.

Return type:

None

sc_tools.storage.smart_read_csv(uri, **kwargs)[source]#

Read a CSV or TSV file from any URI.

Parameters:

  • uri (str | PathLike) – Local path or URI to a CSV or TSV file.

  • **kwargs (Any) – Forwarded to pd.read_csv.

Return type:

pd.DataFrame