zea.Dataset

class zea.Dataset(file_paths, validate=False, directory_splits=None, revision=None, lazy=True, file_filter=None, _suggest_lazy=True, **kwargs)

Bases: H5FileHandleCache

Dataset class managing multiple File objects.

Initializes the Dataset.

Parameters:

• file_paths (Union[Sequence[str | Path], str, Path]) – (list of) path(s) to the folder(s) containing the HDF5 file(s) or list of HDF5 file paths. Can be a mixed list of folders and files.

• validate (bool) – Whether to validate the dataset. Defaults to True.

• directory_splits (list | None) – List of directory split by. Is a list of floats between 0 and 1, with the same length as the number of file_paths given. If none, all files in file_paths are used.

• revision (str | None) – HuggingFace revision (branch, tag, or commit hash). Only used when file_paths contains hf:// paths. Defaults to None (uses HuggingFace Hub default, i.e. the main branch).

• lazy (bool) – If True, hf:// files are not downloaded at init — each file is downloaded on first access. len(ds) returns the number of files (not total frames). Defaults to True.

• file_filter (Union[Callable[[File], bool], dict, None]) – Keep only files whose content matches a predicate. Either a callable File -> bool (a file is kept when it returns True), or a declarative dotted-path dict mapping a path on the File (e.g. "metadata.subject.fat_percentage", "scan.center_frequency") to a condition: the EXISTS() helper (field must be present), a plain value (equality), or a callable on the resolved value. All dict entries are ANDed. Files whose predicate raises (e.g. no metadata group) are excluded, except that I/O errors (OSError from HDF5/network reads) propagate rather than being treated as a mismatch. Evaluating the predicate reads each file; with lazy=True remote (hf://) files are streamed for just the metadata/scan bytes rather than downloaded in full, and the surviving files stay lazy. Defaults to None (no filtering).

__call__()

Call self as a function.

copy(to_path, key, mode=None)

Copy the data for all or a specific key to a new location.

Works for a dataset built from a single file, a list of files, or a folder. Each file is written under to_path, mirroring its location relative to the common parent directory of the dataset’s files (for a single file, its own name is used).

Has the option to copy all keys or only a specific key. By default, it only copies if the destination file does not already contain the key. You can change the mode to ‘w’ to overwrite the destination file. Will always copy metadata such as dataset attributes and scan object.

Parameters:

• to_path (str | Path) – The destination folder where files will be copied.

• key (str) – The key to copy from the source files. If ‘all’ or ‘*’, all keys will be copied.

• mode (str | None) – The mode in which to open the destination files. Defaults to ‘a’ (append mode), and ‘w’ (write mode) if key is ‘all’ or ‘*’. See: https://docs.h5py.org/en/stable/high/file.html#opening-creating-files

find_files(paths)

Find files and optionally validate folders and files.

Return type:

List[str]

classmethod from_config(path, user=None, **kwargs)

Creates a Dataset from a config file.

Return type:

Self

load_file_shapes(key)

Load the shapes of the datasets in each file.

property n_files

Return number of files in dataset.

property total_frames

Return total number of frames in dataset.