Store

class haxr.store.Store(*, base_url: str | haxr.doi.DOI | None, cache_dir: pathlib.Path | None = None, allow_download: bool = True)[source]

Local cache and access layer for radar/AIS data and station metadata.

The store manages an on-disk cache directory and can populate it by downloading missing files from base_url (if enabled). Callers typically discover available data via list_chunks() / get_chunk(), then ensure files are present via ensure() or access content using open(), load_ais_data(), and stations.

If cache_dir is not provided, the store creates a temporary directory and deletes it when close() is called or when exiting a context manager.

Warning

Store is a convenience layer around data files on disk. It helps you locate, open, and (optionally) lazily download files into a local cache directory. It is not a dataset version manager.

Choosing an appropriate base_url (e.g., a specific DOI release) is entirely the caller’s responsibility. The store does not track which release a cache directory belongs to, and it does not prevent you from mixing files from different releases on disk.

If you reuse a cache_dir from an incompatible release (assume releases are pairwise incompatible), you might get lucky and notice the problem via SHA256 checksum mismatches during ensure()/download. However, it is also possible that no verification is triggered (because files are already present), and you end up silently working with a different cached release than the one implied by base_url. Note that passing base_url=None does not make the cache version-safe; it only prevents downloads. You must still avoid mixing incompatible releases in one cache.

If you need reproducibility, you must pin base_url to a versioned release and ensure that each release uses its own cache directory (or that old caches are cleaned before switching).

Parameters:

  • base_url –

    Base URL used to download files into the cache. This may be a plain string URL, a DOI value, or None.

    Passing None enables an “offline” mode: the store will only use files that already exist in cache_dir. If a download is required (e.g., manifest.json or a chunk file is missing, or a cached file fails SHA256 verification and a re-download would be needed), the store raises ValueError.

    Note that manifest.json must already be present in cache_dir when using base_url=None with allow_download=True; otherwise initialization will fail when the manifest needs to be fetched.

  • cache_dir – Directory used for the local cache. If None, a temporary cache directory is created automatically.

  • allow_download – If False, operations that require uncached files fail with FileNotFoundError instead of downloading.

Example

from haxr import DOI, Store, load_cycle

with Store(base_url=DOI.latest) as store:
    chunk = store.get_chunk(station="altona", split_hour_utc=9)
    with store.open(chunk.radar_file) as f:
        ...

Notes

close() → None[source]

Close the store and release any owned resources.

If the store created a temporary cache directory (because cache_dir was not provided), this method deletes that directory and all cached files within it. If a user-supplied cache_dir is used, the directory is left untouched.

This method is idempotent and may be called multiple times.

Returns:

None

list_chunks(*, station: str | None = None) → list[haxr.store.Chunk][source]

List available chunks known to this store.

The returned list is sorted deterministically by (station, split_hour_utc) in ascending order. This method only consults the store’s in-memory chunk index created during initialization; it does not touch the network and does not download data files.

Note that the returned Chunk objects may reference files that are not yet present in the local cache directory. Use ensure() (or ensure_file()) to populate missing files.

Parameters:

station – Optional station identifier. If provided, only chunks whose chunk.station == station are returned. If no chunks match, an empty list is returned.

Returns:

A list of chunks, sorted by (station, split_hour_utc).

get_chunk(*, station: str, split_hour_utc: int | str) → haxr.store.Chunk[source]

Return the chunk identified by station and UTC split hour.

This performs a lookup in the store’s in-memory chunk index populated during initialization. It neither access the network nor it ensures that the chunk’s files are present in the local cache. Use ensure() or ensure_file() to populate missing files.

Parameters:

  • station – Station identifier.

  • split_hour_utc – Hour of day in UTC (typically 0..23) used to partition the dataset. May be given as an int or as a decimal string convertible to int (e.g., "9" or "09")

Returns:

The chunk for the given (station, split_hour_utc) key.

Raises:

  • TypeError – If split_hour_utc is neither an int-like value nor a decimal string.

  • ValueError – If split_hour_utc is a string but cannot be converted to int.

  • KeyError – If no chunk exists for the given (station, split_hour_utc). The error message includes the requested key.

ensure_file(file: haxr.store.File, show_progress: bool = True) → pathlib.Path[source]

Ensure that file is present in the local cache.

If the file is already present, it is reused. If file.sha256 is provided, the cached file is verified.

If the file is missing (or verification fails) and downloads are allowed, the store downloads the file into the cache.

Parameters:

  • file – File reference returned by the store.

  • show_progress – Whether to show a download progress bar.

Returns:

Path to the cached file.

Raises:

  • FileNotFoundError – If the file is not cached and downloads are disabled.

  • ValueError – If a download is required but base_url is None/empty.

  • OSError – If the cached file fails SHA256 verification and recovery is not possible (e.g., downloads disabled).

ensure(chunk: haxr.store.Chunk, show_progress: bool = True) → haxr.store.Chunk[source]

Ensure that all files for a chunk are available in the local cache.

This is a convenience wrapper around ensure_file(). The given chunk is first normalized by resolving it through get_chunk(), ensuring that the store’s canonical metadata (paths, optional checksums) is used.

The chunk’s radar and AIS files are then ensured to exist on disk. If a file is already cached, it is reused. If a SHA256 checksum is available, the cached file may be verified; a mismatch can trigger a re-download when downloads are enabled.

Parameters:

  • chunk – Chunk identifying the data to ensure. The (station, split_hour_utc) fields are used to resolve the canonical chunk via get_chunk().

  • show_progress – Whether to show a progress bar while downloading.

Returns:

The canonical chunk instance as returned by get_chunk().

Raises:

  • KeyError – If the chunk’s (station, split_hour_utc) does not exist in this store.

  • FileNotFoundError – If a required file is missing from the cache and downloads are disabled.

  • ValueError – If a download is required but no base_url was provided.

  • OSError – If a cached file fails SHA256 verification and cannot be recovered (e.g., downloads disabled).

open(file: haxr.store.File, mode: str = 'r', show_progress: bool = False, **kwargs) → collections.abc.Iterator[TypeAliasForwardRef('h5py.File') | IO[Any]][source]

Open a cached file and yield a file handle.

This is a context manager. It first ensures that file is present in the local cache via ensure_file() (downloading if necessary, subject to allow_download and base_url). The file is then opened and automatically closed when leaving the context.

For files ending in .hdf5, this uses h5py.File. For all other paths, it uses Python’s built-in open().

Parameters:

  • file – File reference to open.

  • mode – File mode passed to h5py.File or open().

  • show_progress – Whether to show a progress bar while downloading.

  • **kwargs – Additional keyword arguments forwarded to h5py.File or open() (depending on the file type).

Yields:

An open file handle. This is an h5py.File for .hdf5 files, otherwise a regular Python file object.

Notes

Sphinx currently renders the return type in the generated HTML signature incorrectly as TypeAliasForwardRef('h5py.File') instead of h5py.File. This is a known upstream issue in Sphinx. See https://github.com/sphinx-doc/sphinx/issues/14003

Raises:

  • FileNotFoundError – If the file is missing from the cache and downloads are disabled.

  • ValueError – If a download is required but no base_url was provided.

  • OSError – If a cached file fails SHA256 verification, or if opening/reading the file fails (including errors raised by h5py.File).

load_ais_data(file: haxr.store.File, show_progress: bool = False, **kwargs: Any) → pandas.DataFrame[source]

Load AIS data from a cached CSV file.

This is a convenience wrapper around ensure_file() and pandas.read_csv(). The file is ensured to exist in the local cache (downloading if enabled) and is then read as a CSV. Keyword arguments are forwarded to pandas.read_csv() to customize parsing.

Parameters:

  • file – File reference to an AIS CSV (see Chunk, attribute ais_file).

  • show_progress – Whether to show a progress bar while downloading.

  • **kwargs – Keyword arguments forwarded to pandas.read_csv(). Use these to override pandas defaults.

Returns:

A pandas.DataFrame containing the AIS records from the CSV.

Raises:
property stations: pandas.DataFrame

Station metadata table.

Lazily loads the dataset’s station metadata (stations.csv), caches it on the Store instance, and returns a copy of the cached pandas.DataFrame on each access. Callers may freely mutate the returned DataFrame without affecting the cache.

Returns:

A copy of the cached pandas.DataFrame containing station metadata.

Raises:
class haxr.store.File(path: pathlib.Path, sha256: str | None)[source]

A reference to a file managed by the store.

This is a small value object that identifies a file via its cache path and (optionally) an integrity checksum. It may refer to an already-cached file or a file that can be fetched into the cache by the store.

Variables:

  • path (pathlib.Path) – Absolute path to the file location in the store’s cache.

  • sha256 (str | None) – Optional expected SHA256 hex digest for the file contents. If provided, the store can use it to verify the file’s integrity.

class haxr.store.Chunk(station: str, split_hour_utc: int, radar_file: haxr.store.File, ais_file: haxr.store.File)[source]

A logical unit of data for a station and a specific UTC hour split.

A Chunk bundles the files that belong together for one station and one hour-of-day partition. The store returns Chunk objects so callers can locate, fetch, and open the underlying data files.

Variables:

  • station (str) – Station identifier.

  • split_hour_utc (int) – Hour of day in UTC (0..23) used to partition the data.

  • radar_file (haxr.store.File) – Radar data file for this chunk.

  • ais_file (haxr.store.File) – AIS data file for this chunk.