Skip to content

Schema

Schema

Bases: _Declared

One record's schema.

Attributes:

Name Type Description
version int

Bumped by any change to the declarations. A reader meeting a version it was not written for should refuse rather than guess.

dimensions dict[str, Dimension]

Every declared axis, keyed by dim name.

properties dict[str, dict[str, PropertySpec]]

Subject dim -> property name -> spec: the columns of dims/{dim}.parquet that describe its labels. A name is unique within its subject only, addressed as subject.name. A property of a dim in partial that refers to a dim with one_of categorizes its subject. A dim with one_of has none.

attributes dict[str, AttributeSpec]

Attribute -> spec, record-wide. One attribute is one spec and one attributes/<attr>.parquet with one dtype.

partial Names | None

Which dims a layer may patch value by value. None for a record with no layers, since nothing overrides anything. A dim outside it is one a layer owns entirely once it touches it.

meta dict[str, Any]

A framework's own top-level data - network attributes, CRS, free-form metadata. Stored and never interpreted, since none of it describes the dimensioned data.

Notes

digest cached property

digest: str

The sha256 hex of the schema's compact JSON, as its manifest writes it.

Equal for equal schemas and stable across processes, so it keys a schema wherever the schema itself cannot. Computed once: the nested mappings must not be mutated; derive a changed schema with model_copy(update=...).

dims property

dims: tuple[str, ...]

Every declared dim but the multi-dims, in declaration order.

Notes

multi_dims property

multi_dims: tuple[str, ...]

Every declared multi-dim, in declaration order, each carried in a row as one struct column named for it.

Notes

one_of_dims property

one_of_dims: tuple[str, ...]

Every dim with one_of, in declaration order: those whose axis is the schema's list.

Notes

broadcast_dims property

broadcast_dims: tuple[str, ...]

The coordinates a NULL broadcasts over: every declared dim, the multi-dims last.

A NULL here means "every value of this dim", which the fold expands against the axis; a NULL struct means every tuple of the resolved multi-dim.

What the varies/broadcast structs have a field per, and what expand_dims joins.

Notes

partial_dims property

partial_dims: tuple[str, ...]

The fold key's coordinates, in declaration order.

The dims a layer may patch per value (partial). The fold's key is one fixed tuple over all attributes, so it carries every axis any layer may patch by value or by row, not only those some currently declared attribute varies over. An attribute not owned per one of them writes NULL there, the "NULL means all values" rule - which also lets a schema declare an axis before any attribute uses it.

Notes

long_columns property

long_columns: tuple[str, ...]

The long schema's full column set.

The map's column set, which is uniform across attributes because the map is one relation over all of them. An individual file carries only its own attribute's columns (long_columns_for), and union_by_name supplies NULL for the rest when the fold unions them here.

Notes

owner_map_key property

owner_map_key: tuple[str, ...]

Owner-map key columns, compared NULL-safely when folding.

partial_dims, plus attribute: a layer may patch one value along a partial dim without restating every other's.

A coordinate an attribute's own file does not carry reads as NULL, which is what makes the key one fixed tuple over attributes whose columns differ.

Notes

owner_map_columns property

owner_map_columns: tuple[str, ...]

The owner map's full column set.

Notes

model_copy

model_copy(
    *,
    update: Mapping[str, Any] | None = None,
    deep: bool = False,
) -> Self

A copy as pydantic makes it, without the digest of the original.

Source code in src/datarecord/schema/schema.py
def model_copy(
    self, *, update: Mapping[str, Any] | None = None, deep: bool = False
) -> Self:
    """A copy as pydantic makes it, without the `digest` of the original."""
    copied = super().model_copy(update=update, deep=deep)
    copied.__dict__.pop("digest", None)
    return copied

one_of_properties

one_of_properties() -> dict[tuple[str, str], str]

Map each property that refers to a dim, as (subject, name), to that dim.

Notes
Source code in src/datarecord/schema/schema.py
def one_of_properties(self) -> dict[tuple[str, str], str]:
    """Map each property that refers to a dim, as `(subject, name)`, to that dim.

    Notes
    -----
    - [properties](https://energy-models.github.io/datarecord/reference/data-model/#properties)
    """
    return {
        (subject, name): spec.one_of
        for subject, by_name in self.properties.items()
        for name, spec in by_name.items()
        if spec.one_of is not None
    }

over

over(dim: str) -> dict[str, str]

The coordinates of dim, each to the dim it draws on, in field order; empty for a dim that is no multi-dim.

Raises:

Type Description
KeyError

If the schema declares no dim.

Notes
Source code in src/datarecord/schema/schema.py
def over(self, dim: str) -> dict[str, str]:
    """The coordinates of `dim`, each to the dim it draws on, in field order; empty for a dim that is no multi-dim.

    Raises
    ------
    KeyError
        If the schema declares no `dim`.

    Notes
    -----
    - [multi-dims](https://energy-models.github.io/datarecord/reference/data-model/#multi-dims)
    """
    return self.dimensions[dim].over or {}

coordinates_of

coordinates_of(attribute: str) -> tuple[str, ...]

The coordinate columns one attribute's rows carry, multi-dims first.

A multi-dim in dims is carried as one struct column named for it, so dims={"connection", "snapshot"} gives ("connection", "snapshot").

Per attribute rather than schema-wide: one file per attribute means one column set per attribute, and an all-NULL column of a dim the attribute is not addressed by would claim a coordinate the value has none of.

Raises:

Type Description
KeyError

If the schema does not declare attribute.

Notes
Source code in src/datarecord/schema/schema.py
def coordinates_of(self, attribute: str) -> tuple[str, ...]:
    """The coordinate columns one attribute's rows carry, multi-dims first.

    A multi-dim in `dims` is carried as one struct column named for it, so
    `dims={"connection", "snapshot"}` gives `("connection", "snapshot")`.

    Per attribute rather than schema-wide: one file per attribute means one
    column set per attribute, and an all-NULL column of a dim the attribute
    is not addressed by would claim a coordinate the value has none of.

    Raises
    ------
    KeyError
        If the schema does not declare `attribute`.

    Notes
    -----
    - [multi-dims](https://energy-models.github.io/datarecord/reference/data-model/#multi-dims)
    - [addressing](https://energy-models.github.io/datarecord/reference/data-model/#a-multi-dims-column)
    - [the long schema](https://energy-models.github.io/datarecord/reference/format/#attributesattrparquet)
    """
    if attribute not in self.attributes:
        msg = f"the schema declares no attribute {attribute!r}"
        raise KeyError(msg)
    spec = self.attributes[attribute]
    return (
        *self.multi_dims_of(attribute),
        *(d for d in self.dims if d in spec.dims),
    )

duck_type

duck_type(
    spec: Dimension | AttributeSpec | PropertySpec,
) -> DuckDBPyType

Return the DuckDB type of a declared dim, attribute or property.

A dim with one_of has the ENUM of them, a multi-dim the struct of a field per coordinate, typed as the dim it draws on. A property that refers to a dim has the type of that dim.

Notes
Source code in src/datarecord/schema/schema.py
def duck_type(self, spec: Dimension | AttributeSpec | PropertySpec) -> DuckDBPyType:
    """Return the DuckDB type of a declared dim, attribute or property.

    A dim with `one_of` has the `ENUM` of them, a multi-dim the struct of
    a field per coordinate, typed as the dim it draws on. A property that
    refers to a dim has the type of that dim.

    Notes
    -----
    - [dtypes](https://energy-models.github.io/datarecord/reference/schema/#dtypes)
    - [multi-dims](https://energy-models.github.io/datarecord/reference/data-model/#multi-dims)
    """
    if isinstance(spec, PropertySpec) and spec.one_of is not None:
        return self.duck_type(self.dimensions[spec.one_of])
    if isinstance(spec, Dimension) and spec.over is not None:
        return duckdb.struct_type(
            {c: self.duck_type(self.dimensions[d]) for c, d in spec.over.items()}
        )
    if isinstance(spec, Dimension) and spec.one_of is not None:
        return enum_type(spec.one_of)
    return DUCK_TYPES[cast("str", spec.dtype)]

check_multi_dim_columns

check_multi_dim_columns(
    multi_dims: Iterable[str],
    rel: DuckDBPyRelation,
    *,
    carrier: str,
) -> None

Reject each of multi_dims carried other than as one struct column of exactly its coordinates.

Judged by the relation's columns and types, so no row is read. A carrier without a multi-dim's column passes.

Parameters:

Name Type Description Default
carrier str

What carries the columns, named in the message.

required

Raises:

Type Description
ValueError

On a coordinate as a column of its own, or a multi-dim column not a struct of exactly the coordinates, naming the struct to pass instead.

Notes
Source code in src/datarecord/schema/schema.py
def check_multi_dim_columns(
    self,
    multi_dims: Iterable[str],
    rel: DuckDBPyRelation,
    *,
    carrier: str,
) -> None:
    """Reject each of `multi_dims` carried other than as one struct column of exactly its coordinates.

    Judged by the relation's columns and types, so no row is read. A
    carrier without a multi-dim's column passes.

    Parameters
    ----------
    carrier
        What carries the columns, named in the message.

    Raises
    ------
    ValueError
        On a coordinate as a column of its own, or a multi-dim column not
        a struct of exactly the coordinates, naming the struct to pass
        instead.

    Notes
    -----
    - [multi-dims](https://energy-models.github.io/datarecord/reference/data-model/#multi-dims)
    """
    types = dict(zip(rel.columns, rel.types, strict=True))
    for multi in multi_dims:
        key = list(self.over(multi))
        flat = [c for c in key if c in types]
        dtype = types.get(multi)
        fields = (
            key
            if dtype is None
            else [name for name, _ in dtype.children]
            if dtype.id == "struct"
            else None
        )
        if flat or fields is None or set(fields) != set(key):
            carried = f"{flat} flat" if flat else f"{multi!r} as {dtype}"
            msg = (
                f"{carrier} carries {carried}; pass {multi!r} as one struct "
                f"column with a field per coordinate, {key} (https://energy-models.github.io/datarecord/reference/data-model/#multi-dims)"
            )
            raise ValueError(msg)

refuse_partial_tuples

refuse_partial_tuples(
    multi_dims: Iterable[str],
    rel: DuckDBPyRelation,
    *,
    carrier: str,
) -> None

Reject a struct of any of multi_dims in rel with some fields NULL and not all.

Reads the rows, unlike check_multi_dim_columns. A carrier without a multi-dim's column passes.

Parameters:

Name Type Description Default
carrier str

What carries the columns, named in the message.

required

Raises:

Type Description
ValueError

Naming carrier and the multi-dim.

Notes
Source code in src/datarecord/schema/schema.py
def refuse_partial_tuples(
    self,
    multi_dims: Iterable[str],
    rel: DuckDBPyRelation,
    *,
    carrier: str,
) -> None:
    """Reject a struct of any of `multi_dims` in `rel` with some fields NULL and not all.

    Reads the rows, unlike `check_multi_dim_columns`. A carrier without a
    multi-dim's column passes.

    Parameters
    ----------
    carrier
        What carries the columns, named in the message.

    Raises
    ------
    ValueError
        Naming `carrier` and the multi-dim.

    Notes
    -----
    - [the broadcast rule](https://energy-models.github.io/datarecord/reference/data-model/#the-broadcast-rule)
    """
    for multi in multi_dims:
        if multi not in rel.columns:
            continue
        some = reduce(
            operator.or_, (col(multi)[c].isnull() for c in self.over(multi))
        )
        partly = col(multi).isnotnull() & some
        if rel.filter(partly).limit(1).fetchall():
            msg = (
                f"{carrier} carries a {multi!r} with some fields NULL; a "
                f"multi-dim value is a whole tuple, or NULL for every tuple (https://energy-models.github.io/datarecord/reference/data-model/#the-broadcast-rule)"
            )
            raise ValueError(msg)

long_columns_for

long_columns_for(attribute: str) -> tuple[str, ...]

Return one attribute's full long column set, in order.

An attribute carries the coordinates its dims name and no others, so a per-snapshot weighting has no column for any other dim.

Raises:

Type Description
KeyError

If the schema does not declare attribute.

Notes
Source code in src/datarecord/schema/schema.py
def long_columns_for(self, attribute: str) -> tuple[str, ...]:
    """Return one attribute's full long column set, in order.

    An attribute carries the coordinates its `dims` name and no others, so a
    per-snapshot weighting has no column for any other dim.

    Raises
    ------
    KeyError
        If the schema does not declare `attribute`.

    Notes
    -----
    - [the long schema](https://energy-models.github.io/datarecord/reference/format/#attributesattrparquet)
    """
    return (*self.coordinates_of(attribute), *LONG_TAIL)

categorizing

categorizing(dim: str) -> tuple[str, ...]

Return the categorizing properties of dim, the ones default_by keys on.

A categorizing property is a property of dim that refers to a dim with one_of. Only a dim in partial has any.

Notes
Source code in src/datarecord/schema/schema.py
def categorizing(self, dim: str) -> tuple[str, ...]:
    """Return the categorizing properties of `dim`, the ones `default_by` keys on.

    A categorizing property is a property of `dim` that refers to a dim with
    `one_of`. Only a dim in `partial` has any.

    Notes
    -----
    - [categories](https://energy-models.github.io/datarecord/reference/data-model/#categories)
    """
    if dim not in (self.partial or set()):
        return ()
    return tuple(
        name
        for name, spec in self.properties.get(dim, {}).items()
        if spec.one_of is not None
        and self.dimensions[spec.one_of].one_of is not None
    )

require_categorizing

require_categorizing(
    subject: str, keys: Iterable[str]
) -> None

Refuse a key that is not a categorizing property of subject.

Raises:

Type Description
ValueError

Naming the keys and subject's categorizing properties.

Notes
Source code in src/datarecord/schema/schema.py
def require_categorizing(self, subject: str, keys: Iterable[str]) -> None:
    """Refuse a key that is not a categorizing property of `subject`.

    Raises
    ------
    ValueError
        Naming the keys and `subject`'s categorizing properties.

    Notes
    -----
    - [categories](https://energy-models.github.io/datarecord/reference/data-model/#categories)
    """
    allowed = self.categorizing(subject)
    unknown = sorted(set(keys) - set(allowed))
    if unknown:
        msg = (
            f"{unknown} do not categorize {subject!r}; its categorizing "
            f"properties are {list(allowed)}"
        )
        raise ValueError(msg)

require_one_of

require_one_of(
    subject: str,
    data: Mapping[str, Iterable[Any]],
    what: str,
) -> None

Refuse a label its dim does not list.

data maps column names to values. A name is checked where it is subject and subject has one_of, or where it is a property of subject that refers to a dim with one_of. Other names and NULL values are skipped.

Raises:

Type Description
ValueError

Naming what, the column, the unknown labels and the dim's one_of labels.

Notes
Source code in src/datarecord/schema/schema.py
def require_one_of(
    self, subject: str, data: Mapping[str, Iterable[Any]], what: str
) -> None:
    """Refuse a label its dim does not list.

    `data` maps column names to values. A name is checked where it is
    `subject` and `subject` has `one_of`, or where it is a property of
    `subject` that refers to a dim with `one_of`. Other names
    and NULL values are skipped.

    Raises
    ------
    ValueError
        Naming `what`, the column, the unknown labels and the dim's
        `one_of` labels.

    Notes
    -----
    - [dtypes](https://energy-models.github.io/datarecord/reference/schema/#dtypes)
    """
    specs = self.properties.get(subject, {})
    for name, values in data.items():
        if name in specs:
            labels = self._one_of_labels(specs[name])
        elif name == subject and subject in self.dimensions:
            labels = self._one_of_labels(self.dimensions[subject])
        else:
            continue
        if labels is None:
            continue
        unknown = sorted({str(v) for v in values if v is not None} - {*labels})
        if unknown:
            msg = (
                f"{what} gives {name!r} labels {unknown}, which its dim's "
                f"`one_of` does not hold; `one_of` is {list(labels)}"
            )
            raise ValueError(msg)

default_for

default_for(name: str, **categories: str) -> Any

The default of attribute or subject.name property name for a label of categories.

categories are keyed by each categorizing property's own name. The first default_by key listing the label's category decides, else default; a category left out matches none.

Raises:

Type Description
KeyError

As resolve does, and on a coordinate, which has no default.

Notes
Source code in src/datarecord/schema/schema.py
def default_for(self, name: str, **categories: str) -> Any:
    """The default of attribute or `subject.name` property `name` for a label of `categories`.

    `categories` are keyed by each categorizing property's own name. The
    first `default_by` key listing the label's category decides, else
    `default`; a category left out matches none.

    Raises
    ------
    KeyError
        As `resolve` does, and on a coordinate, which has no default.

    Notes
    -----
    - [AttributeSpec](https://energy-models.github.io/datarecord/reference/schema/#attributes)
    - [default_by](https://energy-models.github.io/datarecord/reference/schema/#default_by)
    """
    resolved = self.resolve(name)
    spec = resolved[1] if isinstance(resolved, tuple) else resolved
    if isinstance(spec, Dimension):
        msg = f"{name!r} is a coordinate, part of its multi-dim's key, and has no default"
        raise KeyError(msg)
    for key, by in spec.default_by.items():
        category = categories.get(key.partition(".")[2])
        if category in by:
            return by[category]
    return spec.default

addresses

addresses(attribute: str, dim: str) -> bool

Whether attribute reaches a label of dim at all.

True where its dims name dim, or a multi-dim one of whose coordinates draws on dim.

Notes
Source code in src/datarecord/schema/schema.py
def addresses(self, attribute: str, dim: str) -> bool:
    """Whether `attribute` reaches a label of `dim` at all.

    True where its `dims` name `dim`, or a multi-dim one of whose
    coordinates draws on `dim`.

    Notes
    -----
    - [where a value lives](https://energy-models.github.io/datarecord/reference/format/#where-a-value-lives)
    """
    spec = self.attributes.get(attribute)
    return (spec is not None and dim in spec.dims) or any(
        dim in self.over(m).values() for m in self.multi_dims_of(attribute)
    )

resolve

resolve(
    name: str,
) -> (
    AttributeSpec
    | tuple[str, PropertySpec]
    | tuple[str, Dimension]
)

What name addresses: an attribute, or a property or coordinate as subject.name.

Read off the name alone: a plain name is never a property.

Returns:

Type Description
AttributeSpec or tuple of str and PropertySpec or Dimension

The attribute's spec, a property's subject and spec, or a coordinate's multi-dim and the dim the coordinate draws on.

Raises:

Type Description
KeyError

If name addresses nothing; for a plain name some subject's property is called, naming each dotted form.

Notes
Source code in src/datarecord/schema/schema.py
def resolve(
    self, name: str
) -> AttributeSpec | tuple[str, PropertySpec] | tuple[str, Dimension]:
    """What `name` addresses: an attribute, or a property or coordinate as `subject.name`.

    Read off the name alone: a plain name is never a property.

    Returns
    -------
    AttributeSpec or tuple of str and PropertySpec or Dimension
        The attribute's spec, a property's subject and spec, or a
        coordinate's multi-dim and the dim the coordinate draws on.

    Raises
    ------
    KeyError
        If `name` addresses nothing; for a plain name some subject's
        property is called, naming each dotted form.

    Notes
    -----
    - [properties](https://energy-models.github.io/datarecord/reference/data-model/#properties)
    - [coordinates](https://energy-models.github.io/datarecord/reference/data-model/#multi-dims)
    """
    subject, dot, prop = name.partition(".")
    if dot:
        spec = self.properties.get(subject, {}).get(prop)
        if spec is not None:
            return subject, spec
        drawn = self.over(subject).get(prop) if subject in self.dimensions else None
        if drawn is None:
            msg = f"the schema declares no property {name!r}"
            raise KeyError(msg)
        return subject, self.dimensions[drawn]
    attribute = self.attributes.get(name)
    if attribute is not None:
        return attribute
    dotted = sorted(
        f"`{s}.{name}`" for s, specs in self.properties.items() if name in specs
    )
    hint = f"; did you mean {' or '.join(dotted)}" if dotted else ""
    msg = f"the schema declares no attribute {name!r}{hint}"
    raise KeyError(msg)

coordinate_over

coordinate_over(multi: str, dim: str) -> str | None

The one coordinate of the multi-dim multi drawing on dim, if any.

Raises:

Type Description
ValueError

If several coordinates of multi draw on dim.

Source code in src/datarecord/schema/schema.py
def coordinate_over(self, multi: str, dim: str) -> str | None:
    """The one coordinate of the multi-dim `multi` drawing on `dim`, if any.

    Raises
    ------
    ValueError
        If several coordinates of `multi` draw on `dim`.
    """
    fields = [f for f, d in self.over(multi).items() if d == dim]
    if len(fields) > 1:
        msg = (
            f"multi-dim {multi!r} is over {dim!r} through {fields}, so no one "
            f"label of it decides what a tuple carries"
        )
        raise ValueError(msg)
    return fields[0] if fields else None

owned_per

owned_per(attribute: str) -> frozenset[str]

Which dims a layer owns attribute per.

Derived rather than declared: AttributeSpec.dims says which axes the attribute may vary over, partial_dims the fold key (the partial dims), and ownership is their intersection. A dim in dims but not the fold key - a non-partial value axis like timestep - is owned whole, so a patch to one of its values restates the attribute's entire extent along it (_owned_whole).

Notes
Source code in src/datarecord/schema/schema.py
def owned_per(self, attribute: str) -> frozenset[str]:
    """Which dims a layer owns `attribute` per.

    Derived rather than declared: `AttributeSpec.dims` says which axes the
    attribute may vary over, `partial_dims` the fold key (the `partial`
    dims), and ownership is their intersection. A
    dim in `dims` but not the fold key - a non-`partial` value axis like
    `timestep` - is owned whole, so a patch to one of its values restates the
    attribute's entire extent along it (`_owned_whole`).

    Notes
    -----
    - [partial](https://energy-models.github.io/datarecord/reference/layers/#partial)
    - [one fold for every axis](https://energy-models.github.io/datarecord/reference/resolution/#axes)
    """
    spec = self.attributes.get(attribute)
    if spec is None:
        return frozenset()
    return spec.dims & set(self.partial_dims)

multi_dims_of

multi_dims_of(attribute: str) -> tuple[str, ...]

Which declared multi-dims address attribute, in declaration order.

Notes
Source code in src/datarecord/schema/schema.py
def multi_dims_of(self, attribute: str) -> tuple[str, ...]:
    """Which declared multi-dims address `attribute`, in declaration order.

    Notes
    -----
    - [addressing](https://energy-models.github.io/datarecord/reference/data-model/#a-multi-dims-column)
    """
    spec = self.attributes.get(attribute)
    if spec is None:
        return ()
    return tuple(m for m in self.multi_dims if m in spec.dims)

column_type

column_type(
    column: str, subject: str | None = None
) -> DuckDBPyType | None

Return the declared DuckDB type of column, or None if the schema declares none.

Covers the structural columns the format fixes, the declared dims, the properties of subject - columns of its axis file, read first - and the owner map's two flag structs, whose fields follow the schema's dims.

No dim is structural: each is declared, and typed from that declaration. So the ENUM of a dim with one_of pins them everywhere the column is built. An attribute is no column but a long row's value, typed by value_type.

A schema declaring no dims at all is "no manifest yet" rather than a record to fold, and DuckDB has no empty struct - so the flag columns are undeclared there, and a caller building an empty relation falls back to VARCHAR for a map that will never hold a row.

Notes
Source code in src/datarecord/schema/schema.py
def column_type(
    self, column: str, subject: str | None = None
) -> DuckDBPyType | None:
    """Return the declared DuckDB type of `column`, or None if the schema declares none.

    Covers the structural columns the format fixes, the declared dims,
    the properties of `subject` - columns of its
    axis file, read first - and the owner map's two flag structs, whose fields
    follow the schema's dims.

    No dim is structural: each is declared, and typed from that
    declaration. So the `ENUM` of a dim with `one_of` pins them
    everywhere the column is built. An attribute is no column but a long row's value,
    typed by `value_type`.

    A schema declaring no dims at all is "no manifest yet" rather
    than a record to fold, and DuckDB has no empty struct - so the flag
    columns are undeclared there, and a caller building an empty relation
    falls back to `VARCHAR` for a map that will never hold a row.

    Notes
    -----
    - [one schema per record](https://energy-models.github.io/datarecord/reference/schema/#one-schema-per-record)
    - [the owner map](https://energy-models.github.io/datarecord/reference/resolution/#the-owner-map)
    """
    found = self.declared(column, subject)
    return None if found is None else found[1]

declared

declared(
    column: str,
    subject: str | None = None,
    attribute: str | None = None,
) -> tuple[str, DuckDBPyType, str] | None

Look up the declaration of column.

Parameters:

Name Type Description Default
subject str | None

The dim whose file holds the column. A column named like one of its properties is that property, named subject.name.

None
attribute str | None

The attribute whose long rows hold the column. Its value column is the attribute, named as it is.

None

Returns:

Type Description
tuple of str, DuckDBPyType and str, or None

The declared name, the DuckDB type, and the dtype as an error message shows it; None where the schema declares no type for column.

Notes
Source code in src/datarecord/schema/schema.py
def declared(
    self, column: str, subject: str | None = None, attribute: str | None = None
) -> tuple[str, DuckDBPyType, str] | None:
    """Look up the declaration of `column`.

    Parameters
    ----------
    subject
        The dim whose file holds the column. A column named like
        one of its properties is that property, named `subject.name`.
    attribute
        The attribute whose long rows hold the column. Its `value` column
        is the attribute, named as it is.

    Returns
    -------
    tuple of str, DuckDBPyType and str, or None
        The declared name, the DuckDB type, and the dtype as an error
        message shows it; None where the schema declares no type for
        `column`.

    Notes
    -----
    - [dtypes](https://energy-models.github.io/datarecord/reference/schema/#dtypes)
    """
    own = self.properties.get(subject or "", {})
    spec: Dimension | AttributeSpec | PropertySpec
    if attribute is not None and column == "value":
        name, spec = attribute, self.attributes[attribute]
    elif column in own:
        name, spec = f"{subject}.{column}", own[column]
    elif column in self.dimensions and column not in STRUCTURAL_TYPES:
        name, spec = column, self.dimensions[column]
    else:
        duck = self._structural_type(column)
        return None if duck is None else (column, duck, str(duck))
    duck = self.duck_type(spec)
    return name, duck, f"{_typed_as(spec)} ({duck})"

require_takes

require_takes(
    rel: DuckDBPyRelation,
    what: str,
    subject: str | None = None,
    attribute: str | None = None,
) -> DuckDBPyRelation

Check that each column of rel has a type that fits its declaration, and return rel, with its all-NULL columns cast to their declared types.

A column type fits where takes says so. A column that fails this but holds only NULLs passes and is cast to its declared type, because dataframe libraries guess the type of an empty or all-None column. Only such columns have their rows read. subject and attribute are as in declared.

Raises:

Type Description
ValueError

Naming what, and for each refused column its type, its declared name and its dtype.

Notes
Source code in src/datarecord/schema/schema.py
def require_takes(
    self,
    rel: DuckDBPyRelation,
    what: str,
    subject: str | None = None,
    attribute: str | None = None,
) -> DuckDBPyRelation:
    """Check that each column of `rel` has a type that fits its declaration, and return `rel`, with its all-NULL columns cast to their declared types.

    A column type fits where `takes` says so. A column
    that fails this but holds only NULLs passes and is cast to its declared
    type, because dataframe libraries guess the type of an empty or
    all-None column. Only such columns have their rows read. `subject` and
    `attribute` are as in `declared`.

    Raises
    ------
    ValueError
        Naming `what`, and for each refused column its type, its
        declared name and its dtype.

    Notes
    -----
    - [dtypes](https://energy-models.github.io/datarecord/reference/schema/#dtypes)
    - [set](https://energy-models.github.io/datarecord/reference/editing/#set)
    """
    wrong = []
    nulls: dict[str, DuckDBPyType] = {}
    for c, given in zip(rel.columns, rel.types, strict=True):
        found = self.declared(c, subject, attribute)
        if found is None or takes(found[1], given):
            continue
        if not rel.filter(col(c).isnotnull()).limit(1).fetchall():
            nulls[c] = found[1]
            continue
        name, _, shown = found
        wrong.append(
            f"column {c!r} has type {given}, which does not fit {name!r} "
            f"of dtype {shown}"
        )
    if wrong:
        msg = f"{what}: {'; '.join(wrong)}"
        raise ValueError(msg)
    if not nulls:
        return rel
    return rel.project(
        *(
            col(c).cast(nulls[c]).alias(c) if c in nulls else col(c)
            for c in rel.columns
        )
    )

value_type

value_type(attribute: str) -> DuckDBPyType | None

Return the DuckDB type of the value column of attribute.

One attribute is one attributes/<attr>.parquet with one value column, so the dtype is the attribute's alone.

Notes
Source code in src/datarecord/schema/schema.py
def value_type(self, attribute: str) -> DuckDBPyType | None:
    """Return the DuckDB type of the `value` column of `attribute`.

    One attribute is one `attributes/<attr>.parquet` with one `value`
    column, so the dtype is the attribute's alone.

    Notes
    -----
    - [the long schema](https://energy-models.github.io/datarecord/reference/format/#attributesattrparquet)
    """
    spec = self.attributes.get(attribute)
    return None if spec is None else DUCK_TYPES[spec.dtype]

compatible_with

compatible_with(other: Schema) -> list[str]

Why layers written under other would not read under self.

Empty when the change is compatible: old layers stay readable and only version moves. The compatible changes are those where NULL already means what the new schema needs it to mean, so the broadcast rule absorbs them without touching a row.

Returns:

Type Description
list of str

One reason per incompatibility, empty if there are none.

Notes
Source code in src/datarecord/schema/schema.py
def compatible_with(self, other: Schema) -> list[str]:
    """Why layers written under `other` would not read under `self`.

    Empty when the change is compatible: old layers stay readable and only
    `version` moves. The compatible changes are those where NULL already
    means what the new schema needs it to mean, so the broadcast rule absorbs
    them without touching a row.

    Returns
    -------
    list of str
        One reason per incompatibility, empty if there are none.

    Notes
    -----
    - [the broadcast rule](https://energy-models.github.io/datarecord/reference/data-model/#the-broadcast-rule)
    - [versioning](https://energy-models.github.io/datarecord/reference/schema/#compatibility)
    """
    problems = []

    for dim, was in other.dimensions.items():
        now = self.dimensions.get(dim)
        if now is None:
            problems.append(f"dim {dim!r} removed")
            continue
        if _typed_as(now) != _typed_as(was):
            problems.append(f"dim {dim!r} {_typed_as(was)} -> {_typed_as(now)}")

    for attr, was_spec in other.attributes.items():
        now_spec = self.attributes.get(attr)
        if now_spec is None:
            problems.append(f"attribute {attr!r} removed")
            continue
        if now_spec.dtype != was_spec.dtype:
            problems.append(
                f"attribute {attr!r} dtype {was_spec.dtype} -> {now_spec.dtype}"
            )
        narrowed = was_spec.dims - now_spec.dims
        if narrowed:
            problems.append(
                f"attribute {attr!r} no longer varies over {sorted(narrowed)}; "
                f"rows setting those dims have no valid reading"
            )

    for dim, specs in other.properties.items():
        for name, was_prop in specs.items():
            now_prop = self.properties.get(dim, {}).get(name)
            if now_prop is None:
                problems.append(f"property {name!r} of {dim!r} removed")
            elif _typed_as(now_prop) != _typed_as(was_prop):
                problems.append(
                    f"property {name!r} of {dim!r} {_typed_as(was_prop)} "
                    f"-> {_typed_as(now_prop)}"
                )

    if other.partial is not None and self.partial is not None:
        lost = other.partial - self.partial
        if lost:
            problems.append(
                f"{sorted(lost)} no longer `partial`; a layer that patched one "
                f"value along such an axis is now a partial override of an axis "
                f"owned whole"
            )
    return problems

Dimension

Bases: _Declared

One axis attribute data may vary over: its shape, not its data.

Not which dims an attribute varies over (AttributeSpec.dims), nor the patch granularity (Schema.partial), nor order - an axis is ordered by its file's row order, undeclared.

A dim declaring over is a multi-dim: its labels are tuples of other dims' labels, carried in a row as one struct column named for it, a field per coordinate (Schema.multi_dim_type). Coordinates rather than dims because two of them may draw on one dim: a corridor between two buses is (from, to).

Attributes:

Name Type Description
over dict[str, str] | None

Coordinate name -> the dim it draws its labels from, or None for a dim that is no multi-dim. A list is sugar for the dict with identical keys and values. Declared without dtype and one_of.

dtype DimensionDtype | None

The axis labels' type, a mathspec DimensionDtype; "str" where undeclared, None on a multi-dim.

one_of tuple[str, ...] | None

The dim's labels, in order, or None for labels a layer writes. A dim with one_of has its axis derived from the list, no axis file and no properties, and an ENUM of the labels as its column; add and remove refuse it. Only a str dim takes one_of.

unit str | None

What this axis's labels measure, if anything - None is undeclared, "" genuinely dimensionless.

description str | None

What the axis is, in prose. Never interpreted.

Notes

AttributeSpec

Bases: _ValueSpec

What shape one attribute's data may take.

Attributes:

Name Type Description
dtype ParameterDtype

The value column's type, a mathspec ParameterDtype.

dims Names

Dims this attribute may vary over; a subset of those declared. Whatever they are, its values are long rows of attributes/<attr>.parquet.

default Any | None

The value a coordinate no row covers takes.

default_by dict[str, dict[str, Any]]

A categorizing property, as subject.name, -> category -> the default for labels of that category, overriding default; the first key, in declaration order, listing a label's category wins. Each key must categorize a dim the attribute reaches. Read through Schema.default_for; never applied by the record layer.

breakpoints bool

Whether it may carry a piecewise-linear curve.

unit str | None

What the values measure - "MW", "EUR/MWh". Stored and never interpreted; None is undeclared, "" genuinely dimensionless.

description str | None

What the attribute is, in prose. Never interpreted.

Notes

PropertySpec

Bases: _ValueSpec

One value per label of a dim, describing it.

A property is keyed by its subject (Schema.properties) and stored as a column of dims/{dim}.parquet. Its name is its subject's alone, so it is addressed as subject.name (Schema.resolve). A property declares exactly one of dtype and one_of. A property of a dim in partial that refers to a dim with one_of categorizes its subject (Schema.categorizing).

Attributes:

Name Type Description
dtype ParameterDtype | None

The column's type, a mathspec ParameterDtype.

one_of str | None

The dim the property refers to. Its values have that dim's type, and a stored value that is not NULL must be a live label of the dim.

default Any | None

The value a label no row sets takes.

default_by dict[str, dict[str, Any]]

A categorizing property of a dim the subject reaches, as subject.name, -> category -> the default for labels of that category, overriding default. Refused on a categorizing property itself. Read through Schema.default_for.

unit str | None

What the values measure; None is undeclared.

description str | None

What the property is, in prose. Never interpreted.

Notes