Tools

Tool

Bases: Protocol

One modelling framework's view of a record.

Implementations are stateless (a module-level singleton is fine): every method takes the record or model it operates on. A structural type, for annotating code that takes any tool - not a dispatch table; tools are reached by importing them.

A Record rather than a record throughout, so a tool builds from a directory as readily as from an overlay and has no reason to know layering exists.

Notes

• consuming a record

requires

requires(record: RecordLike) -> Requirements

What this tool needs from record to build a model.

Record-dependent, not a constant: which attributes are required follows the record's own component types and its declared schema.

Source code in src/datarecord/tools/base.py

def requires(self, record: RecordLike) -> Requirements:
    """What this tool needs from `record` to build a model.

    Record-dependent, not a constant: which attributes are required
    follows the record's own component types and its declared schema.
    """
    ...

verify

verify(record: RecordLike) -> Requirements

What record fails to supply; falsy when the record is usable.

Source code in src/datarecord/tools/base.py

def verify(self, record: RecordLike) -> Requirements:
    """What `record` fails to supply; falsy when the record is usable."""
    ...

build

build(record: RecordLike) -> Any

The tool's model object, built from the resolved record.

Source code in src/datarecord/tools/base.py

def build(self, record: RecordLike) -> Any:
    """The tool's model object, built from the resolved record."""
    ...

to_datarecord

to_datarecord(model: Any) -> RecordLike

model presented as a layer write_record can persist.

The inverse of build. Framework-specific: undoing a framework's own shape is exactly what a tool knows and the record layer does not.

Notes

• the record format

Source code in src/datarecord/tools/base.py

def to_datarecord(self, model: Any) -> RecordLike:
    """`model` presented as a layer `write_record` can persist.

    The inverse of `build`. Framework-specific: undoing a framework's own
    shape is exactly what a tool knows and the record layer does not.

    Notes
    -----
    - [the record format](https://energy-models.github.io/datarecord/design/format/)
    """
    ...

results

results(model: Any) -> Frames

This model's result attributes in the record's long form.

Keyed by attribute, each frame in the long schema - name, the dim columns, value - the same shape Record.outputs presents, so results go straight to write_record or to set(..., kind="outputs"). One frame spans every component type, needing no type column.

Narwhals frames, so the seam names no one dataframe library, and lazy so an implementation may fetch a result attribute on demand rather than materialising every one.

Notes

• the long schema
• entity is unique across types

Source code in src/datarecord/tools/base.py

def results(self, model: Any) -> Frames:
    """This model's result attributes in the record's long form.

    Keyed by attribute, each frame in the long schema - `name`, the dim
    columns, `value` - the same shape `Record.outputs` presents, so results
    go straight to `write_record` or to `set(..., kind="outputs")`. One frame
    spans every component type, needing no type column.

    Narwhals frames, so the seam names no one dataframe library, and lazy so
    an implementation may fetch a result attribute on demand rather than
    materialising every one.

    Notes
    -----
    - [the long schema](https://energy-models.github.io/datarecord/design/format/#the-long-schema)
    - [entity is unique across types](https://energy-models.github.io/datarecord/design/format/#entity-is-unique-across-types)
    """
    ...

Requirements dataclass

Requirements(
    dims: frozenset[str] = frozenset(),
    entity_types: frozenset[str] = frozenset(),
    attributes: frozenset[tuple[str, str]] = frozenset(),
    unsupported_keys: frozenset[
        tuple[str, str]
    ] = frozenset(),
    unsupported_values: frozenset[
        tuple[str, str]
    ] = frozenset(),
    names: frozenset[str] = frozenset(),
)

What a tool needs from a record, and what a given record fails to supply.

Parameters:

Name	Type	Description	Default
dims	frozenset of str	Dims the tool cannot build a model without.	frozenset()
entity_types	frozenset of str	Component types the tool requires the record to define members for.	frozenset()
attributes	frozenset of tuple of (str, str)	(component_type, attribute) pairs the tool requires a value for. Named in the record's vocabulary, not the tool's, so a caller can act on them: an attribute the tool renames or computes (Schema) is reported by the source attribute it is missing.	frozenset()
unsupported_keys	frozenset of tuple of (str, str)	(key, dim) pairs the schema declares that this tool cannot honour; key is "input_key", the fold's unit of ownership. The record layer trusts every declared key, so this is a tool's verdict on the record it was handed, not a schema rejection.	frozenset()
unsupported_values	frozenset of tuple of (str, str)	(component_type, attribute) pairs whose stored shape this tool cannot represent, as opposed to a value it is missing - a piecewise-linear attribute where the tool takes a scalar.	frozenset()
names	frozenset of str	Names two of the framework's components claim, which a record scopes across every type.	frozenset()

Notes

• wide and long rows
• entity is unique across types
• consuming a record

describe

describe() -> str

A one-line summary, for an error message or a log line.

Source code in src/datarecord/tools/base.py

def describe(self) -> str:
    """A one-line summary, for an error message or a log line."""
    parts = []
    if self.dims:
        parts.append(f"dims {sorted(self.dims)}")
    if self.entity_types:
        parts.append(f"component types {sorted(self.entity_types)}")
    if self.attributes:
        parts.append(f"attributes {sorted(self.attributes)}")
    if self.unsupported_keys:
        unsupported = ", ".join(
            f"{dim} as {key}" for key, dim in sorted(self.unsupported_keys)
        )
        parts.append(f"unsupported keys ({unsupported})")
    if self.unsupported_values:
        parts.append(
            f"piecewise-linear attributes {sorted(self.unsupported_values)}"
        )
    if self.names:
        parts.append(
            f"names claimed by more than one component type "
            f"{sorted(self.names)} (https://energy-models.github.io/datarecord/design/format/#entity-is-unique-across-types)"
        )
    return ", ".join(parts) if parts else "nothing"

Schema dataclass

Schema(attrs: dict[str, tuple[Attr, ...]] = dict())

A tool's attribute mapping against the record's vocabulary.

Only differing attributes need an entry; anything unlisted is read under the same name, so an empty Schema is the identity. Attr.source names attributes in the record's vocabulary.

Parameters:

Name	Type	Description	Default
attrs	dict of str to tuple of Attr	Per component type, the attributes that are renamed or computed.	dict()

Notes

• AttributeSpec
• consuming a record

attr

attr(ctype: str, name: str) -> Attr

ctype's mapping for name, or the identity when it has none.

Source code in src/datarecord/tools/base.py

def attr(self, ctype: str, name: str) -> Attr:
    """`ctype`'s mapping for `name`, or the identity when it has none."""
    for a in self.attrs.get(ctype, ()):
        if a.name == name:
            return a
    return Attr(name=name, source=(name,))

sources

sources(ctype: str, name: str) -> tuple[str, ...]

Which record attribute(s) ctype's name needs; itself, if unmapped.

What verify checks resolvability against: a renamed or computed attribute is satisfied by its sources, not by its own name.

Source code in src/datarecord/tools/base.py

def sources(self, ctype: str, name: str) -> tuple[str, ...]:
    """Which record attribute(s) `ctype`'s `name` needs; itself, if unmapped.

    What `verify` checks resolvability against: a renamed or computed
    attribute is satisfied by its sources, not by its own name.
    """
    return self.attr(ctype, name).source

resolve

resolve(
    record: RecordLike, ctype: str, name: str
) -> DuckDBPyRelation

ctype's name as a long relation over record, mapping applied.

Source code in src/datarecord/tools/base.py

def resolve(self, record: RecordLike, ctype: str, name: str) -> DuckDBPyRelation:
    """`ctype`'s `name` as a long relation over `record`, mapping applied."""
    return self.attr(ctype, name).resolve(record)

Attr dataclass

Attr(
    name: str,
    source: tuple[str, ...],
    compute: Callable[..., DuckDBPyRelation] | None = None,
)

One tool attribute and the record attribute(s) it is built from.

The vocabulary seam: a rename, or a value computed from several, declared rather than open-coded in a build.

Parameters:

Name	Type	Description	Default
name	str	The tool's name for the attribute.	required
source	tuple of str	Record attribute name(s) it is read from. A rename has one.	required
compute	callable	Maps one resolved long relation per source, in order, to this attribute's long relation; None for a plain rename, which requires exactly one source. A relation in, a relation out, so the plan stays unmaterialised and a computed attribute needs no special case downstream.	None

Notes

• consuming a record

resolve

resolve(record: RecordLike) -> DuckDBPyRelation

This attribute's long relation, read through the Record interface.

The record rather than the record, so a tool builds from any backing.

Notes

• the Record protocol

Source code in src/datarecord/tools/base.py

def resolve(self, record: RecordLike) -> DuckDBPyRelation:
    """This attribute's long relation, read through the `Record` interface.

    The record rather than the record, so a tool builds from any backing.

    Notes
    -----
    - [the Record protocol](https://energy-models.github.io/datarecord/design/record/)
    """
    rels = [to_relation(record.attributes[s]) for s in self.source]
    if self.compute is None:
        return rels[0]
    return self.compute(*rels)

UnsupportedRecordError

UnsupportedRecordError(tool: str, missing: Requirements)

Bases: ValueError

A record does not define everything the tool needs (Tool.verify).

Source code in src/datarecord/tools/base.py

def __init__(self, tool: str, missing: Requirements) -> None:
    super().__init__(
        f"record cannot build a {tool} model; missing: {missing.describe()}"
    )
    self.missing = missing

PyPSA

The PyPSA tool needs the optional extra: pip install datarecord[pypsa].

PyPSA module-attribute

PyPSA = PyPSATool()