Skip to content

Tools

Tool

Bases: Protocol

One modelling framework's view of a record.

Implementations are stateless (a module-level singleton is fine): every method takes the record or model it operates on. A structural type, for annotating code that takes any tool - not a dispatch table; tools are reached by importing them.

A Record rather than a record throughout, so a tool builds from a directory as readily as from an overlay and has no reason to know layering exists.

Notes

requires

requires(record: RecordLike) -> Requirements

What this tool needs from record to build a model.

Record-dependent, not a constant: which attributes are required follows the categories the record's labels have and its declared schema.

Source code in src/datarecord/tools/base.py
def requires(self, record: RecordLike) -> Requirements:
    """What this tool needs from `record` to build a model.

    Record-dependent, not a constant: which attributes are required
    follows the categories the record's labels have and its declared
    schema.
    """
    ...

verify

verify(record: RecordLike) -> Requirements

What record fails to supply; falsy when the record is usable.

Source code in src/datarecord/tools/base.py
def verify(self, record: RecordLike) -> Requirements:
    """What `record` fails to supply; falsy when the record is usable."""
    ...

build

build(record: RecordLike) -> Any

The tool's model object, built from the resolved record.

Source code in src/datarecord/tools/base.py
def build(self, record: RecordLike) -> Any:
    """The tool's model object, built from the resolved record."""
    ...

to_datarecord

to_datarecord(
    model: Any,
    con: DuckDBPyConnection,
    *,
    results_only: bool = False,
    base: RecordLike | None = None,
) -> RecordLike

model presented as a layer write_layer can persist.

The inverse of build. Framework-specific: undoing a framework's own shape is exactly what a tool knows and the record layer does not.

Parameters:

Name Type Description Default
con DuckDBPyConnection

The connection the relations are planned on - in practice ws.con of the workspace that will write them.

required
results_only bool

Present only what a solve produced - its result attributes and the intermediates they need - as a thin layer to write as a child of base.

False
base RecordLike | None

The record the layer is written over as a child: required with results_only.

None

Raises:

Type Description
ValueError

If base is missing with results_only.

Notes
Source code in src/datarecord/tools/base.py
def to_datarecord(
    self,
    model: Any,
    con: DuckDBPyConnection,
    *,
    results_only: bool = False,
    base: RecordLike | None = None,
) -> RecordLike:
    """`model` presented as a layer `write_layer` can persist.

    The inverse of `build`. Framework-specific: undoing a framework's own
    shape is exactly what a tool knows and the record layer does not.

    Parameters
    ----------
    con
        The connection the relations are planned on - in practice `ws.con` of
        the workspace that will write them.
    results_only
        Present only what a solve produced - its result attributes and the
        intermediates they need - as a thin layer to write as a child of
        `base`.
    base
        The record the layer is written over as a child: required with
        `results_only`.

    Raises
    ------
    ValueError
        If `base` is missing with `results_only`.

    Notes
    -----
    - [the record format](https://energy-models.github.io/datarecord/reference/format/)
    - [writing results](https://energy-models.github.io/datarecord/reference/tools/#writing-results)
    """
    ...

Requirements dataclass

Requirements(
    dims: frozenset[str] = frozenset(),
    categories: frozenset[str] = frozenset(),
    attributes: frozenset[tuple[str, str]] = frozenset(),
    unsupported_keys: frozenset[
        tuple[str, str]
    ] = frozenset(),
    unsupported_dims: frozenset[str] = frozenset(),
    unsupported_values: frozenset[
        tuple[str, str]
    ] = frozenset(),
    names: frozenset[str] = frozenset(),
    ignored: frozenset[tuple[str, str]] = frozenset(),
)

What a tool needs from a record, and what a given record fails to supply.

Parameters:

Name Type Description Default
dims frozenset of str

Dims the tool cannot build a model without.

frozenset()
categories frozenset of str

Categories the tool requires the record to define labels of, or, in a verify result, ones it has no counterpart for.

frozenset()
attributes frozenset of tuple of (str, str)

(category, attribute) pairs: an attribute the tool requires a value for on the labels of that category. Named in the record's vocabulary, not the tool's, so a caller can act on them: an attribute the tool renames or computes (Schema) is reported by the source attribute it is missing.

frozenset()
unsupported_keys frozenset of tuple of (str, str)

(key, dim) pairs the schema declares that this tool cannot honour; key is "owner_map_key", the fold's unit of ownership. The record layer trusts every declared key, so this is a tool's verdict on the record it was handed, not a schema rejection.

frozenset()
unsupported_dims frozenset of str

Dims the record declares that this tool has no place for in the model it builds.

frozenset()
unsupported_values frozenset of tuple of (str, str)

(category, attribute) pairs whose stored shape this tool cannot represent, as opposed to a value it is missing - a piecewise-linear attribute where the tool takes a scalar, or one whose rows differ along a dim the tool holds it without.

frozenset()
names frozenset of str

Labels the framework scopes per category and two of its categories claim, where a record has one axis row per label.

frozenset()
ignored frozenset of tuple of (str, str)

(category, attribute) pairs the record stores a value for that the tool's category does not take, so the build leaves it out. A report, not a failure: it leaves the result falsy and describe silent.

frozenset()
Notes

describe

describe() -> str

A one-line summary, for an error message or a log line.

Source code in src/datarecord/tools/base.py
def describe(self) -> str:
    """A one-line summary, for an error message or a log line."""
    parts = []
    if self.dims:
        parts.append(f"dims {sorted(self.dims)}")
    if self.categories:
        parts.append(f"categories {sorted(self.categories)}")
    if self.attributes:
        parts.append(f"attributes {sorted(self.attributes)}")
    if self.unsupported_keys:
        unsupported = ", ".join(
            f"{dim} as {key}" for key, dim in sorted(self.unsupported_keys)
        )
        parts.append(f"unsupported keys ({unsupported})")
    if self.unsupported_dims:
        parts.append(f"unsupported dims {sorted(self.unsupported_dims)}")
    if self.unsupported_values:
        parts.append(f"unsupported values {sorted(self.unsupported_values)}")
    if self.names:
        parts.append(
            f"labels claimed by more than one category "
            f"{sorted(self.names)} (https://energy-models.github.io/datarecord/reference/format/#labels)"
        )
    return ", ".join(parts) if parts else "nothing"

Schema dataclass

Schema(attrs: dict[str, tuple[Attr, ...]] = dict())

A tool's attribute mapping against the record's vocabulary.

Only differing attributes need an entry; anything unlisted is read under the same name, so an empty Schema is the identity. Attr.source names attributes in the record's vocabulary.

Parameters:

Name Type Description Default
attrs dict of str to tuple of Attr

Per entity type, the attributes that are renamed or computed.

dict()
Notes

attr

attr(ctype: str, name: str) -> Attr

ctype's mapping for name, or the identity when it has none.

Source code in src/datarecord/tools/base.py
def attr(self, ctype: str, name: str) -> Attr:
    """`ctype`'s mapping for `name`, or the identity when it has none."""
    for a in self.attrs.get(ctype, ()):
        if a.name == name:
            return a
    return Attr(name=name, source=(name,))

sources

sources(ctype: str, name: str) -> tuple[str, ...]

Which record attribute(s) ctype's name needs; itself, if unmapped.

What verify checks resolvability against: a renamed or computed attribute is satisfied by its sources, not by its own name.

Source code in src/datarecord/tools/base.py
def sources(self, ctype: str, name: str) -> tuple[str, ...]:
    """Which record attribute(s) `ctype`'s `name` needs; itself, if unmapped.

    What `verify` checks resolvability against: a renamed or computed
    attribute is satisfied by its sources, not by its own name.
    """
    return self.attr(ctype, name).source

resolve

resolve(
    record: RecordLike,
    ctype: str,
    name: str,
    labels: Mapping[str, DuckDBPyRelation],
    flags: Mapping[str, Flags],
    *,
    filter: bool = True,
) -> DuckDBPyRelation

ctype's name as a long relation over record, mapping applied and settled (Attr.resolve).

Source code in src/datarecord/tools/base.py
def resolve(
    self,
    record: RecordLike,
    ctype: str,
    name: str,
    labels: Mapping[str, DuckDBPyRelation],
    flags: Mapping[str, Flags],
    *,
    filter: bool = True,
) -> DuckDBPyRelation:
    """`ctype`'s `name` as a long relation over `record`, mapping applied and settled (`Attr.resolve`)."""
    return self.attr(ctype, name).resolve(record, labels, flags, filter=filter)

Attr dataclass

Attr(
    name: str,
    source: tuple[str, ...],
    compute: Callable[..., DuckDBPyRelation] | None = None,
)

One tool attribute and the record attribute(s) it is built from.

The vocabulary seam: a rename, or a value computed from several, declared rather than open-coded in a build.

Parameters:

Name Type Description Default
name str

The tool's name for the attribute.

required
source tuple of str

Record attribute name(s) it is read from. A rename has one.

required
compute callable

Maps one settled long relation per source (settle), in order, to this attribute's long relation; None for a plain rename, which requires exactly one source. A relation in, a relation out, so the plan stays unmaterialised and a computed attribute needs no special case downstream.

None
Notes

resolve

resolve(
    record: RecordLike,
    labels: Mapping[str, DuckDBPyRelation],
    flags: Mapping[str, Flags],
    *,
    filter: bool = True,
) -> DuckDBPyRelation

This attribute's long relation, each source settled (settle) before compute.

Parameters:

Name Type Description Default
flags Mapping[str, Flags]

Per source, its Flags as settle takes them.

required
filter bool

As settle takes it.

True
Notes
Source code in src/datarecord/tools/base.py
def resolve(
    self,
    record: RecordLike,
    labels: Mapping[str, DuckDBPyRelation],
    flags: Mapping[str, Flags],
    *,
    filter: bool = True,
) -> DuckDBPyRelation:
    """This attribute's long relation, each source settled (`settle`) before `compute`.

    Parameters
    ----------
    flags
        Per source, its `Flags` as `settle` takes them.
    filter
        As `settle` takes it.

    Notes
    -----
    - [the Record protocol](https://energy-models.github.io/datarecord/reference/record/)
    - [settling rows](https://energy-models.github.io/datarecord/reference/record/#settle)
    """
    rels = [
        settle(record.attributes[s], labels, flags[s], filter=filter)
        for s in self.source
    ]
    if self.compute is None:
        return rels[0]
    return self.compute(*rels)

UnsupportedRecordError

UnsupportedRecordError(tool: str, missing: Requirements)

Bases: ValueError

A record does not define everything the tool needs (Tool.verify).

Source code in src/datarecord/tools/base.py
def __init__(self, tool: str, missing: Requirements) -> None:
    super().__init__(
        f"record cannot build a {tool} model; missing: {missing.describe()}"
    )
    self.missing = missing

PyPSA

The PyPSA tool needs the optional extra: pip install datarecord[pypsa].

PyPSATool

Bases: Tool

Build a pypsa.Network from a record and read its results back.

component_types

component_types() -> frozenset[str]

The component types PyPSA's type registry knows.

Source code in src/datarecord/tools/pypsa.py
def component_types(self) -> frozenset[str]:
    """The component types PyPSA's type registry knows."""
    from pypsa.components.types import all_components

    return frozenset(c.name for c in all_components.values())

requires

requires(record: RecordLike) -> Requirements

PyPSA's axes, the record's component types PyPSA knows, and their required attributes by source.

Source code in src/datarecord/tools/pypsa.py
def requires(self, record: RecordLike) -> Requirements:
    """PyPSA's axes, the record's component types PyPSA knows, and their required attributes by source."""
    known = self.component_types()
    ctypes = {ct for ct in _components(record) if ct in known}
    return Requirements(
        dims=REQUIRED_DIMS,
        categories=frozenset(ctypes),
        attributes=frozenset(
            (ct, src)
            for ct in ctypes
            for attr in _required_attributes(ct)
            for src in self.schema.sources(ct, attr)
        ),
    )

verify

verify(record: RecordLike) -> Requirements

What this record, a selection narrowed as build narrows it, fails to supply for a PyPSA build; falsy if it is usable.

Reports PyPSA's axes the schema lacks, keys and dims of its own this tool cannot honour (and nothing else where it finds any), component types PyPSA does not know, required attributes (by source) with no row, property or declared default, a port's bus where a member has no connection there, curves and values a network would hold one of (_unheld), and in ignored stored attributes a type does not take, which build leaves out.

Raises:

Type Description
ValueError

If record is a selection build refuses (_narrowed).

Notes
Source code in src/datarecord/tools/pypsa.py
def verify(self, record: RecordLike) -> Requirements:
    """What this record, a selection narrowed as `build` narrows it, fails to supply for a PyPSA build; falsy if it is usable.

    Reports PyPSA's axes the schema lacks, keys and dims of its own this
    tool cannot honour (and nothing else where it finds any), component
    types PyPSA does not know, required attributes (by source) with no
    row, property or declared default, a port's bus where a member has no
    connection there, curves and values a network would hold one of
    (`_unheld`), and in `ignored` stored attributes a type does not take,
    which `build` leaves out.

    Raises
    ------
    ValueError
        If `record` is a selection `build` refuses (`_narrowed`).

    Notes
    -----
    - [AttributeSpec](https://energy-models.github.io/datarecord/reference/schema/#attributes)
    - [consuming a record](https://energy-models.github.io/datarecord/reference/tools/)
    - [selecting before a build](https://energy-models.github.io/datarecord/reference/tools/#selecting-before-a-build)
    """
    return self._verify(_narrowed(record))[0]

build

build(record: RecordLike) -> Network

The resolved network.

A selection (Record.select) builds the network of its labels, selected once more as PyPSA holds it (_narrowed): a squeezed period is a single-period network without investment_period, a squeezed scenario a deterministic one, and kept scenarios' weights are renormalised. A record with a dim of its own is refused, as PyPSACollection builds it, and a selection at one label of that dim builds the member there.

Raises:

Type Description
ValueError

If record is a selection squeezing a dim but period, scenario or a dim of its own, holding no snapshot, or keeping scenarios weighing nothing in all, or if an attribute's rows name several labels of a dim PyPSA holds it without (_unheld).

UnsupportedRecordError

If verify reports anything missing, a dim of the record's own included.

Notes
Source code in src/datarecord/tools/pypsa.py
def build(self, record: RecordLike) -> pypsa.Network:
    """The resolved network.

    A selection (`Record.select`) builds the network of its labels,
    selected once more as PyPSA holds it (`_narrowed`): a squeezed
    `period` is a single-period network without `investment_period`, a
    squeezed `scenario` a deterministic one, and kept scenarios' weights
    are renormalised. A record with a dim of its own is refused, as
    `PyPSACollection` builds it, and a selection at one label of that dim
    builds the member there.

    Raises
    ------
    ValueError
        If `record` is a selection squeezing a dim but `period`,
        `scenario` or a dim of its own, holding no snapshot, or keeping
        scenarios weighing nothing in all, or if an attribute's rows name
        several labels of a dim PyPSA holds it without (`_unheld`).
    UnsupportedRecordError
        If `verify` reports anything missing, a dim of the record's own included.

    Notes
    -----
    - [consuming a record](https://energy-models.github.io/datarecord/reference/tools/)
    - [the broadcast rule](https://energy-models.github.io/datarecord/reference/data-model/#the-broadcast-rule)
    - [selecting before a build](https://energy-models.github.io/datarecord/reference/tools/#selecting-before-a-build)
    """
    record = _narrowed(record)
    flagged = record.flags()
    if unheld := _unheld(record, flagged):
        over = ", ".join(f"`{attr}` over {dims}" for attr, dims in unheld.items())
        msg = (
            f"A network holds one value of {over}, where the record holds "
            "several labels its rows name: select one label as a list of "
            "one, or build a network per `period` with `PyPSACollection`"
        )
        raise ValueError(msg)
    missing, read = self._verify(record)
    if missing:
        raise UnsupportedRecordError(self.name, missing)

    con = record.con
    schema = record.schema
    shape = NetworkShape(schema, record.dims)
    n = _new_network(record, shape)

    periodless = {_INVESTMENT_PERIOD} if PERIOD in _squeezed(record) else set()
    over_entity = [
        a
        for a in schema.attributes
        if schema.addresses(a, ENTITY) and a not in periodless
    ]
    settled = _one_labelled(record)
    if shape.stochastic:
        settled[SCENARIO] = record.dims[SCENARIO]
    tuples = record.dims.get(CONNECTION)
    for ctype in _AXIS_TYPES:
        self._add_labels(n, record, ctype, flagged, shape, settled, con)
    _attach_sub_networks(n)
    for ctype, members in read.items():
        static = members.members
        if static is None:
            continue
        labels = {**settled, ENTITY: static}
        if tuples is not None:
            labels[CONNECTION] = _tuples_of(tuples, static)
        if not shape.stochastic and SCENARIO in static.columns:
            static = static.project(star(exclude=[SCENARIO]))
        static = static.project(star(), _POS)
        connections = members.connections
        static = _collapse_connections(static, connections, ctype, con)
        attributes = {}
        carried = _taken(schema, ctype, over_entity)
        for attr, flags in flagged.items():
            if attr not in carried:
                continue
            long = unpacked(
                schema,
                CONNECTION,
                self.schema.resolve(
                    record, ctype, attr, labels, flagged, filter=False
                ),
            )
            if CONNECTION in schema.attributes[attr].dims:
                for column, rows in _port_columns(
                    long, attr, connections, ctype, con
                ).items():
                    attributes[column] = (rows, flags)
            else:
                attributes[_INTERNAL_COLUMNS.get(attr, attr)] = (long, flags)

        _add_component_type(n, ctype, static, attributes, shape, con)

    _add_shapes(n, record)
    _broadcast_standard_types(n)
    return n

to_datarecord

to_datarecord(
    model: Network,
    con: DuckDBPyConnection,
    *,
    results_only: bool = False,
    base: RecordLike | None = None,
) -> RecordLike

Present a Network as the Record write_layer persists.

The inverse of build: static and series values become long rows on con, port columns connection rows carrying a role, buses and carriers the bus and carrier axes in PyPSA's order, "" NULL, and each shape the shape_{type} property of the label it outlines. Outputs are attributes like any other. A static value is left out where it repeats what the record resolves to without it: over base, base's value, else the default; over nothing, the default.

With results_only the record is a thin layer over base: the outputs, at their default too, base's output rows at the scenarios, buses, entities and connections the network holds that it does not write again restated as NULL, and this solve's SubNetworks as the sub_network axis, unless it holds fewer buses or entities than base, whose SubNetworks it then leaves. A series is written at the network's snapshots, base's standing at the others; over a multi-period base, a value over no snapshot is written for the entities active in the network's periods alone. base may be the selection the network was built from (Record.select): the layer is then over its record, each row at the labels it squeezes, one period squeezed writing at that period. Over the selection at one label of the record's dim of its own, the network is that member, written as PyPSACollection writes a collection of it alone.

Raises:

Type Description
ValueError

If base is missing with results_only. If a network over a multi-period base whose investment periods it does not carry is written without results_only, or as results carries none and base squeezes none. If a full export is written over a selection or over a record with a dim of its own, or results over a record keeping one. If a deterministic network's results are written over a base holding scenarios, results over no scenario of fewer scenarios than the record holds, or a network's results holding a snapshot, bus or entity base lacks. On a full export: if types of one dim carry a custom property with different dtypes, a custom column is named shape_... or like snapshot, timestep, period, scenario, entity or connection, or a Shape carries one; if a shape lacks component, idx or type, two share all three, one on a PyPSA type names no member the network writes, or one's component names a dim PyPSA's schema declares; or if a custom text column named like a dim holds a value naming no label of it.

UnsupportedRecordError

If two entity types share a name.

Notes
Source code in src/datarecord/tools/pypsa.py
def to_datarecord(
    self,
    model: pypsa.Network,
    con: DuckDBPyConnection,
    *,
    results_only: bool = False,
    base: RecordLike | None = None,
) -> RecordLike:
    """Present a `Network` as the `Record` `write_layer` persists.

    The inverse of `build`: static and series values become long rows on
    `con`, port columns connection rows carrying a `role`, buses and
    carriers the `bus` and `carrier` axes in PyPSA's order, `""` NULL,
    and each shape the `shape_{type}` property of the label it outlines.
    Outputs are attributes like any other. A static value is left out
    where it repeats what the record resolves to without it: over `base`,
    `base`'s value, else the default; over nothing, the default.

    With `results_only` the record is a thin layer over `base`: the
    outputs, at their default too, `base`'s output rows at the scenarios,
    buses, entities and connections the network holds that it does not
    write again restated as NULL, and this solve's SubNetworks as the
    `sub_network` axis, unless it holds fewer buses or entities than
    `base`, whose SubNetworks it then leaves. A series is written at the
    network's snapshots, `base`'s standing at the others; over a
    multi-period `base`, a value over no snapshot is written for the
    entities active in the network's periods alone. `base` may be the
    selection the network was built from (`Record.select`): the layer is
    then over its record, each row at the labels it squeezes, one period
    squeezed writing at that period. Over the selection at one label of
    the record's dim of its own, the network is that member, written as
    `PyPSACollection` writes a collection of it alone.

    Raises
    ------
    ValueError
        If `base` is missing with `results_only`. If a network over a
        multi-period `base` whose investment periods it does not carry is
        written without `results_only`, or as results carries none and
        `base` squeezes none. If a full export is written over a
        selection or over a record with a dim of its own, or results over
        a record keeping one. If a deterministic network's results are
        written over a `base` holding scenarios, results over no
        `scenario` of fewer scenarios than the record holds, or a
        network's results holding a snapshot, bus or entity `base` lacks.
        On a full export: if types of one dim carry a custom property
        with different dtypes, a custom column is named `shape_...` or
        like `snapshot`, `timestep`, `period`, `scenario`, `entity` or
        `connection`, or a Shape carries one; if a shape lacks
        `component`, `idx` or `type`, two share all three, one on a PyPSA
        type names no member the network writes, or one's `component`
        names a dim PyPSA's schema declares; or if a custom text column
        named like a dim holds a value naming no label of it.
    UnsupportedRecordError
        If two entity types share a name.

    Notes
    -----
    - [ports](https://energy-models.github.io/datarecord/reference/tools/#ports)
    - [a label is unique within its dim](https://energy-models.github.io/datarecord/reference/format/#labels)
    - [writing a whole record](https://energy-models.github.io/datarecord/reference/layers/)
    - [writing results](https://energy-models.github.io/datarecord/reference/tools/#writing-results)
    - [a collection](https://energy-models.github.io/datarecord/reference/tools/#a-collection)
    """
    _require_base(results_only, base)
    if not results_only and isinstance(base, Selection):
        msg = (
            "A full export is written over a record, not over a selection "
            "of one: its rows would miss the labels the selection squeezes"
        )
        raise ValueError(msg)
    own = [] if base is None else _own_dims(_root(base).schema)
    squeezed: dict[str, Any] = {}
    if isinstance(base, Selection):
        squeezed = {d: base.labels[d] for d in _squeezed(base)}
    if own and (not results_only or len(own) > 1 or own[0] not in squeezed):
        msg = (
            f"A network is not written over a record with a dim of its own, "
            f"{own}, where its rows would reach every label: write its "
            f"results over the selection it was built from, "
            f"`record.select({own[0]}=label)`, or the collection with "
            "`PyPSACollection`"
        )
        raise ValueError(msg)
    if not results_only and base is not None and _multiperiod(base):
        periods = _axis_labels(base, PERIOD)
        if list(model.investment_periods) != periods:
            msg = (
                f"A network of investment periods {list(model.investment_periods)} "
                f"is written over a `base` of {periods} with "
                "`results_only=True` only"
            )
            raise ValueError(msg)
    source = self._source(model, con, base, results_only)
    if not own:
        return source
    (dim,) = own
    return _StackedSource.of({squeezed[dim]: source}, dim, con, base)

PyPSA module-attribute

PyPSA = PyPSATool()

PyPSACollection module-attribute

PyPSACollection = PyPSACollectionTool(PyPSA)