Skip to content

Record

Record dataclass

Record(
    revision_id: UUID,
    sources: list[LayerSource],
    workspace: Workspace,
)

A record's layers folded last-writer-wins, read as DuckDB relations.

One class for every backing, because a fold over one source degenerates to a scan of it - so a plain parquet directory is read as the one layer it is (Workspace.at) rather than by a second code path. RecordLike is the protocol a caller annotates against; this is the class a caller holds.

Nothing up to the last frozen source can go stale, so the fold over them is kept in shared tables. A source after it is folded onto those tables per read, so a staged edit reads back with nothing to invalidate. The record holds those tables from construction until close(), or until it and every relation taken from it are collected.

Parameters:

Name Type Description Default
revision_id UUID

The node whose frozen layers are folded, which names the shared tables their fold is kept in.

required
sources list of LayerSource

Root first, ending in the layer being resolved, truncated at the deepest materialised ancestor (sources_to_read). A WorkingRecord's staged rows are one more source on the end.

required
workspace Workspace

Where the calling thread's cursor comes from.

required

Raises:

Type Description
ValueError

On a read once the record is closed.

Notes

closed property

closed: bool

Whether close() ran, after which the record refuses every read.

schema property

schema: Schema

The schema the record is read under: its last source's.

con property

con: DuckDBPyConnection

The cursor of the calling thread, which the relations of this record are planned on.

Notes

frozen property

frozen: bool

Whether every source is frozen, so a fold over them cannot go stale.

False exactly when the last source is a staging area.

Notes

close

close() -> None

Release the tables this record reads, each dropped once no other record or registration holds it.

Closing again does nothing. A closed record refuses every read with a ValueError, and a relation taken from it before fails once its tables are dropped.

Notes
Source code in src/datarecord/resolve/record.py
def close(self) -> None:
    """Release the tables this record reads, each dropped once no other record or registration holds it.

    Closing again does nothing. A closed record refuses every read with a
    `ValueError`, and a relation taken from it before fails once its
    tables are dropped.

    Notes
    -----
    - [the database's schemas](https://energy-models.github.io/datarecord/reference/store/#the-database)
    """
    if self._release.detach() is not None:
        self.workspace._holds.release(self._holder)

dims

dims() -> LazyRelations

Every dim the schema declares, an empty axis where no layer holds one.

Source code in src/datarecord/resolve/record.py
@_stable_cache
def dims(self) -> LazyRelations:
    """Every dim the schema declares, an empty axis where no layer holds one."""
    return LazyRelations(
        tuple(self.schema.dimensions), partial(self._handed, self._axis_frame)
    )

properties

properties() -> LazyRelations

Every subject the schema declares properties of, an empty axis where no layer holds one.

Source code in src/datarecord/resolve/record.py
@_stable_cache
def properties(self) -> LazyRelations:
    """Every subject the schema declares properties of, an empty axis where no layer holds one."""
    return LazyRelations(
        tuple(self.schema.properties), partial(self._handed, self._property_frame)
    )

attributes

attributes() -> LazyRelations

Every attribute the schema declares, an empty relation where no layer holds a row.

Source code in src/datarecord/resolve/record.py
@_stable_cache
def attributes(self) -> LazyRelations:
    """Every attribute the schema declares, an empty relation where no layer holds a row."""
    return LazyRelations(
        tuple(self.schema.attributes), partial(self._handed, self._relation)
    )

flags

flags(
    **categories: str | Sequence[str],
) -> dict[str, Flags]

Straight off the owner map, which folded these in for free.

Notes
Source code in src/datarecord/resolve/record.py
def flags(self, **categories: str | Sequence[str]) -> dict[str, Flags]:
    """Straight off the owner map, which folded these in for free.

    Notes
    -----
    - [the owner map](https://energy-models.github.io/datarecord/reference/resolution/#the-owner-map)
    """
    return self._fold.flags(**categories)

select

select(**labels: Any) -> Selection

This record at the labels named, as datarecord.select takes them.

Raises:

Type Description
ValueError

As datarecord.select does, naming the keyword.

Notes
Source code in src/datarecord/resolve/record.py
def select(self, **labels: Any) -> Selection:
    """This record at the labels named, as `datarecord.select` takes them.

    Raises
    ------
    ValueError
        As `datarecord.select` does, naming the keyword.

    Notes
    -----
    - [selecting](https://energy-models.github.io/datarecord/reference/selecting/)
    """
    return select(self, **labels)

register

register(name: str) -> None

Create the schema name with a view per dim, property subject and attribute.

The views sit in name.dims, name.properties and name.attributes of the workspace's database, as in datarecord.name.dims.bus, each reading as its relation here; over a WorkingRecord they follow every edit. They hold the tables the record reads, its folds and a WorkingRecord's staging tables, until unregister(name), whether the record is closed or not. name is listed in Workspace.registered().

Raises:

Type Description
CatalogException

If the schema name exists in the workspace's database, as _fold and _staged always do.

TransactionException

Where the cursor already holds an open transaction, or another cursor registers name at the same time.

Notes
Source code in src/datarecord/resolve/record.py
def register(self, name: str) -> None:
    """Create the schema `name` with a view per dim, property subject and attribute.

    The views sit in `name.dims`, `name.properties` and `name.attributes` of
    the workspace's database, as in `datarecord.name.dims.bus`, each reading as
    its relation here; over a `WorkingRecord` they follow every edit. They hold
    the tables the record reads, its folds and a `WorkingRecord`'s staging
    tables, until `unregister(name)`, whether the record is closed or not.
    `name` is listed in `Workspace.registered()`.

    Raises
    ------
    duckdb.CatalogException
        If the schema `name` exists in the workspace's database, as
        `_fold` and `_staged` always do.
    duckdb.TransactionException
        Where the cursor already holds an open transaction, or another
        cursor registers `name` at the same time.

    Notes
    -----
    - [registered views](https://energy-models.github.io/datarecord/reference/format/#registered-views)
    """
    ws = self.workspace
    kinds = {
        "dims": self.dims,
        "properties": self.properties,
        "attributes": self.attributes,
    }
    rendered = {
        (ws.name, name, kind, view): _render(rel, literals=True, wrap_text=True)[0]
        for kind, relations in kinds.items()
        for view, rel in relations.items()
    }
    con = self.con
    holds = ws._holds
    with transaction(
        con,
        "register",
        "https://energy-models.github.io/datarecord/reference/format/#registered-views",
    ) as on_rollback:
        con.execute(f"CREATE SCHEMA {col(ws.name, name)}")
        holds.hold(name, self._kept_schemas)
        on_rollback(partial(holds.release, name))
        for kind in kinds:
            con.execute(f"CREATE SCHEMA {col(ws.name, name, kind)}")
        for path, query in rendered.items():
            con._create_view(path, query)
        con.execute(
            f"INSERT INTO {ws.qualified('registered')} (name, revision, working, held)"
            " VALUES (?, ?, ?, ?)",
            [
                name,
                self.revision_id,
                not self.frozen,
                sorted(self._kept_schemas),
            ],
        )

RecordLike

Bases: Protocol

What a record answers, however it is backed, as DuckDB relations.

Read-only: writing is store.commit(source, parent=...) or ws.write(source, uri), whose writer reads a LayerData rather than this - a framework's own RecordLike reaches it through an adapter.

Notes

schema property

schema: Schema

The record's schema: its dims, its attributes, its patch granularity.

Notes

con property

con: DuckDBPyConnection

The connection every relation here is planned on, the calling thread's.

Notes

dims property

dims: Relations

Axis relations keyed by dim, each holding only the dim's label column.

A multi-dim's label column is its struct column, a field per coordinate. Properties are in properties, attributes in attributes.

Notes

properties property

properties: Relations

Per dim declaring properties, its key and their stored values, no default applied.

Notes

attributes property

attributes: Relations

Long attribute relations, keyed by attribute name - one per file.

A row carries its coordinates alone; select by category through properties.

Notes

flags

flags(
    **categories: str | Sequence[str],
) -> dict[str, Flags]

Every attribute with rows, mapped to the shape its rows take.

Given categories of one dim, only rows reaching a label among them count, and only attributes those labels carry.

Raises:

Type Description
ValueError

If the keywords categorize no dim or several.

Notes
Source code in src/datarecord/protocols.py
def flags(self, **categories: str | Sequence[str]) -> dict[str, Flags]:
    """Every attribute with rows, mapped to the shape its rows take.

    Given `categories` of one dim, only rows reaching a label among them
    count, and only attributes those labels carry.

    Raises
    ------
    ValueError
        If the keywords categorize no dim or several.

    Notes
    -----
    - [Flags](https://energy-models.github.io/datarecord/reference/record/#flags)
    """
    ...

Relations module-attribute

Relations = Mapping[str, DuckDBPyRelation]

What a Record hands over: named relations, each an unmaterialised plan.

The Mapping ABC, so a plain dict satisfies it as fully as LazyRelations does.

Notes

LazyRelations

LazyRelations(
    keys: tuple[str, ...],
    build: Callable[[str], DuckDBPyRelation],
)

Bases: Mapping[str, DuckDBPyRelation]

A Relations whose values are built on __getitem__, not up front.

For a backing where building a relation is itself I/O: read_parquet reads the footer to bind the schema, a round trip per file against a remote record.

Parameters:

Name Type Description Default
keys tuple[str, ...]

Every key this mapping holds, in iteration order. Listing them must be cheap: nothing here builds a relation.

required
build Callable[[str], DuckDBPyRelation]

Called with one key to produce its relation, once per __getitem__; memoise inside build if repeated lookups matter.

required
Notes
Source code in src/datarecord/protocols.py
def __init__(self, keys: tuple[str, ...], build: Callable[[str], DuckDBPyRelation]):
    self._keys = keys
    self._build = build

Flags dataclass

Flags(
    varies: frozenset[str],
    broadcast: frozenset[str],
    breakpoints: bool = False,
)

Which axes an attribute's rows actually use.

The two sets are not complements: a dim in both means the rows disagree, which is the instruction to use both containers rather than an ambiguity to resolve. varies | broadcast is the test for whether an attribute touches a dim at all.

Parameters:

Name Type Description Default
varies frozenset[str]

Dims some row of this attribute sets.

required
broadcast frozenset[str]

Dims some row leaves NULL, i.e. "all values of that dim".

required
breakpoints bool

Whether any row carries a breakpoint. Not a dim.

False
Notes

select

select(record: RecordLike, /, **labels: Any) -> Selection

record at the labels named, as xarray's sel takes them.

A keyword names a dim; one naming a multi-dim's coordinate that is no dim narrows its tuples, and any other names a property by its bare name and narrows its subject to the labels whose value, its default applied, is selected. One label squeezes its dim. A list, a slice (both ends inclusive, None open) or a partial date string on a datetime axis keeps it. Every label referring to a dropped one goes with it, through properties and multi-dim tuples, until nothing more goes. Over a Selection, its keywords are merged and its base selected.

Returns:

Type Description
Selection

The view, which reads the axes it matched and propagated on as they were at the call, and plans every attribute on the base's connection.

Raises:

Type Description
ValueError

Naming the keyword: one naming no dim, coordinate or property, a multi-dim, a coordinate of several multi-dims or a property several subjects declare; a label the axis lacks; an empty list; a slice with a step, of an axis neither numeric nor datetime, with an end the axis cannot compare or covering no label; a squeezed label another keyword's selection drops; a squeeze folding a category into the default a kept label NULL on it does not take; squeezing every dim of a record with attributes.

Notes
Source code in src/datarecord/select.py
def select(record: RecordLike, /, **labels: Any) -> Selection:
    """`record` at the labels named, as xarray's `sel` takes them.

    A keyword names a dim; one naming a multi-dim's coordinate that is no dim
    narrows its tuples, and any other names a property by its bare name and
    narrows its subject to the labels whose value, its default applied, is
    selected. One label squeezes its dim. A list, a slice (both ends
    inclusive, `None` open) or a partial date string on a datetime axis keeps
    it. Every label referring to a dropped one goes with it, through
    properties and multi-dim tuples, until nothing more goes. Over a
    `Selection`, its keywords are merged and its base selected.

    Returns
    -------
    Selection
        The view, which reads the axes it matched and propagated on as they
        were at the call, and plans every attribute on the base's connection.

    Raises
    ------
    ValueError
        Naming the keyword: one naming no dim, coordinate or property, a
        multi-dim, a coordinate of several multi-dims or a property several
        subjects declare; a label the axis lacks; an empty list; a slice with
        a step, of an axis neither numeric nor datetime, with an end the axis
        cannot compare or covering no label; a squeezed label another
        keyword's selection drops; a squeeze folding a category into the
        default a kept label NULL on it does not take; squeezing every dim of
        a record with attributes.

    Notes
    -----
    - [selecting](https://energy-models.github.io/datarecord/reference/selecting/)
    - [what goes with a dropped label](https://energy-models.github.io/datarecord/reference/selecting/#what-goes-with-a-dropped-label)
    """
    if isinstance(record, Selection):
        return record.select(**labels)
    schema = record.schema
    read = _Read(record)
    asked: dict[str, Any] = {}
    squeezed: dict[str, Any] = {}
    kept: dict[str, set[Any]] = {}
    for keyword, value in labels.items():
        if keyword in schema.multi_dims:
            raise _refusal(
                keyword,
                value,
                "names a multi-dim, whose tuples are selected through its coordinates",
            )
        if keyword in schema.dimensions:
            matched = _match(
                keyword, value, read.axis(keyword), _dtype(schema, keyword)
            )
            asked[keyword] = matched
            if not isinstance(matched, list):
                squeezed[keyword] = matched
            _narrow(kept, keyword, {_key(m) for m in _listed(matched)})
            continue
        subject, allowed, asked[keyword] = _narrowing(schema, read, keyword, value)
        _narrow(kept, subject, allowed)
    for m in schema.multi_dims:
        over = schema.over(m)
        if all(d in squeezed for d in over.values()):
            squeezed[m] = {c: squeezed[d] for c, d in over.items()}
            _narrow(kept, m, {_key(t) for t in read.axis(m)})
    if schema.attributes and schema.dimensions.keys() <= squeezed.keys():
        raise _refused(
            {k: v for k, v in asked.items() if k in squeezed},
            "squeezes every dim, and attributes over no axis are no record; "
            "keep one dim as a list of one",
        )
    _propagate(schema, read, kept)
    for keyword, label in asked.items():
        if keyword in squeezed and _key(label) not in kept[keyword]:
            raise _refusal(
                keyword,
                label,
                "is squeezed at a label the selection drops through another keyword",
            )
    _refuse_uncategorized(schema, read, squeezed, kept)
    narrowed = {
        d: numbered_relation(
            record.con,
            d,
            [label for label in read.axis(d) if _key(label) in keys],
            _dtype(schema, d),
            _AT,
        )
        for d, keys in kept.items()
        if d in squeezed or len(keys) < len(read.axis(d))
    }
    view = _schema(schema, squeezed, read.category(squeezed))
    return Selection(record, asked, view, squeezed, narrowed)

Selection dataclass

Selection(
    base: RecordLike,
    labels: Mapping[str, Any],
    schema: Schema,
    _squeezed: Mapping[str, Any],
    _kept: Mapping[str, DuckDBPyRelation],
)

A record at the labels select names, itself a RecordLike.

A dim given one label is squeezed out of schema; every other dim keeps the labels that refer to no label the selection drops. dims and properties keep the base's order, and every relation is a plan on the base's connection.

Attributes:

Name Type Description
base RecordLike

The record selected from, never a Selection.

labels Mapping[str, Any]

The keywords asked for, normalized: one label for a squeezed dim, else the list of labels matched.

schema Schema

The base's schema without the squeezed dims, their axes, their properties and every property referring to one.

Notes

con property

con: DuckDBPyConnection

The base's connection, which every relation here is planned on.

dims property

The axes of the dims left, each at its kept labels in axis order.

properties property

properties: LazyRelations

The stored properties left, at the kept labels of their subject.

attributes property

attributes: LazyRelations

Long rows at the kept labels or NULL there, each squeezed dim settled and dropped.

Notes

flags

flags(
    **categories: str | Sequence[str],
) -> dict[str, Flags]

The base's flags without the squeezed dims, overstating the view's rows.

Raises:

Type Description
ValueError

If the keywords categorize no dim of the view, or several.

Notes
Source code in src/datarecord/select.py
def flags(self, **categories: str | Sequence[str]) -> dict[str, Flags]:
    """The base's flags without the squeezed dims, overstating the view's rows.

    Raises
    ------
    ValueError
        If the keywords categorize no dim of the view, or several.

    Notes
    -----
    - [Flags](https://energy-models.github.io/datarecord/reference/record/#flags)
    """
    if categories:
        categorized_dim(self.schema, categories)
    gone = frozenset(self._squeezed)
    return {
        name: Flags(f.varies - gone, f.broadcast - gone, f.breakpoints)
        for name, f in self.base.flags(**categories).items()
    }

select

select(**labels: Any) -> Selection

Select base once more, labels replacing the keywords of the same name.

Raises:

Type Description
ValueError

As select does.

Source code in src/datarecord/select.py
def select(self, **labels: Any) -> Selection:
    """Select `base` once more, `labels` replacing the keywords of the same name.

    Raises
    ------
    ValueError
        As `select` does.
    """
    return select(self.base, **{**self.labels, **labels})

lift

lift(
    attribute: str, rel: DuckDBPyRelation
) -> DuckDBPyRelation

rel, rows of attribute over this view, with the squeezed labels put back.

Each squeezed dim the base declares attribute over is added as a column at its label, and each squeezed coordinate back into its multi-dim's struct; a NULL struct stays NULL. An attribute the base declares over no squeezed dim comes back as it came.

Raises:

Type Description
KeyError

If the base declares no attribute.

Notes
Source code in src/datarecord/select.py
def lift(self, attribute: str, rel: DuckDBPyRelation) -> DuckDBPyRelation:
    """`rel`, rows of `attribute` over this view, with the squeezed labels put back.

    Each squeezed dim the base declares `attribute` over is added as a
    column at its label, and each squeezed coordinate back into its
    multi-dim's struct; a NULL struct stays NULL. An attribute the base
    declares over no squeezed dim comes back as it came.

    Raises
    ------
    KeyError
        If the base declares no `attribute`.

    Notes
    -----
    - [lifting rows back](https://energy-models.github.io/datarecord/reference/selecting/#lift)
    """
    base = self.base.schema
    coordinates = base.coordinates_of(attribute)
    changed = [
        c
        for c in coordinates
        if c in self._squeezed or (c in rel.columns and self._shrunk(c))
    ]
    if not changed:
        return rel
    lifted: list[Expression] = []
    for c in coordinates:
        if c in self._squeezed:
            lifted.append(_literal(self._squeezed[c], self._type(c)).alias(c))
        elif c in changed:
            fields = {
                f: col(c)[f]
                if f in self.schema.over(c)
                else _literal(self._squeezed[d], self._type(d))
                for f, d in base.over(c).items()
            }
            lifted.append(_tuple(c, fields, self._type(c)))
        elif c in rel.columns:
            lifted.append(col(c))
    rest = [col(c) for c in rel.columns if c not in coordinates]
    return rel.project(*lifted, *rest)

settle

settle(
    rel: DuckDBPyRelation,
    labels: Mapping[str, DuckDBPyRelation],
    flags: Flags,
    *,
    filter: bool = True,
) -> DuckDBPyRelation

Settle the long rows of rel by the broadcast rule, one row per coordinate.

A row NULL on a coordinate in labels takes each label there, and a row naming a label outside labels, or reaching none, is dropped. At each coordinate the row naming every coordinate in labels that the other rows there name is kept. A coordinate outside labels is kept as written, a NULL there being one more label, and breakpoint is no coordinate, so a curve is kept or dropped whole.

Parameters:

Name Type Description Default
labels Mapping[str, DuckDBPyRelation]

Per coordinate, a relation holding its labels, or its tuples for a multi-dim, in a column of the coordinate's name.

required
flags Flags

The attribute's Flags, or flags overstating them; they only spare work, so overstating changes no value.

required
filter bool

False keeps a row naming a label outside labels as written, for a caller whose own joins on the labels drop it anyway.

True

Raises:

Type Description
Error

On fetch, where no row names every coordinate the rows reaching it name.

Notes
Source code in src/datarecord/settle.py
def settle(
    rel: DuckDBPyRelation,
    labels: Mapping[str, DuckDBPyRelation],
    flags: Flags,
    *,
    filter: bool = True,
) -> DuckDBPyRelation:
    """Settle the long rows of `rel` by the broadcast rule, one row per coordinate.

    A row NULL on a coordinate in `labels` takes each label there, and a row
    naming a label outside `labels`, or reaching none, is dropped. At each
    coordinate the row naming every coordinate in `labels` that the other rows
    there name is kept. A coordinate outside `labels` is kept as written, a
    NULL there being one more label, and `breakpoint` is no coordinate, so a
    curve is kept or dropped whole.

    Parameters
    ----------
    labels
        Per coordinate, a relation holding its labels, or its tuples for a
        multi-dim, in a column of the coordinate's name.
    flags
        The attribute's `Flags`, or flags overstating them; they only spare
        work, so overstating changes no value.
    filter
        False keeps a row naming a label outside `labels` as written, for a
        caller whose own joins on the labels drop it anyway.

    Raises
    ------
    duckdb.Error
        On fetch, where no row names every coordinate the rows reaching it name.

    Notes
    -----
    - [the broadcast rule](https://energy-models.github.io/datarecord/reference/data-model/#the-broadcast-rule)
    - [settling rows](https://energy-models.github.io/datarecord/reference/record/#settle)
    """
    over = [d for d in labels if d in rel.columns and (filter or d in flags.broadcast)]
    mixed = [d for d in over if d in flags.broadcast and d in flags.varies]
    if mixed:
        rel = rel.project(star(), named_mask(mixed).alias("_named"))
    for d in over:
        rows = rel.set_alias("_r")
        if d not in flags.broadcast:
            axis = labels[d].project(col(d)).set_alias("_ax")
            rel = rows.join(axis, col("_r", d) == col("_ax", d), how="semi")
        elif filter:
            rel = _broadcast_onto(rows, labels[d], d)
        else:
            rel = _broadcast_beside(rows, labels[d], d)
    if not mixed:
        return rel
    coordinates = [c for c in rel.columns if c not in ("_named", "breakpoint", "value")]
    coordinate = fn.concat_ws(
        lit(", "),
        *(
            fn.concat(
                lit(f"{c}="), coalesce(col(c).cast(sqltypes.VARCHAR), lit("NULL"))
            )
            for c in coordinates
        ),
    )
    crossed = fn.error(
        fn.concat(
            lit("two rows cross at "),
            coordinate,
            lit(
                " and no row reaching it names both, so neither is the more specific "
                "there (https://energy-models.github.io/datarecord/reference/data-model/#the-broadcast-rule)"
            ),
        )
    )
    return most_specific(rel, coordinates, crossed)