Skip to content

WorkingRecord

WorkingRecord

WorkingRecord(base: RecordLike)

Bases: Record

A Record whose last source is a staging area, plus the edit surface.

A Record in the type as well as in the fold: what it reads is the data with its pending edits applied, and it reads it by being one layer deeper than its base rather than by overlaying anything of its own. Every read member is inherited unchanged, so an edit reads back through the same code path a committed layer would.

Staged rows live in connection-scoped DuckDB tables, the only place a staged row exists: the reads fold them rather than holding a copy, so what is staged is asked of the reads themselves.

The source list is fixed at construction and never changes; a set changes what the last source's tables hold, not which sources there are. That is what lets the inherited members stay correct - they cache a key set only where the fold is stable, which a staged source makes it not.

Raises:

Type Description
ValueError

On a read, an edit or edits once the record is closed, and on an edit or edits once a WorkingRecord it is staged over is.

Notes

Stage edits over base, which must be a Record.

Raises:

Type Description
TypeError

For any other RecordLike, naming ws.write(source, uri), which gives one.

TransactionException

Where the cursor holds an open transaction, whose staging tables no other cursor could see to drop.

Notes
Source code in src/datarecord/mutable.py
def __init__(self, base: RecordLike) -> None:
    """Stage edits over `base`, which must be a `Record`.

    Raises
    ------
    TypeError
        For any other `RecordLike`, naming `ws.write(source, uri)`, which
        gives one.
    duckdb.TransactionException
        Where the cursor holds an open transaction, whose staging tables no
        other cursor could see to drop.

    Notes
    -----
    - [WorkingRecord](https://energy-models.github.io/datarecord/reference/editing/)
    """
    if not isinstance(base, Record):
        msg = (
            f"a `WorkingRecord` folds its staged rows over a `Record`'s layers, "
            f"and a {type(base).__name__} has none: write it first, "
            f"`WorkingRecord(ws.write(source, uri))`"
        )
        raise TypeError(msg)
    refuse_open_transaction(
        base.con,
        "WorkingRecord",
        "https://energy-models.github.io/datarecord/reference/editing/#transactions-and-staging",
    )
    # `object.__setattr__` throughout: the base is a frozen dataclass, and
    # these are set before `super().__init__` because the `StagedSource`
    # below reads them off `self`.
    object.__setattr__(self, "_layer_id", uuid4())
    object.__setattr__(self, "_staged", {})
    object.__setattr__(self, "_touched", set())
    object.__setattr__(self, "_dirty", set())
    if not isinstance(base, WorkingRecord):
        base = _Base(base.revision_id, base.sources, base.workspace)
    object.__setattr__(self, "_base", base)
    object.__setattr__(
        self,
        "_staging_schema",
        f"{col(base.workspace.name)}.{nested(STAGED, self._layer_id)}",
    )
    base.con.execute(f"CREATE SCHEMA {self._staging_schema}")
    super().__init__(
        base.revision_id,
        [*base.sources, StagedSource(self, self._layer_id)],
        base.workspace,
    )
    if isinstance(base, _Base):
        object.__setattr__(base, "_release", self._release)
        object.__setattr__(base, "_holder", self._holder)
    for kind, attribute in self._kinds():
        name = self._table(kind, attribute)
        create_table(self.con, name, self._shape(kind, attribute))
        self._staged[kind, attribute] = name
        self._fill(kind, attribute)
    shape = self._table(_OWNERS)
    empty = empty_relation(self.con, attribute=sqltypes.VARCHAR)
    create_table(
        self.con, shape, owner_keys(empty, self.schema, self._layer_id, ())
    )
    for attribute in self.schema.attributes:
        name = self._table(_OWNERS, attribute)
        self.con.execute(f"CREATE TABLE {name} AS FROM {shape}")
        self._staged[_OWNERS, attribute] = name
    self.con.execute(f"DROP TABLE {shape}")
    self._frozen_owners  # noqa: B018 - built outside any edit's transaction

edits property

edits: StagedSource

The staged layer alone: the delta against the base, as a LayerData.

What store.commit(w) writes, as a child of the revision this record was built over - only the edits, the fold resolving the rest from the parent. The record itself is the flattened whole, base plus edits, which is what a standalone ws.write(w, uri) wants.

Writing clears nothing: rollback() does, called or not.

Notes

set

set(attribute: str, value: Any, **coords: Any) -> None

Stage values of one attribute.

Every coordinate is a keyword - entity=, snapshot=, a multi-dim's coordinate such as bus= or a corridor's from= - since which exist is declared rather than fixed. A keyword holds one label, fixing that coordinate, or a list of labels the value is staged at each of. A coordinate no keyword names is NULL, which by the broadcast rule is every label of it. The call replaces every row of attribute staged earlier that its own rows cover.

value takes five forms:

  • a scalar, staged at every combination of the listed labels;
  • a sequence, aligned positionally to the one list keyword;
  • a mapping or a one-dimensional labelled series, keyed by labels of the series' index.name where that names a coordinate, else of the one coordinate no keyword names;
  • a long frame supplying its own keys, which keywords may complete with single labels.

An attribute's values are long rows, whatever its dims. A property - a category included - patches one column of existing axis rows: a mapping or series from label to value, a dim= keyword naming the labels, or a scalar for every label the axis has. A multi-dim's property takes a scalar, its coordinates as keywords selecting the tuples, none selecting every tuple. A coordinate is part of the key, which add and remove change, and is refused here.

A property is named subject.name - set("entity.carrier", ...) - and a plain name is an attribute.

A keyword naming no coordinate of attribute is a property of a dim it reaches - ctype="Generator", or a list of values - and selects the labels of that dim matching it on the record as staged, the keywords combining as an intersection; set writes one row per label, and stores nothing of the filter, so a label added later is not reached. A property of a multi-dim attribute is over - role="output" - selects its tuples the same way, narrowed by any keyword naming one of its coordinates. On a multi-dim's property, the keywords select its tuples as on an attribute over it: set("connection.note", v, role="output") patches the output connections, and ctype="Link" the connections of links.

Raises:

Type Description
KeyError

On an undeclared attribute or keyword, a label that its dim's axis, as staged, does not hold, a tuple that its multi-dim, as staged, does not hold, a property's label no layer wrote, or a value of a property that refers to a dim and is no label of it.

ValueError

On a coordinate named as subject.name, a keyword the attribute does not vary over, a sequence with no single list keyword to align to, keys whose coordinate is ambiguous, a label its dim does not list, a property keyword whose dim the attribute does not reach through exactly one coordinate, or that is given beside that coordinate's own keyword, a frame column the schema does not declare for attribute, or a frame column or label whose type does not fit its dtype, timestamps with a time zone among them.

Notes
Source code in src/datarecord/mutable.py
@_atomic
def set(
    self,
    attribute: str,
    value: Any,
    **coords: Any,
) -> None:
    """Stage values of one attribute.

    Every coordinate is a keyword - `entity=`, `snapshot=`, a multi-dim's
    coordinate such as `bus=` or a corridor's `from=` - since which exist is declared
    rather than fixed. A keyword holds one label, fixing that coordinate, or
    a list of labels the value is staged at each of. A coordinate no keyword
    names is NULL, which by the broadcast rule is every label of it. The
    call replaces every row of `attribute` staged earlier that its own
    rows cover.

    `value` takes five forms:

    - a scalar, staged at every combination of the listed labels;
    - a sequence, aligned positionally to the one list keyword;
    - a mapping or a one-dimensional labelled series, keyed by labels of
      the series' `index.name` where that names a coordinate, else of the
      one coordinate no keyword names;
    - a long frame supplying its own keys, which keywords may complete with
      single labels.

    An attribute's values are long rows, whatever its `dims`. A property -
    a category included - patches one column of existing axis rows: a
    mapping or series from label to value, a `dim=` keyword naming the
    labels, or a scalar for every label the axis has. A multi-dim's
    property takes a scalar, its coordinates as keywords selecting the
    tuples, none selecting every tuple. A coordinate is part of the key,
    which `add` and `remove` change, and is refused here.

    A property is named `subject.name` - `set("entity.carrier", ...)` -
    and a plain name is an attribute.

    A keyword naming no coordinate of `attribute` is a property of a dim it
    reaches - `ctype="Generator"`, or a list of values - and selects
    the labels of that dim matching it on the record as staged, the
    keywords combining as an intersection; `set` writes one row per label,
    and stores nothing of the filter, so a label added later is not reached.
    A property of a multi-dim `attribute` is over - `role="output"` -
    selects its tuples the same way, narrowed by any keyword naming one of
    its coordinates. On a multi-dim's property, the keywords select its
    tuples as on an attribute over it: `set("connection.note", v,
    role="output")` patches the output connections, and `ctype="Link"`
    the connections of links.

    Raises
    ------
    KeyError
        On an undeclared attribute or keyword, a label that its dim's
        axis, as staged, does not hold, a tuple that its multi-dim, as
        staged, does not hold, a property's label no layer wrote, or a
        value of a property that refers to a dim and is no
        label of it.
    ValueError
        On a coordinate named as `subject.name`, a keyword the attribute
        does not vary over, a sequence with no single list keyword to align to,
        keys whose coordinate is ambiguous, a label its dim does not
        list, a property keyword whose dim the attribute does not reach
        through exactly one coordinate, or that is given beside that
        coordinate's own keyword, a frame column the schema does not
        declare for `attribute`, or a frame column or label whose type
        does not fit its dtype, timestamps with a time zone among them.

    Notes
    -----
    - [the shape of an edit](https://energy-models.github.io/datarecord/reference/editing/#the-shape-of-an-edit)
    - [set](https://energy-models.github.io/datarecord/reference/editing/#set)
    - [validation](https://energy-models.github.io/datarecord/reference/editing/#validation)
    """
    keyed = isinstance(value, Mapping) or _series_index(value) is not None
    if isinstance(self.schema.resolve(attribute), tuple):
        self._set_dotted(attribute, value, coords)
        return
    coords, tuples = self._select(attribute, coords)
    listed = {c: list(v) for c, v in coords.items() if _is_labels(v)}

    if is_frame(value) and not keyed:
        if listed or tuples:
            msg = (
                f"`set({attribute!r}, <frame>)` supplies its own keys, so "
                f"keywords {sorted(listed)} may give one label each, not a "
                f"list, and none may select the tuples of {sorted(tuples)}"
            )
            raise ValueError(msg)
        call = f"`set({attribute!r}, <frame>)`"
        rel = as_relation(value, self.con)
        multi_dims = self.schema.multi_dims_of(attribute)
        self.schema.check_multi_dim_columns(multi_dims, rel, carrier=call)
        self.schema.refuse_partial_tuples(multi_dims, rel, carrier=call)
        self._validate_input(attribute, coords)
        table = self._ensure(_ATTRIBUTES, attribute)
        known = [
            *self._staged_coordinates(attribute),
            *_flat_coordinates(self.schema, attribute),
            "value",
        ]
        extra = sorted(set(rel.columns) - set(known))
        if extra:
            msg = (
                f"{call} was given columns {extra} the schema does not "
                f"declare for {attribute!r}; it takes {sorted(set(known))}"
            )
            raise ValueError(msg)
        rel = self.schema.require_takes(rel, call, attribute=attribute)
        self._insert_long(rel, table, attribute, set(rel.columns), coords)
        return

    self._validate_input(attribute, coords)

    if keyed:
        selected = [c for m in tuples for c in self.schema.over(m)]
        key = self._key_coordinate(attribute, value, [*coords, *selected])
        keys, values = normalise_value(value)
        dim = _flat_coordinates(self.schema, attribute)[key]
        self._require_typed(
            dim, cast("list[Any]", keys), f"`set({attribute!r}, ...)`"
        )
        column: dict[str, Sequence[Any]] = {key: cast("list[Any]", keys)}
    elif _is_labels(value):
        if len(listed) != 1:
            msg = (
                f"`set({attribute!r}, <sequence>)` aligns to exactly one list "
                f"keyword, and was given {sorted(listed) or 'none'}"
            )
            raise ValueError(msg)
        ((key, labels),) = listed.items()
        del listed[key]
        _, values = normalise_value(value, labels)
        column = {key: labels}
    else:
        values = [value]
        column = {}
    rel = self._columns_relation(
        {**column, "value": values},
        {
            **{c: self._coordinate_type(attribute, c) for c in column},
            "value": None,
        },
    )
    for c, labels in listed.items():
        rel = rel.cross(
            self._columns_relation(
                {c: labels}, {c: self._coordinate_type(attribute, c)}
            )
        )
    for chosen in tuples.values():
        rel = rel.cross(chosen)
    fixed = {c: v for c, v in coords.items() if not _is_labels(v)}
    self._insert_long(
        rel,
        self._ensure(_ATTRIBUTES, attribute),
        attribute,
        set(rel.columns),
        fixed,
    )

add

add(
    dim: str,
    labels: Sequence[Any]
    | Sequence[Mapping[str, Any]]
    | DuckDBPyRelation,
    **columns: Any,
) -> None

Stage labels of dim, each as one whole row of its axis file.

columns are dim's properties, each a scalar for every label or a sequence aligned to labels. A column no keyword names is NULL on these rows: add restates a label's row whole, replacing any this record staged for it. A label keeps its place in the axis, a label of the base removed here takes its base place back, and a new one is appended.

A multi-dim's labels are tuples, given as a frame holding the struct column named for the multi-dim, as dims[dim] reads it, or one with a column per coordinate instead, or as a list of mappings from coordinate to label. A frame may carry property columns too and takes a scalar keyword only.

Not a sequence of set calls: a label exists by virtue of its axis row, so adding one with no columns at all is an edit. An attribute's value, whatever its dims, goes through set.

Raises:

Type Description
KeyError

If the schema declares no dim, on a value of a property that refers to a dim and is no label of it, or on a tuple naming a label its coordinate's axis, as staged, does not hold.

TypeError

On a frame for a dim that is no multi-dim, or a value that is no frame DuckDB rescans.

ValueError

On a dim with one_of, whose axis is the schema's, an attribute, a column the axis file has no place for, a sequence of the wrong length, a property value its dim does not list, a label whose type does not fit the dtype of dim, a tuple missing a coordinate or naming NULL in one, or a frame carrying both the struct column and a coordinate column.

Notes
Source code in src/datarecord/mutable.py
@_atomic
def add(
    self,
    dim: str,
    labels: Sequence[Any] | Sequence[Mapping[str, Any]] | DuckDBPyRelation,
    **columns: Any,
) -> None:
    """Stage labels of `dim`, each as one whole row of its axis file.

    `columns` are `dim`'s properties, each a scalar for every label or a
    sequence aligned to `labels`. A column no keyword names is NULL on
    these rows: `add` restates a label's row whole, replacing any this
    record staged for it. A label keeps its place in the axis, a label of
    the base removed here takes its base place back, and a new one is
    appended.

    A multi-dim's labels are tuples, given as a frame holding the struct
    column named for the multi-dim, as `dims[dim]` reads it, or one with a
    column per coordinate instead, or as a list of mappings from
    coordinate to label. A frame may carry property columns too and takes
    a scalar keyword only.

    Not a sequence of `set` calls: a label exists by virtue of its axis row,
    so adding one with no columns at all is an edit. An attribute's value,
    whatever its `dims`, goes through `set`.

    Raises
    ------
    KeyError
        If the schema declares no `dim`, on a value of a property
        that refers to a dim and is no label of it, or on a tuple naming
        a label its coordinate's axis, as staged, does not hold.
    TypeError
        On a frame for a dim that is no multi-dim, or a value that is no
        frame DuckDB rescans.
    ValueError
        On a dim with `one_of`, whose axis is the schema's, an
        attribute, a column the axis file has no place for, a sequence of
        the wrong length, a property value its dim does not list, a label
        whose type does not fit the dtype of `dim`, a tuple missing a
        coordinate or naming NULL in one, or a frame carrying both the
        struct column and a coordinate column.

    Notes
    -----
    - [add / remove](https://energy-models.github.io/datarecord/reference/editing/#add-and-remove)
    - [where a value lives](https://energy-models.github.io/datarecord/reference/format/#where-a-value-lives)
    - [multi-dims](https://energy-models.github.io/datarecord/reference/data-model/#multi-dims)
    """
    if dim not in self.schema.dimensions:
        msg = f"the schema declares no dim {dim!r}"
        raise KeyError(msg)
    call = f"add({dim!r}, ...)"
    self._refuse_one_of(dim, f"`{call}`")
    if dim in self.schema.multi_dims:
        self._add_tuples(dim, labels, columns)
        return
    kind = f"{_AXIS_PREFIX}{dim}"
    names = _plain_labels(dim, labels, f"`{call}`")
    self._reject_undeclared(f"`{call}`", _axis_columns(self.schema, dim), columns)
    self._require_typed(dim, names, f"`{call}`")
    data: dict[str, list[Any]] = {dim: names}
    for c, given in columns.items():
        data[c] = list(given) if _is_labels(given) else [given] * len(names)
        if len(data[c]) != len(names):
            msg = (
                f"`{call}` has {len(data[c])} {c!r} values for {len(names)} labels"
            )
            raise ValueError(msg)
    self.schema.require_one_of(dim, data, f"`{call}`")
    self._require_live(dim, data, f"`{call}`")
    rel = self._columns_relation(data, {c: self._column_type(c, dim) for c in data})
    table = self._ensure(kind)
    self._insert_snapshot(
        self._positioned(rel, table, kind, dim), table, {}, key=(dim,)
    )

remove

remove(
    dim: str,
    labels: Sequence[Any]
    | Sequence[Mapping[str, Any]]
    | DuckDBPyRelation,
) -> None

Stage dim's axis without labels.

Need not enumerate what it deletes: the fold drops every value keyed by a removed label where dim is partial. No labels stage nothing.

And every multi-dim with a coordinate over dim without the tuples naming one of labels, read from the record as it currently resolves: a multi-dim attribute's value goes with its tuple.

A multi-dim's labels are tuples, given as add takes them.

Raises:

Type Description
TypeError

As add does.

ValueError

On a dim with one_of, whose axis is the schema's, a label whose type does not fit the dtype of dim, a tuple given as add refuses it, or while a property that refers to dim stores one of labels in the record as staged.

Notes
Source code in src/datarecord/mutable.py
@_atomic
def remove(
    self,
    dim: str,
    labels: Sequence[Any] | Sequence[Mapping[str, Any]] | DuckDBPyRelation,
) -> None:
    """Stage `dim`'s axis without `labels`.

    Need not enumerate what it deletes: the fold drops every value keyed
    by a removed label where `dim` is `partial`. No labels stage nothing.

    And every multi-dim with a coordinate over `dim` without the tuples
    naming one of `labels`, read from the record as it currently resolves:
    a multi-dim attribute's value goes with its tuple.

    A multi-dim's labels are tuples, given as `add` takes them.

    Raises
    ------
    TypeError
        As `add` does.
    ValueError
        On a dim with `one_of`, whose axis is the schema's, a label whose
        type does not fit the dtype of `dim`, a tuple given as `add`
        refuses it, or while a property that refers to `dim` stores one of
        `labels` in the record as staged.

    Notes
    -----
    - [add / remove](https://energy-models.github.io/datarecord/reference/editing/#add-and-remove)
    - [deletion](https://energy-models.github.io/datarecord/reference/layers/#removing-labels)
    """
    call = f"`remove({dim!r}, ...)`"
    self._refuse_one_of(dim, call)
    if dim in self.schema.multi_dims:
        if not is_frame(labels) and not labels:
            return
        rel = self._tuples(dim, self._tuple_frame(dim, labels, {}, call), call)
        table = self._ensure(f"{_AXIS_PREFIX}{dim}")
        self._delete(table, rel, dim, col("t", dim))
        return
    labels = _plain_labels(dim, labels, call)
    if not labels:
        return
    self._require_typed(dim, labels, call)
    self._require_unnamed(dim, labels)
    gone = self._columns_relation(
        {dim: list(labels)}, {dim: self._column_type(dim)}
    )
    staged = self.dims
    for m in self.schema.multi_dims:
        fields = [f for f, d in self.schema.over(m).items() if d == dim]
        if not fields or m not in staged:
            continue
        naming = reduce(
            operator.or_,
            (_field_in(m, f, labels) for f in fields),
        )
        if not staged[m].filter(naming).limit(1).fetchall():
            continue
        table = self._ensure(f"{_AXIS_PREFIX}{m}")
        for f in fields:
            self._delete(table, gone, dim, col("t", m)[f])
    table = self._ensure(f"{_AXIS_PREFIX}{dim}")
    self._delete(table, gone, dim, col("t", dim))

rollback

rollback() -> None

Clear every staged row without writing.

Notes
Source code in src/datarecord/mutable.py
@_atomic
def rollback(self) -> None:
    """Clear every staged row without writing.

    Notes
    -----
    - [WorkingRecord](https://energy-models.github.io/datarecord/reference/editing/)
    """
    for kind, attribute in self._touched:
        self.con.execute(f"DELETE FROM {self._staged[kind, attribute]}")
        self._fill(kind, attribute)
        if kind == _ATTRIBUTES:
            self.con.execute(f"DELETE FROM {self._staged[_OWNERS, attribute]}")
    self._touched.clear()