Present a framework object as a record¶
This guide shows how to wrap a framework's own object so that any reader of a record reads it, and how to commit it. It is for a tool author, who knows the framework. RecordLike is the protocol the wrapper satisfies, and the data model defines the shapes below.
Implement the six members¶
RecordLike is a structural protocol, so a class with its six members satisfies it without importing anything from datarecord. The members are schema, con, dims, properties, attributes and flags, and the record states what each returns.
A mapping may be a plain dict. A dim with one_of needs no entry in dims, because its axis is the schema's list.
Write the wrapper¶
This framework's object has a table of units and a table of hourly availability. The wrapper declares a schema for them and renames their columns to the record's:
from collections.abc import Sequence
from dataclasses import dataclass
import pandas as pd
from duckdb import DuckDBPyConnection
from datarecord import AttributeSpec, Dimension, Flags, PropertySpec, Relations, Schema
from datarecord.resolve.fold import Fold
SCHEMA = Schema(
dimensions={
"entity": Dimension(dtype="str"),
"ctype": Dimension(one_of=["Generator", "Load"]),
"snapshot": Dimension(dtype="int"),
},
properties={"entity": {"ctype": PropertySpec(one_of="ctype")}},
attributes={
"p_nom": AttributeSpec(dtype="float", dims={"entity"}, default=0.0),
"p_max_pu": AttributeSpec(
dtype="float", dims={"entity", "snapshot"}, default=1.0
),
},
partial={"entity"},
)
class Fleet:
def __init__(self) -> None:
self.units = pd.DataFrame(
{
"name": ["wind1", "gas1"],
"kind": ["Generator", "Generator"],
"capacity": [100.0, 50.0],
}
)
self.availability = pd.DataFrame(
{"name": ["wind1"] * 3, "hour": [0, 1, 2], "pu": [0.3, 0.8, 0.5]}
)
@dataclass(frozen=True)
class FleetRecord:
fleet: Fleet
con: DuckDBPyConnection
schema: Schema = SCHEMA
@property
def dims(self) -> Relations:
units = self.con.from_df(self.fleet.units)
hours = self.con.from_df(self.fleet.availability)
return {
"entity": units.project("name AS entity"),
"snapshot": hours.project("hour::BIGINT AS snapshot")
.distinct()
.order("snapshot"),
}
@property
def properties(self) -> Relations:
units = self.con.from_df(self.fleet.units)
return {"entity": units.project("name AS entity, kind AS ctype")}
@property
def attributes(self) -> Relations:
units = self.con.from_df(self.fleet.units)
hours = self.con.from_df(self.fleet.availability)
return {
"p_nom": units.project(
"name AS entity, 'p_nom' AS attribute, NULL::DOUBLE AS breakpoint, capacity AS value"
),
"p_max_pu": hours.project(
"name AS entity, hour::BIGINT AS snapshot, 'p_max_pu' AS attribute,"
" NULL::DOUBLE AS breakpoint, pu AS value"
),
}
def flags(self, **categories: str | Sequence[str]) -> dict[str, Flags]:
return Fold.as_one_layer(self).flags(**categories)
Fold.as_one_layer reads the wrapper's rows as one layer and answers flags as a record written from them would. The PyPSA tool answers flags the same way.
Check the wrapper reads as a record¶
from datarecord import RecordLike, Workspace, select
ws = Workspace()
record = FleetRecord(Fleet(), ws.con)
print(isinstance(record, RecordLike))
print(record.flags(ctype="Generator")["p_max_pu"])
print(select(record, snapshot=1).attributes["p_max_pu"].df())
True
Flags(varies=frozenset({'entity', 'snapshot'}), broadcast=frozenset(), breakpoints=False)
entity attribute breakpoint value
0 wind1 p_max_pu NaN 0.8
p_max_pu varies along both dims, and the selection at snapshot 1 returns wind1's 0.8. isinstance checks only that the members exist. The select call is the test that the relations have the shapes the reference states.
Commit the wrapper¶
store.commit and ws.write take the wrapper as they take any record, and validate every row as they write it:
import tempfile
store = ws.store(tempfile.mkdtemp())
root = store.commit(record, parent=None)
print(root.record.properties["entity"].df())
The committed root has both units as generators. A WorkingRecord stages edits over a committed Record alone, so commit the wrapper before you edit it:
TypeError: a `WorkingRecord` folds its staged rows over a `Record`'s layers, and a FleetRecord has none: write it first, `WorkingRecord(ws.write(source, uri))`
WorkingRecord(root.record) stages edits over the committed wrapper.
Build relations on demand¶
Where building a relation costs a read, as for a file on a remote store, return a LazyRelations instead of a dict. It lists its keys at once and builds a relation when a reader looks one up:
from datarecord import LazyRelations
lazy = LazyRelations(("p_nom", "p_max_pu"), lambda name: record.attributes[name])
print(list(lazy), "p_nom" in lazy)
Listing the keys and testing for one builds no relation.