Your first record¶
In this tutorial you build a record of two generators, commit it to a store and read it back. It takes about 25 minutes.
You need datarecord installed, as Installation shows.
The schema¶
A record's schema declares what the record may contain. It lists the sets of labels that values vary along (dims), the values that describe one label (properties) and the values over dims (attributes). The data model defines each. Run this:
from datarecord import AttributeSpec, Dimension, PropertySpec, Schema
schema = Schema(
dimensions={
"entity": Dimension(dtype="str"),
"ctype": Dimension(one_of=["Generator", "Load"]),
"bus": Dimension(dtype="str"),
"carrier": Dimension(dtype="str"),
"snapshot": Dimension(dtype="int"),
"connection": Dimension(over=["entity", "bus"]),
},
properties={
"entity": {
"ctype": PropertySpec(one_of="ctype"),
"carrier": PropertySpec(one_of="carrier"),
},
},
attributes={
"p_nom": AttributeSpec(dtype="float", dims={"entity"}, default=0.0, unit="MW"),
"p_max_pu": AttributeSpec(
dtype="float", dims={"entity", "snapshot"}, default=1.0
),
},
partial={"entity", "bus", "connection"},
)
print(schema.dims)
print(schema.multi_dims)
Check it:
connection is a dim whose labels are (entity, bus) tuples (a multi-dim), so the schema lists it apart from the five plain dims. ctype lists its labels in the schema with one_of, and partial lists the dims along which a later layer overrides one label's value.
An empty root¶
A record is kept in one location (a store) as a tree of the data each commit adds (layers). You open each layer by its id (a revision). You stage edits over a revision, so the first revision has the schema and no rows. Any object with the members of RecordLike can be committed, and this dataclass has the fewest:
import tempfile
from dataclasses import dataclass, field
from duckdb import DuckDBPyConnection
from datarecord import Workspace
@dataclass(frozen=True)
class Empty:
schema: Schema
con: DuckDBPyConnection
dims: dict = field(default_factory=dict)
properties: dict = field(default_factory=dict)
attributes: dict = field(default_factory=dict)
def flags(self, **categories):
return {}
ws = Workspace()
store = ws.store(tempfile.mkdtemp())
root = store.commit(Empty(schema, ws.con), parent=None)
print(root.record.dims["entity"].df())
Check it:
The root revision's entity dim has no labels. The Workspace is the DuckDB database that its stores share.
Labels¶
A WorkingRecord stages edits over a revision. Add the labels:
from datarecord import WorkingRecord
w = WorkingRecord(root.record)
w.add("carrier", ["wind", "gas"])
w.add("bus", ["north", "south"])
w.add("entity", ["wind1", "gas1"], ctype="Generator", carrier=["wind", "gas"])
w.add(
"connection",
[{"entity": "wind1", "bus": "north"}, {"entity": "gas1", "bus": "south"}],
)
w.add("snapshot", [0, 1, 2])
print(w.properties["entity"].df())
print(w.dims["connection"].df())
Check it:
entity ctype carrier
0 wind1 Generator wind
1 gas1 Generator gas
connection
0 {'entity': 'wind1', 'bus': 'north'}
1 {'entity': 'gas1', 'bus': 'south'}
Each generator has its ctype and carrier properties, and each connection is one struct with an entity and a bus field. The carriers and buses come first, because add refuses a property value or a tuple field that is not yet a label of its dim.
Values¶
set stages attribute values at the labels you name:
w.set("p_nom", [100.0, 50.0], entity=["wind1", "gas1"])
w.set("p_max_pu", [0.3, 0.8, 0.5], entity="wind1", snapshot=[0, 1, 2])
w.set("p_max_pu", 0.9, entity="gas1")
print(w.attributes["p_max_pu"].df())
Check it:
entity snapshot attribute breakpoint value
0 wind1 0 p_max_pu NaN 0.3
1 gas1 <NA> p_max_pu NaN 0.9
2 wind1 1 p_max_pu NaN 0.8
3 wind1 2 p_max_pu NaN 0.5
attributes returns long rows, one per value, in no promised order. The gas1 row leaves snapshot NULL, and a NULL coordinate means every label of its dim (the broadcast rule).
The values, wide¶
settle applies that rule and gives one row per label at each point, which pandas pivots wide:
from datarecord import settle
labels = {"entity": w.dims["entity"], "snapshot": w.dims["snapshot"]}
settled = settle(w.attributes["p_max_pu"], labels, w.flags()["p_max_pu"])
print(settled.df().pivot(index="snapshot", columns="entity", values="value"))
Check it:
settle turned the one gas1 row into a value of 0.9 at each of the three snapshots. properties is already wide, with one row per label.
A committed revision¶
store.commit writes the staged edits as a layer, a child of the revision w was made over:
The new revision rev is a child of the root. Open it by its id and read it:
The committed revision contains the values you set. The root still has no rows, because a layer is written once and a commit never changes the parent:
What you have built¶
You have built a store with two revisions. The root is empty, and its child contains two generators, their buses and connections, and values for p_nom and p_max_pu. You read the values long, as they are stored, and wide, with the broadcast rule applied.
Where to next¶
- A scenario on a branch branches this record and overrides one value.
- Solve a record with PyPSA hands a record to PyPSA.
- Read a record lists the other ways to read one.