Declare a schema¶
This guide shows how to declare each part of a schema, and how to check a changed schema against the old one. The data model defines each term.
This schema has every part the sections below use:
from datarecord import AttributeSpec, Dimension, PropertySpec, Schema
schema = Schema(
dimensions={
"entity": Dimension(dtype="str"),
"ctype": Dimension(one_of=["Generator", "Load", "StorageUnit"]),
"bus": Dimension(dtype="str"),
"carrier": Dimension(dtype="str"),
"snapshot": Dimension(dtype="datetime"),
"connection": Dimension(over=["entity", "bus"]),
},
properties={
"entity": {
"ctype": PropertySpec(one_of="ctype"),
"carrier": PropertySpec(one_of="carrier"),
"sign": PropertySpec(
dtype="float", default=1.0, default_by={"entity.ctype": {"Load": -1.0}}
),
},
"bus": {
"carrier": PropertySpec(one_of="carrier", default="AC"),
"x": PropertySpec(dtype="float"),
},
},
attributes={
"p_nom": AttributeSpec(dtype="float", dims={"entity"}, default=0.0, unit="MW"),
"p_max_pu": AttributeSpec(
dtype="float", dims={"entity", "snapshot"}, default=1.0, unit="per unit"
),
"p_min_pu": AttributeSpec(
dtype="float",
dims={"entity", "snapshot"},
default=0.0,
default_by={"entity.ctype": {"StorageUnit": -1.0}},
),
"efficiency": AttributeSpec(dtype="float", dims={"connection"}, default=1.0),
},
partial={"entity", "bus", "connection"},
)
Declare the dims¶
Give each dim a Dimension with a dtype of "str", "int", "float" or "datetime". Where the schema fixes the labels, as for ctype, pass one_of with the list instead. Such a dim has no axis file and no properties, and add refuses its labels.
Declare a multi-dim¶
Pass over with the dims whose labels form its tuples. A list is short for a mapping in which each coordinate is named after its dim:
over("connection") returns the mapping that the list stands for. To name a coordinate differently from its dim, pass the mapping, as in Dimension(over={"from": "bus", "to": "bus"}) for a line between two buses.
Declare properties¶
Under properties, key each property by its subject dim and give it exactly one of dtype and one_of. With one_of, the property's values are labels of the dim you give, as the values of entity.carrier are labels of carrier. A property name belongs to its subject, so bus.x and an attribute x are two names.
datarecord refuses a spec with both:
pydantic_core._pydantic_core.ValidationError: 1 validation error for PropertySpec
Value error, a property declares exactly one of `dtype` and `one_of`, not both [type=value_error, input_value={'one_of': 'carrier', 'dtype': 'str'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/value_error
Set a default per category¶
A default_by overrides default for some categories, on a property or an attribute alike. Key it on the dotted name of a categorising property, such as entity.ctype:
print(schema.categorizing("entity"))
print(schema.default_for("p_min_pu", ctype="StorageUnit"))
print(schema.default_for("entity.sign", ctype="Load"))
ctype is the one categorising property of entity, and a storage unit's p_min_pu and a load's sign both default to -1.0. If the subject dim is outside partial, datarecord refuses the default_by:
pydantic_core._pydantic_core.ValidationError: 1 validation error for Schema
Value error, property entity.sign's `default_by` is keyed by 'entity.ctype', which is a property of 'entity', which `partial` does not list [type=value_error, input_value={'version': 1, 'dimension...onnection'}, 'meta': {}}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/value_error
Declare attributes¶
Give each attribute a dtype and the dims it varies over, multi-dims included, and a default for a point that no row covers. unit and description are stored as written and never interpreted:
efficiency is keyed by the one connection coordinate, a struct of entity and bus. datarecord refuses a dim in dims that the schema does not declare:
Schema(
dimensions={"entity": Dimension(dtype="str")},
attributes={"p_nom": AttributeSpec(dtype="float", dims={"entity", "bus"})},
)
pydantic_core._pydantic_core.ValidationError: 1 validation error for Schema
Value error, attribute 'p_nom' is addressed by undeclared dims ['bus'] [type=value_error, input_value={'dimensions': {'entity':...}), breakpoints=False)}}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.13/v/value_error
Choose partial¶
List in partial the dims along which a child layer overrides one label's value. Leave out a dim whose values change together, such as snapshot for a time series. Layers states the rule.
PyPSA.verify reports snapshot in partial, and PyPSA.build raises for it (tools).
Check a changed schema against the old one¶
compatible_with returns one reason per change that stops datarecord reading layers written under the old schema with the new one. An empty list means datarecord can still read them:
cost = AttributeSpec(dtype="float", dims={"entity", "snapshot"})
wider = Schema(
**schema.model_dump() | {"attributes": schema.attributes | {"marginal_cost": cost}}
)
retyped = Schema(
**schema.model_dump()
| {
"attributes": schema.attributes
| {"p_nom": AttributeSpec(dtype="int", dims={"entity"})}
}
)
print(wider.compatible_with(schema))
print(retyped.compatible_with(schema))
Adding an attribute is compatible, and retyping one is not. store.commit runs the same check against the parent's schema, and raises a ValueError that lists the reasons.