The schema¶
This page describes every key a schema accepts, in the order a schema declares them, and what datarecord refuses at load.
A schema is the Schema of a record: the dims, properties, attributes and partial that may exist, with their dtypes.
The data model defines each of those words.
The running example declares this schema:
from datarecord import AttributeSpec, Dimension, PropertySpec, Schema
schema = Schema(
dimensions={
"entity": Dimension(),
"bus": Dimension(),
"carrier": Dimension(),
"snapshot": Dimension(dtype="int"),
"ctype": Dimension(one_of=["Generator", "Link", "Load"]),
"connection": Dimension(over=["entity", "bus"]),
},
attributes={
"p_nom": AttributeSpec(dtype="float", dims={"entity"}, default=0.0, unit="MW"),
"efficiency": AttributeSpec(dtype="float", dims={"connection"}, default=1.0),
"p_max_pu": AttributeSpec(
dtype="float",
dims={"entity", "snapshot"},
default=1.0,
default_by={"entity.ctype": {"Load": 0.0}},
),
},
properties={
"entity": {
"ctype": PropertySpec(one_of="ctype"),
"carrier": PropertySpec(one_of="carrier"),
},
},
partial={"entity", "bus", "connection"},
)
Schema.default_for("p_max_pu", ctype="Load") returns 0.0, and Schema.default_for("p_max_pu") returns 1.0.
A Schema and each spec are frozen pydantic models that refuse an undeclared key.
Every refusal on this page raises when the schema is built or read from a manifest.
version¶
version is an integer, 1 by default, that you raise with each change to the declarations.
datarecord stores it and compares nothing against it.
Compatibility lists the changes a child revision's schema may make.
dimensions¶
dimensions maps each dim's name to a Dimension, multi-dims among them.
| Key | Value | Default |
|---|---|---|
over |
a multi-dim's coordinates, each mapped to the dim it draws on | None |
dtype |
"float", "int", "str" or "datetime" |
"str" |
one_of |
the dim's labels, in order | None |
unit |
what the labels measure | None |
description |
what the dim is, in prose | None |
A schema that declares attributes declares at least one dim, because attribute data over no dim is no record.
over¶
over declares a multi-dim.
A list is short for the dict whose keys equal its values, so over=["entity", "bus"] is over={"entity": "entity", "bus": "bus"}.
The keys are the coordinate names, in field order, and Schema.over("connection") returns them.
datarecord refuses a multi-dim that also declares dtype or one_of, and a coordinate that draws on an undeclared dim or on a multi-dim.
one_of¶
one_of declares a dim with one_of, whose axis is the list.
Only a "str" dim takes it.
datarecord refuses an empty list, a label listed twice, and one_of on any other dtype.
attributes¶
attributes maps each attribute's name to an AttributeSpec.
One name is one spec, one file and one value dtype, record-wide.
| Key | Value | Default |
|---|---|---|
dtype |
"float", "int", "bool" or "str" |
required |
dims |
the dims and multi-dims that address a value | frozenset() |
default |
the value at a point no row covers | None |
default_by |
a default per category, which overrides default |
{} |
breakpoints |
whether a row may have a piecewise-linear curve | False |
unit |
what the values measure | None |
description |
what the attribute is, in prose | None |
datarecord refuses a dims entry that is not a declared dim.
It also refuses a multi-dim in dims beside a dim named like one of its coordinates, such as {"connection", "bus"}.
default_by¶
default_by maps a categorising property, written subject.name, to a map from category to default.
At a label whose category a map lists, the first key in declaration order gives the default.
Elsewhere default applies.
Schema.default_for(name, **categories) returns the default for an attribute or a subject.name property, taking each category by the property's plain name.
datarecord refuses a key that is not a categorising property, naming the reason:
attribute 'p_nom''s `default_by` is keyed by 'entity.carrier', which refers to 'carrier', but 'carrier' is not a one_of dim
It also refuses a category that the referred dim's one_of does not list.
It refuses a key on a dim that the attribute or property is not indexed over.
A property is indexed over its subject and the dims that the subject's coordinates draw on.
properties¶
properties maps each subject dim to a map from property name to PropertySpec.
| Key | Value | Default |
|---|---|---|
dtype |
"float", "int", "bool" or "str" |
None |
one_of |
the dim the property refers to | None |
default |
the value for a label no layer wrote one for | None |
default_by |
a default per category, as on an attribute | {} |
unit |
what the values measure | None |
description |
what the property is, in prose | None |
A property declares exactly one of dtype and one_of.
With one_of, the property refers to that dim and takes its type.
Schema.resolve("entity.carrier") returns the subject and the spec, and Schema.categorizing(dim) lists the categorising properties of dim.
datarecord refuses a property:
- that declares both
dtypeandone_of, or neither - of an undeclared subject, or of a dim with
one_of - that refers to an undeclared dim or to a multi-dim
- named like its subject's key column, or like a coordinate of its multi-dim
- that categorises its subject and declares
default_by
partial¶
partial is the set of dims along which the data one revision adds (a layer) overrides one value per label, a multi-dim among them.
Layers states what a layer restates along any other dim.
None, the default, declares a record with no layers.
datarecord refuses a name in partial that the schema does not declare as a dim.
meta¶
meta is a dict of a framework's own data, such as network attributes or a coordinate reference system.
datarecord stores it and never reads it.
unit and description¶
A Dimension, an AttributeSpec and a PropertySpec each take a unit and a description.
datarecord stores both and reads neither, so it converts no unit.
None means undeclared, and "" declares a quantity without a unit.
Dtypes¶
A dtype is mathspec's DimensionDtype or ParameterDtype, written as its string.
Each declaration maps to one DuckDB type:
| Declares | Declaration | DuckDB type |
|---|---|---|
| dim, attribute, property | dtype: float |
DOUBLE |
| dim, attribute, property | dtype: int |
BIGINT |
| attribute, property | dtype: bool |
BOOLEAN |
| dim, attribute, property | dtype: str |
VARCHAR |
| dim | dtype: datetime |
TIMESTAMP (no time zone) |
| dim | one_of: [labels] |
ENUM of the labels |
| multi-dim | over: {...} |
STRUCT, a field per coordinate typed as its dim |
| property | one_of: <dim> |
that dim's type |
An attribute takes no one_of, and no attribute or property is datetime.
A property whose values are times refers to a datetime dim.
The format fixes three more columns: attribute VARCHAR, breakpoint DOUBLE and value, typed as the attribute's dtype.
Names¶
A declared dim, coordinate, attribute or property name becomes a column, and datarecord refuses a name that cannot be one:
- a name that is not ASCII letters, digits and
_, starting with a letter, which is mathspec's rule for a name - a column datarecord writes, compared ignoring case:
attribute,breakpoint,breakpoints,broadcast,layer_uuid,order_key,valueandvaries - a name that starts with
_ - two names that differ only in case, which DuckDB reads as one column
One spelling declared twice is one column and passes, as when a coordinate shares its dim's name. Labels are data, and these rules do not apply to them.
Compatibility¶
schema.compatible_with(other) returns one reason for each way that layers written under other would not read under schema, and an empty list when they would.
Store.commit checks a child's schema against its parent's this way and refuses an incompatible one before it writes any data.
| Change | Compatible |
|---|---|
| add a dim, an attribute or a property | yes |
add a dim to an attribute's dims |
yes |
add a dim to partial |
yes |
change a default, default_by, unit or description |
yes |
| remove a dim, an attribute or a property | no |
change a dtype, a one_of or an over |
no |
remove a dim from an attribute's dims |
no |
remove a dim from partial |
no |
Moving a name between attributes and properties, or between subjects, removes it, so compatible_with reports it.
An incompatible change needs the record rewritten under the new schema, such as a standalone directory written with ws.write.
One schema per record¶
A record is read under the schema of its last layer.
Each layer's file of its id, parent and files (its manifest) has the full schema the layer was written under.
A root's schema is free, and a child's passes compatible_with against its parent's, so two sibling branches may declare one new name differently.
A WorkingRecord reads and commits under its base's schema.
See also
- The data model defines what each key declares.
- Layers describes
partialand what a layer restates. - Declare a schema shows the steps.
- The
SchemaAPI lists every method.