The data model¶
This page describes what a record contains: dims and their labels, multi-dims, properties and categories, attributes, and the broadcast rule for an attribute's rows.
A record is the object you hold: dims, properties and attributes under one schema, exposed as DuckDB relations through RecordLike.
Record, WorkingRecord and Selection are records.
The reasons for each choice below are on the about page.
The examples read one record of the running example, whose schema the schema reference declares.
It has the entities wind, gas and dc, the buses north and south, and the snapshots 0, 1 and 2.
Dims and labels¶
A dim is a set of unique labels that values vary along, declared by a Dimension under dimensions.
A label is one value of a dim, unique within it, typed by the dim's dtype.
An axis is the labels one dim has in one record, in order, which record.dims[dim] returns:
The axis lists the three entities in axis order.
A label exists because its axis has a row for it, whether or not any attribute row contains it. The schema declares no order. A record's data is the data each of its revisions adds (a layer), and resolution states which layer's file sets the axis order.
A dim with one_of¶
A dim with one_of has exactly the labels its list declares, in list order.
Its axis comes from the schema and from no file, so record.dims["ctype"] returns Generator, Link and Load in every record of the schema.
Its column is an ENUM of the labels.
A dim with one_of has no properties, and add and remove refuse it: a new label is a schema change.
Otherwise it is a dim like any other, which an attribute may range over and a property may refer to.
Multi-dims¶
A multi-dim is a dim declared with over, whose labels are tuples of other dims' labels and list the combinations that exist.
connection over (entity, bus) lists which entity attaches to which bus:
[({'entity': 'wind', 'bus': 'north'},), ({'entity': 'gas', 'bus': 'south'},),
({'entity': 'dc', 'bus': 'north'},), ({'entity': 'dc', 'bus': 'south'},)]
The link dc attaches to both buses, and each generator to one.
A multi-dim is one struct column named for it, in its axis and in every attribute row over it.
A coordinate is one named position in a key: a column of an attribute's rows, or a field of a multi-dim's tuple.
The struct has one field per coordinate, in over order, each typed as the dim it draws on.
Two coordinates may draw on one dim, such as from and to of a corridor between two buses.
Each coordinate draws on a dim that is no multi-dim.
The struct's value is a whole tuple or NULL.
A struct with some fields NULL, such as {entity: dc, bus: NULL}, matches no tuple, and set, add and store.commit refuse it.
connection.bus is the bus coordinate wherever a dotted subject.name stands.
Schema.resolve("connection.bus") returns connection and the dim bus that the coordinate draws on.
A coordinate is part of the key and has no value of its own, so datarecord refuses set("connection.bus", ...).
A tuple changes only through a remove of the old tuple and an add of the new one.
Removing a label that a tuple contains removes the tuple and its values (layers).
For a multi-dim with no tuples, record.dims returns an empty relation.
Properties¶
A property is a value per label of its subject dim, declared by a PropertySpec under properties.
It is stored as a column of that dim's axis file, so it has no dims of its own.
You name it subject.name, such as entity.carrier or connection.role, and a plain name always means an attribute.
record.properties[subject] returns the subject's key and every property declared for it:
The columns are entity, ctype and carrier, and dc has no carrier because no layer wrote one.
A property's name belongs to its subject.
bus.x and an attribute x are two names, and carrier may be a property of entity, a property of bus and a dim at once.
A multi-dim's tuples are its labels, so a multi-dim may have properties too.
A property may refer to a dim with one_of in place of dtype, and its values are then that dim's labels.
A value that is not NULL must be a label the record has.
set and add refuse a label the axis lacks, and remove refuses a label that is still a property's value.
A NULL value refers to no label.
A chain such as bus to state to country is two properties, bus.state and state.country, which a reader joins.
A property has one value per label and does not vary along another dim.
A mapping that varies, such as a bus's country per scenario, is a multi-dim over (bus, scenario) with a property country.
Categories¶
A category is a label's value of a categorising property. A property categorises its subject when two conditions hold:
- The subject is in
partial, the dims along which a layer overrides one value per label, a multi-dim among them. - The property refers to a dim with
one_of.
entity.ctype refers to ctype, which has one_of, and entity is in partial, so ctype categorises entity.
entity.carrier refers to carrier, which has no one_of, so it categorises nothing.
Schema.categorizing("entity") lists ("ctype",), the categorising properties in declaration order.
A category selects labels and keys a default:
flags(ctype="Generator"), asetkeyword and aselectkeyword each apply to the labels with that category.default_bykeyedentity.ctypegives a default per category, whichSchema.default_for(name, ctype="Load")returns.
The keyword is the property's plain name, ctype.
A category that no label has selects no label.
When set changes a label's category, the label keeps its attribute values, and a read per category uses the current category.
Attributes¶
An attribute is the values at points over the dims its AttributeSpec lists, stored as long rows, one file per attribute.
A point is one label or NULL on each coordinate of an attribute, which is where a row's value applies.
Each row has the attribute's coordinates, then attribute, breakpoint and value:
[('wind', 0, 'p_max_pu', None, 0.3), ('wind', 1, 'p_max_pu', None, 0.8),
('wind', 2, 'p_max_pu', None, 0.5), (None, None, 'p_max_pu', None, 1.0)]
wind has a value per snapshot, and the last row, NULL on both coordinates, gives 1.0 at every other point.
An attribute's coordinates are the entries of its dims, with a multi-dim as its one struct column, so p_nom has entity alone.
An attribute is indexed over each dim its dims lists and each dim that a coordinate of one of its multi-dims draws on.
dims sets which coordinates address a value and nothing about what the value means.
A solve's results are attributes too, which a child layer inherits like any other row.
breakpoint is NULL in an ordinary row.
On an attribute with breakpoints, a curve is one row per breakpoint, with breakpoint the abscissa and value the ordinate there.
datarecord checks no property of the curve, such as convexity.
The broadcast rule¶
The broadcast rule is the rule that a NULL coordinate means every label of its dim.
Where rows cover one point, datarecord keeps the row that leaves no dim NULL that another row there sets.
The settled rows below take the rows of p_max_pu above and give one row per point:
wind keeps its three values, because a row that sets both coordinates is more specific than the NULL row.
gas and dc take 1.0 at every snapshot.
A NULL coordinate means every label of its dim, labels that a later layer adds included.
set("p_max_pu", 0.9) writes one row with entity NULL, which applies to every entity, loads included.
A value for some categories is one row per label: set("p_max_pu", 0.9, ctype="Generator") writes a row for wind and one for gas.
A row that sets a dim is more specific than one that leaves that dim NULL.
Where two rows of one layer each leave NULL a dim that the other row sets, neither is more specific.
store.commit refuses such a layer, naming the attribute and the point.
Rows of two layers meet the same way, and the fold adds a row that sets both where they cross.
A point that no row covers takes the attribute's default, from default_by for the label's category, else default.
A default is never stored.
record.attributes returns the rows as written, NULL included, and a reader applies the rule with settle and the default with Schema.default_for.
A multi-dim's column¶
A multi-dim's struct column broadcasts over its tuples.
A NULL connection means every tuple, and a whole tuple means one:
The tuple {dc, south} keeps 0.9, and the three other tuples take 1.0 from the NULL row.
The dims a multi-dim draws on broadcast on their own.
A coordinate stays inside its struct, so bus inside connection is a field, and an attribute over the dim bus has a separate bus column.
An attribute's dims may not list a multi-dim beside a dim with the name of one of its coordinates. A keyword such as bus= would then address both.
Wide and long rows¶
A record returns dims and properties wide, with one row per label or tuple, and attributes long, with one row per value:
| Relation | Columns |
|---|---|
dims["entity"] |
entity |
dims["connection"] |
connection STRUCT(entity, bus) |
properties["entity"] |
entity, then one column per property |
attributes["p_max_pu"] |
entity, snapshot, attribute, breakpoint, value |
an attribute over connection |
connection, attribute, breakpoint, value |
An attribute row has no property columns.
To read one category's rows, a reader joins properties["entity"], filtered on the category, on entity.
To read a multi-dim's fields flat, a reader takes connection.bus in SQL or unnests the struct.
See also
- The schema declares every name on this page.
- The record lists what each accessor returns.
- Layers describes how a layer writes and removes labels.
- Why the data model is shaped this way.