DEP 1: dims, mappings, groups and traits¶
| DEP | 1 |
|---|---|
| Title | dims, mappings, groups and traits |
| Author | Jonas Hörsch |
| Status | Implemented (2026-08-16) |
| Created | 2026-08-15 |
| Version | v0.1.0a1 |
| Superseded-By | DEP 16 (its traits part), DEP 20 (its entity tombstones, and an order the fold generates) |
The design landed in the data model, the schema, the record format and the fold. Where they and this DEP differ, follow those pages. DEP 2 absorbs the on mapping into groups as into.
Abstract¶
AttributeSpec.dims becomes the one way to address an attribute. The component name becomes a dim, entity. Attributes are declared flat, types subscribe through traits, groups replace connections, mappings classify one dim by another, and Dimension.keys goes.
Motivation¶
An attribute that belongs to no component type has nowhere to live. A snapshot weighting varies by snapshot alone, and Schema.attributes is keyed by component type. The PyPSA tool stores it as an undeclared objective column on dims/snapshots.parquet and recognises it by name.
Such a column has no dtype, default, unit or description, appears in no flags and does not fold across layers. Scenario weightings have the same problem. Yet name is an axis like snapshot, and only looks structural because the schema nests attributes under it.
Specification¶
Four mechanisms declare where labels come from. within stays a fifth, distinct from a mapping.
| Mechanism | Declares | Lives in |
|---|---|---|
| dim | an axis of labels | dims/<dim>.parquet, with any attribute addressed by it alone |
| mapping | each label of the dim it is on has one of its own |
its own axis file, plus a column on the classified dim's file |
| group | which tuples over several dims exist | its own table, which subsumes connections |
| trait | which entities an attribute applies to | a subscription on the component type |
entityis a declared dim.dims/entity.parquethas the columnsentity | component_type | deleted, so a label is unique by construction,entity -> component_typeis a column read, and a deletion has one home.order_keyis a column of the owner map and never of a layer's file, and persists only where a node's maps are materialised. The fold computes it per layer asmax(parent) + row_number() OVER (ORDER BY _row), before the union. Nothing sorts eagerly: a reader appliesORDER BY order_key. A table that gains rows across layers has it, and an ordinary dim does not.dims/components/<Type>.parquetcontains only the values that vary over nothing, partitioned by type because each type has its own attribute vocabulary.Schema.attributesis flat: one spec per attribute, one file, one dtype.AttributeSpec.busis deleted.- An attribute whose
dimscontain exactly one addressing coordinate is a column on that thing's table. Any other attribute is long rows ininputs/<attr>.parquet.varyingmeans "has dims beyond its address".rolebecomesAttributeSpec(dtype="VARCHAR", dims={"connection"}), a column of the group's table. - A trait is a named set of attributes.
Schema.attributes_for(ctype)returns the union of a type's traits and its own attributes. Two traits that declare one attribute with different specs fail validation. - A group's
overmaps coordinate name to dim. An attribute over a group has the group's coordinate names as columns and never the group's name, so two groups may share a coordinate set. A group indimsexpands to its coordinates, and an attribute not addressed by a coordinate writes NULL there. busis no longer structural. The long schema's fixed prefix isentity | attribute | breakpoint | value.entityis the one dim whose name the format fixes, because the component types partition it.- A NULL group coordinate broadcasts over the group's rows, not the product of its dims.
- The protocol's
connectionsbecomesgroups, keyed by group name.dimsincludesentityand every mapping.shape()joinsflags(ctype), keyed by attribute and record-wide, withdims,broadcast,breakpointsandtypes. - A mapping is a
Dimensionwithon, and is a dim. It has its own axis file, is not inpartial, never scopes existence, and is not denormalised down a chain. Dims and mappings share one namespace, so a name declared twice fails validation. A many-to-many classification is a group. The record does not project an attribute across levels of a mapping. Dimension.keysis deleted and nothing replaces it. An entity or a group row exists or does not, and neither varies along another dim.
So {"entity"} is a column of dims/components/<Type>.parquet, {"scenario"} a column of dims/scenario.parquet, and {"entity", "snapshot"} long rows.
Rationale¶
Nesting attributes under a type lets two types declare one attribute with different dtypes over one file and one value column. Nothing rejects that, and the read is silently wrong. Flat declaration makes the case unrepresentable.
datarecord keeps traits because the shared sets are measured from PyPSA's registry, and lpspec dispatches an equation on a trait. Someone has to write traits and keep them up to date.
over maps names to dims because a corridor between two entities needs two columns, from and to, from one dim. The fold's key stays one tuple, the union of plain dims and coordinate names.
within does not subsume bus, because groups do it. The recorded blocker was that bus inverts the NULL rule. Expansion already depends on whether a dim is in dims ∩ partial, so bus is not an exception.
The fold assigns order_key before the union, because row_number() over what UNION ALL emits has no defined order. datarecord keeps it for a group because a framework numbers positional ports by it, and a reshuffle would move a connection's attributes to another port.
A mapping has its own axis file so that it has an order and a place for data such as co2_budget. A denormalised chain would let a layer that restates dims/state.parquet leave every bus's country stale. A generator's country follows from its bus, so a mapping cannot scope existence.
within scopes a dim's labels per parent, so the axis key is (period, timestep). on classifies another dim with one flat set of labels. Neither can express the other's case.
keys cannot survive groups, because KeyKind lists a "connection" table that groups delete. A replacement was drafted and removed, because its meaning is unsettled. Dropping a mechanism whose meaning is unresolved is better than carrying it into a redesign under a new name.
Flattening brings out three vocabulary questions that the storage already had:
- One concept has one name. PyPSA's
p_nom,s_nomande_nombecomecapacity, soinvestablecan contain the capacity itself. The record's vocabulary is not PyPSA's. - The tool renames attributes through its schema on the way in and out.
Bus.xandLine.xalready collide in oneinputs/x.parquet, and now fail at declaration. A tool prefixes, renames or refuses inverify.Link.efficiency2collapses to a stem over theconnectiongroup, andbus0andbus1become rows of the group.
Every step is incompatible: version bumps once, a record is rewritten, and scenario-scoped membership becomes unrepresentable.
Usage¶
attributes = {
"p_max_pu": AttributeSpec(
dtype="DOUBLE", dims={"entity", "snapshot", "scenario"}, default=1.0
),
"efficiency": AttributeSpec(
dtype="DOUBLE", dims={"connection", "snapshot"}, default=1.0
),
"objective_weighting": AttributeSpec(
dtype="DOUBLE", dims={"snapshot"}, default=1.0
),
}
groups = {
"connection": Group(over={"entity": "entity", "bus": "bus"}),
"corridor": Group(over={"from": "entity", "to": "entity"}),
}
objective_weighting lands on dims/snapshot.parquet, declared and typed.
Implementation notes¶
- Flat attributes and traits. Only the manifest changes.
- Singular parquet names,
dims/<dim>.parquet, which fixesdims/buss.parquet. - Mappings, which the fold treats as dims.
entityas a dim, with no order column. The long schema's columns become schema-derived.- Groups, with the fold's key schema-derived, and
keysdeleted in the same step withKeyKind,_keyed,component_dims,connection_dimsand the tests of scenario-scoped existence. This is the one functional regression, and should not land before the existence question is answered. - The protocol:
connectionsbecomesgroups, andshape()joinsflags. WorkingRecordcan address attributes with noentityand groups of two coordinates.
Two breaks do not announce themselves. At step 4, add routes by varying, which turns true for dims={"entity"}. And _overlay would put entity in broadcast, so it would apply a staged row with a NULL entity to every entity and overwrite each value. The fix derives the key's address coordinates instead of spelling them, and a group coordinate belongs in the key.
The PyPSA tool's trait table and its p_nom -> capacity renames must exist before step 1 ships against a real network.
On implementation, shape() was not built, so flags(ctype) stands alone. set has no bus= parameter: every coordinate but entity goes through **dims, so a group of two coordinates can be addressed.
Rejected ideas¶
- A separate payload declaration on
GrouporDimension: two ways to say one thing. - Trait roles: one more indirection for what the tool's schema already records.
- Qualified names or per-type namespaces: both put the component type back into an attribute's address.
- Widening
flagsto record-level attributes:flagsis per type by construction. - A field declaring per-dim existence: drafted, then removed, because its meaning is unsettled.
- Last-wins for conflicting traits: a conflict fails validation instead.
Open issues¶
- Whether
flags(ctype)survives, or becomesshape()filtered bytypes. - Outputs are out of scope.
- Whether existence may depend on a dim, as a generator built in
highand absent inlow. Ifentityvaries overscenarioandconnectiondoes not, one connection row cannot be both kept and dropped. The fold drops it, pinned by an xfail. A validator could forbid a group whose existence dims are coarser than those ofentity, which is a decision and not a default.
Three questions follow from the last, and a "no" to the first deletes the whole area:
- Is per-dim existence needed, or does an
active-style attribute cover it? PyPSA has both. - If needed, may tables disagree, or is uniformity enforced?
- If they may disagree, what does the coarser table's row mean, and can the fold compute it?