DEP 12: properties, metadata declared apart from data¶
| DEP | 12 |
|---|---|
| Title | properties, metadata declared apart from data |
| Author | Jonas Hörsch |
| Status | Implemented (2026-10-13, steps 1 and 2) |
| Requires | DEP 10, DEP 11 |
| Created | 2026-10-12 |
| Version | Unreleased |
| Replaces | DEP 6 |
The design landed in the data model, the schema, the record format, the RecordLike protocol and WorkingRecord. Where they and this DEP differ, follow those pages.
Abstract¶
Properties, which classify or describe one label of a dim, get their own section of the manifest, apart from attributes, the data. The entity-type axis becomes an Enum property of entity, whose values are categories that per-category defaults and traits key on. entity becomes a dim like any other, listed in partial, and core names neither entity nor entity_type.
Motivation¶
Where a value is read depends on how many coordinates address it, not on what it is. p_nom and carrier both declare dims: [entity], so both land on the entity axis, and record.attributes returns neither. Widening p_nom over scenario is compatible, yet moves it to inputs/p_nom.parquet, away from the rows a layer wrote before.
The entity-type axis is a special case at every level. It is a dim plus a group into it whose rows are the entity axis file, a reserved name, and two entity-axis layouts. entity is too: always in the fold key, never broadcast, the only dim whose tombstones cascade, and the key of the read API.
The schema also cannot state a framework's per-type defaults:
| Name | Default per type |
|---|---|
sign |
1 for Generator, Store, StorageUnit, -1 for Load, ShuntImpedance |
carrier |
"AC" for Bus, "" otherwise |
type |
"primary_energy" for GlobalConstraint, "" otherwise |
p_min_pu |
-1 for StorageUnit, 0 otherwise |
p_set |
0 for Load, NaN otherwise |
AttributeSpec.default is one value, so a tool picks one and writes the others out as data, or loses them. None of PyPSA's 142 input names differs in dtype between types, only in default.
Specification¶
- A property is one value per label of its subject dim, declared under
properties.<dim>, with nodims. An attribute is data and keepsdims, which may list one coordinate. The author declares which a name is. - One name has one spec across every section. A name declared as a property and an attribute, or as a property of two subjects, is refused. So is a property named like a dim or a group.
- An
Enumproperty of a dim inpartialcategorises its subject, and its values are categories.Trait.onanddefault_bymay refer to it, and datarecord refuses any other property there. A dim may have several. Schema.categorizing(dim)lists them.attributes_for(dim, **categories)anddefault_for(name, **categories)answer per label.default_bymaps a categorising property to per-category defaults, on properties and attributes alike. For an unset value, a read takes the first key, in declaration order, whose map lists the label's category, otherwisedefault. Each key categorises a dim that the value applies to, directly or through a group field. A category outside theEnumis refused. Only the default varies.Trait.onis one categorising property. A trait narrows and does not grant, as DEP 16 states. An attribute that no trait claims applies to every label, and a claimed one applies to every label that some claiming trait matches. A switch is addressed by exactly the dim its trait's property categorises, and a trait with a switch must have anon.- A NULL on a categorised dim applies only to the labels that carry the attribute, judged by their categories at read time.
- The fold key is exactly
partial. Every dim and group broadcasts. Deletion follows DEP 11. A label is unique across types by construction, one axis row per label. Record,Resolver,FoldandLayerDataloseentity_types,entity_type(ctype)andentity_axis, andflags()takes no type. A per-category frame isdims[dim]filtered on a categorising column.- Every property of a dim is a column of
dims/<dim>.parquet, NULL where the category does not have it.dims/entity_type/goes. WorkingRecordhasadd(dim, labels, **columns),set(attribute, value, **coords)andremove(dim, labels).addtakes only columns the axis file has, and anEnumvalue must be a category.settakes every coordinate by keyword, a group's fields included.setwrites a property, a category included, as one column of the axis row. A property keyword selects the matching labels whensetruns, and nothing of the filter is stored. Sosetdoes not apply the value to a label added later. A plain NULL anddefault_byare the two ways to give a value to labels not yet added.- A category may change. Values for attributes the new category does not carry stay stored, outside every per-category read and broadcast.
- Step 2: a property lands in
dims/<dim>.parquetand every attribute ininputs/<attr>.parquet. An axis file has its labels,withinkeys, tombstones and properties only.record.properties[dim]returns stored values,record.attributes["p_nom"]returns long rows, andattributes_forsplits intoproperties_forandattributes_for. Properties of a group and per-category flags are step 2 too.
Rationale¶
Enum makes the categories a trait or default lists checkable against a vocabulary. This settles the String-typed axis question of DEP 6: a set of categories is an Enum, or nothing keys on it. The dim must be in partial, because only a dim in partial is in the owner key and checked per label.
"Category" is used because "label" already means an axis value. A category's frame is a filter and a projection over one file. Per-type files would state each label's category twice, and Parquet stores an all-NULL chunk for almost nothing.
switch is not subsumed. A category never varies along another dim, while p_nom_extendable per investment_period does. Lifting DEP 3's restriction to the categorised dim is step 2.
The design has these costs:
- No per-type flags, frames or applied defaults before step 2, and PyPSA's
verifycounts an attribute of one type as one of every type. - A schema keying on
entity's categories must listentityinpartial, or is refused. - Adding a carrier to an
Enumcarrieris a schema change. - A read of entity-only data goes through the fold. A static frame of forty columns is forty attribute reads and a pivot.
- Nothing derives whether a name is metadata or data, and moving it between sections is incompatible.
- A value per entity type alone, such as an
icon, has no file keyed by category. - Every manifest is rewritten and every record re-exported.
Usage¶
properties:
entity:
entity_type: { dtype: { Enum: [Bus, Generator, Load, StorageUnit] } }
sign: { dtype: Float64, default: 1, default_by: { entity_type: { Load: -1 } } }
attributes:
p_min_pu:
dtype: Float64
dims: [entity, snapshot]
default: 0
default_by: { entity_type: { StorageUnit: -1 } }
partial: [entity, scenario, connection]
w.add("entity", ["g1", "g2"], entity_type="Generator", carrier="wind")
w.set("p_max_pu", 0.9) # every generator, now and later
w.set("efficiency", 0.7, entity="l1", bus="b0")
w.remove("entity", ["g1"]) # tombstones g1 and its connections
Implementation notes¶
Steps 1 and 2 landed on feat/properties-structs, with two departures:
partialhas no rule for groups. A dim a group draws on need not be inpartial, because its dead labels cover deletion.- A reader applies
default_bythroughSchema.default_for, as the PyPSA tool does.record.propertiesandrecord.attributesshow stored rows.
The checks: no entity or entity_type in core outside tools/, docstrings and tests. Records with and without categories differ on disk by one column. A bus_type category works like entity_type, and two categorising properties of one dim combine. set("p_max_pu", 0.9) applies to a later generator and to no bus. PyPSA round-trips sign and p_min_pu with nothing written. Widening p_nom over scenario reads old values in every scenario.
Open issues¶
members(dim, **categories), a per-category read with defaults applied. Deferred until a caller needs it.- Categories in attribute rows, to be decided. An
inputs/row would contain a categorising property, soentityNULL withentity_type = Generatorapplies to generators added later. It needs a specificity rule within a layer, explicit label over category row over plain NULL, with two rows of equal specificity on one label refused at staging. A read would take such a row overdefault_bywhere the row applies.flagswould count it as broadcast or gain a third state. A group drawing twice on one categorised dim is refused. The decision waits for a case where layer data must apply to every label of a category, future ones included. - Whether a functional group over one coordinate, such as
countryover[bus], is a property. - Whether wide-frame editing returns over
add,add_groupandset. - Whether a broadcast survives materialisation. A materialised owner map stores a NULL already expanded, so
set("p_max_pu", 0.9)misses a generator added in a child. AWorkingRecord's committed prefix behaves the same. - Whether groups edit like dims, through
add(name, keys, **columns).