DEP 15: a dim is flat¶
| DEP | 15 |
|---|---|
| Title | a dim is flat |
| Author | Jonas Hoersch |
| Status | Implemented (2026-10-20) |
| Requires | DEP 13 |
| Created | 2026-10-19 |
| Version | Unreleased |
The design landed in the schema, the data model, the fold and the tools reference. Where they and this DEP differ, follow those pages.
Abstract¶
Dimension.within leaves core, so every dim is one label set, keyed by its label alone. A label scoped to a parent becomes a label unique across its dim, plus a property into the parent, such as snapshot.period. DEP 13 added this reference. A chain is a chain of such properties, and several parents are several properties.
Motivation¶
A multi-period PyPSA network declares snapshot within period, and every place that touches the snapshot axis has to handle it:
- Every series carries its period. Every input and result over
snapshotis overperiodtoo, sop_max_puis keyed by(entity, period, snapshot). A NULLperiodis then a second way of addressing a snapshot. - Checks skip a nested dim.
removeand the writer do not check a propertyintoone, or a group field over one, andremovedoes not cascade to its tuples. The trait check passes over a nestedpartialdim, andset("dim.name", mapping)refuses one. - Two bugs follow. datarecord accepts an attribute over a nested dim without its parent, so its rows match no point. A pointwise
setwith a NULLperioddrops the rest of the series. - The framework has no nested dims. In math-spec, the model-spec language datarecord feeds, every dimension has one list of labels. A multi-period snapshot there is one flat dim with a
snapshot → periodrelation. Its guidance states that two dependent dimensions, declared as a product cut back by a mask, are one relation. A tool that feeds it flattens the(period, snapshot)key anyway.
Specification¶
One key column¶
Dimension has dtype, unit and description. A dim's key is its label: dims/<dim>.parquet is keyed by the one column <dim>, and record.dims[dim] is that column. Its labels are unique across the dim. datarecord refuses a manifest that declares within at load, and the error names the property into the parent that replaces it.
A label scoped per parent becomes a flat label and a property into the parent. The source picks the labels, which need only be unique across the dim. Where the point's name inside its parent matters, it is a further property.
| Declared before | Declared under this DEP |
|---|---|
timestep within period |
snapshot, snapshot.period into period, snapshot.timestep |
timestep in period in horizon |
snapshot.period into period, period.horizon into horizon |
timestep within {period, stage} |
snapshot.period into period, snapshot.stage into stage |
A chain is not denormalised, as for any classification. snapshot has no horizon column, and the horizon is a join through period.
A selector keyword selects by a parent. The keyword is bare and matches any property of a dim that the attribute's coordinates cover. period=2030 on p_max_pu selects the snapshots whose snapshot.period is 2030, and timestep=t0 the snapshot at t0 in every period. datarecord still refuses a keyword when two such dims both declare period.
What loses its special case¶
| Where | Before, for a nested dim | Under this DEP |
|---|---|---|
axis key, record.dims[dim] |
(*parents, dim), Schema.axis_key |
dim |
| axis fold, dead labels | keyed by the parents too | by the label |
| broadcast | a series has its parent, and a NULL parent means every period | a NULL dim means every label |
add(dim, …) |
parent keys required | properties only |
set("dim.name", mapping) |
refused | as on any dim |
remove, the writer |
no into check, no group cascade, no field check |
every dim |
label and trait check on set |
skipped for a partial dim |
every partial dim |
| property names | none named like a within parent |
snapshot.period allowed |
| versioning | changing within incompatible |
the entry goes |
The PyPSA tool¶
- A single-period network's schema is unchanged.
- A multi-period network's
snapshotis flat. Its label is PyPSA's timestep where the timesteps are unique across periods, and an integer position otherwise. snapshot.periodisintoperiod. The tool declaressnapshot.timesteponly beside positions, because a timestep label already contains it.- Inputs and results over
snapshotare no longer overperiod.period_weighting_objectiveand_yearsstay overperiod. buildrebuilds the snapshotMultiIndexin axis order fromsnapshot.periodand the label orsnapshot.timestep.
Rationale¶
The scoping comes from how the source picks its labels, not from the axis. t1 identifies no snapshot alone because PyPSA reuses it per period, and a source that numbers its snapshots once has no nesting to declare. A property records the same relation in the one form that datarecord already checks, folds and selects on. A reader then reads snapshot.period as it reads bus.carrier.
With one key column, datarecord needs one rule. The axis key, the fold key and the broadcast rule each had a branch for (*parents, dim). Every check that could not place a label by one column skipped it. Removing the shape removes the skips. The bugs above become unrepresentable. When no dim has a parent, no attribute omits one and no series leaves one NULL.
into can now express timestep. The distinction between within and into held because within scoped a label set and into could not. With a flat label set, the snapshot is a classification like any other, and each snapshot has one period.
The NULL-parent set bug is narrower than within. It is a fold bug with any two dims outside partial, which within made routine by putting a second such dim on every series. The bug goes from the snapshot and stays for other schemas.
The costs: a source whose inner labels repeat across parents must make unique ones. datarecord can no longer read a record that declares within, so each is re-exported, as breaking changes are free. "The same timestep in every period" is no longer one row with a NULL period. It is a selector over snapshot.timestep, which stages one row per matching snapshot. To get (period, timestep) per row, a reader joins the snapshot's properties.
Usage¶
schema = Schema(
dimensions={
"entity": Dimension(dtype=nw.String()),
"period": Dimension(dtype=nw.Int64()),
"snapshot": Dimension(dtype=nw.Int64()),
},
properties={
"snapshot": {
"period": PropertySpec(into="period"),
"timestep": PropertySpec(dtype=nw.Datetime("us")),
}
},
attributes={
"p_max_pu": AttributeSpec(dtype=nw.Float64(), dims={"entity", "snapshot"})
},
)
w.add("period", [2030, 2040])
w.add(
"snapshot", [0, 1, 2, 3], period=[2030, 2030, 2040, 2040], timestep=[t0, t1, t0, t1]
)
w.set("p_max_pu", 0.8, entity="g1", period=2030) # snapshots 0 and 1
w.set("p_max_pu", 0.5, entity="g1", timestep=t0) # snapshots 0 and 2
w.set("snapshot.period", {3: 2030}) # was refused on a nested dim
w.remove("period", [2040]) # refused while a snapshot.period is 2040
Implementation notes¶
- Schema:
Dimension.within, its validation,_ancestorsandaxis_keygo, and callers key by the dim. Awithinin a manifest is refused withintonamed. The property-name and versioning rules losewithin. - Read: the axis fold, dead labels and the
dimsandpropertiesframes are keyed by the label. - Writer: an axis file's columns are the dim and its properties. Every
intoproperty and group field is checked against live labels. WorkingRecord:addwithout parent keys,seton any dim's property, andremove's cascade,intorefusal, trait check and staging keys over every dim.- Tool: a flat
snapshotwithsnapshot.periodandsnapshot.timestep, series oversnapshotalone, andbuildfrom the properties. - Pages, usage pages, changelog and tests. The links of DEP 12 and DEP 13 to the removed
withinanchors are retargeted, which revises no argument.
Checks: no within or axis_key in src/. A multi-period network with repeating timesteps and one with unique ones round-trip. remove and the writer both refuse remove("period", …) while a snapshot's snapshot.period is the label. A manifest that declares within is refused with into named.
Rejected ideas¶
- A group over
(period, timestep)as the snapshot: a masked product, which math-spec's guidance rejects, and a struct-valued key in every time series. withinkept beside the property: two spellings of one relation, and every special case above stays for the one that keeps it.- Composite string labels as a rule (
"2030|t1"): the source picks its labels, and a rule would make datarecord parse them.
Open issues¶
- Whether
Schemashould offer a helper that declares a dim with its parent property in one call. - Whether the selector needs a qualified form when two dims that the attribute's coordinates cover declare
period. - The NULL-parent pointwise
setbug, for schemas with two dims outsidepartial.