DEP 23: multi-dims replace groups¶
| DEP | 23 |
|---|---|
| Title | multi-dims replace groups |
| Author | Bryn Pickering |
| Status | Implemented (2026-10-26) |
| Requires | DEP 10, DEP 12, DEP 13, DEP 18, DEP 20 |
| Created | 2026-10-21 |
| Version | Unreleased |
| Replaces | the groups of DEP 2 and DEP 10 |
The design landed in the data model, the schema, editing and the record format. Where they and this DEP differ, follow those pages.
Abstract¶
A group becomes a multi-dim: a dim declared under dimensions with over, whose labels are tuples of other dims' labels. It has its own table in dims/, and its coordinates are addressed as connection.bus, as properties are. A functional group's property {one_of: <dim>} becomes a property of the multi-dim, unchanged. The groups section and every public name with group in it go.
Motivation¶
A group already behaves as a dim does. The amendment to DEP 10 found that the fold treats it as a dim in every way. It is one struct column, broadcasts, is in partial or outside it, and has properties. DEP 20 made a layer restate its file whole, as for an axis file. Only its spelling still differs from a dim's, in each of these places:
| A dim | A group |
|---|---|
declared under dimensions |
declared under groups |
dims/bus.parquet |
groups/connection.parquet |
add, remove |
add_group, remove_group |
Schema.dimensions |
Schema.groups, group_columns, group_key, groups_of |
dims: [X] resolves a dim first |
dims: [X] resolves a group if no dim has the name |
a property {one_of: <dim>} may categorise it |
never categorised |
So every reader looks up a name as a dim and then as a group, and every rule for both is stated twice.
Specification¶
A multi-dim¶
A multi-dim is a dim whose labels are tuples of other dims' labels. It is declared under dimensions with over, as a group was, and has no dtype:
dimensions:
entity: { dtype: str }
bus: { dtype: str }
snapshot: { dtype: datetime }
role: { one_of: [input, output, attached] }
connection: { over: { entity: entity, bus: bus } }
corridor: { over: { from: bus, to: bus } }
properties:
connection:
role: { one_of: role }
attributes:
efficiency: { dtype: float, dims: [connection, snapshot], default: 1 }
partial: [entity, bus, connection]
overis a list or a map from coordinate to dim, as a group's was.over: [entity, bus]isover: {entity: entity, bus: bus}, andcorridorhas two coordinates onbus. Each coordinate maps to a declared dim that is not a multi-dim.- datarecord refuses at load a dim that declares
overbesidedtypeorone_of. A dim withoutoverkeeps its defaultdtype: str. - Its table is
dims/connection.parquet, keyed by its struct column, with its properties as columns:connection STRUCT(entity, bus) | role. A layer that changes its tuples or their properties writes it whole and in order. Dimension.overhas the map, andSchema.dimensionshas every dim, multi or not.
What DEP 10 settled¶
- A row addresses a multi-dim by one struct column named for it, with a field per coordinate in
overorder. - It may be listed in
partial, and is then in the fold key. Otherwise it is outsidepartialand in no key. - A value is a whole tuple or NULL. datarecord refuses a struct with some fields NULL in
set, inaddand in the writer. - A NULL broadcasts over its live tuples, expanded against
dims/connection.parquetas a NULLbusis againstdims/bus.parquet. - datarecord refuses an attribute over a multi-dim and a dim named like one of its coordinates, such as
dims: [connection, bus], becausebus=would address both. A coordinate onbusunder another name, such asport, may sit besidebus. - Removing a coordinate's label cascades:
remove("bus", ["south"])removes every tuple that containssouth.
Coordinates are addressed as properties are¶
connection.bus is the bus of each tuple, wherever DEP 13 lets a dotted name stand, Schema.resolve included. A coordinate is part of the key and not a property column, so datarecord refuses set("connection.bus", …) and names remove and add. datarecord resolves a default_by keyed on entity.ctype through the coordinate entity.
- A selector keyword stays bare.
bus="north"on an attribute overconnectionselects the tuples whosebusisnorth, coordinate first. - datarecord still refuses a property of a multi-dim named like one of its coordinates.
- A multi-dim is never categorised. A property of it may refer to a dim with
one_of, asroledoes, and still categorises nothing, whetherpartiallists the multi-dim or not. datarecord refuses adefault_bykeyed on one.
A functional group is a property of a multi-dim¶
A group property {one_of: <dim>}, which DEP 18 put in place of a group's into, becomes a property of the multi-dim, declared the same way:
dimensions:
plant_period: { over: [plant, period] } # was under `groups`
properties:
plant_period:
zone: { one_of: zone }
A property has one value per label of its subject, so one zone per tuple is a property. The live-label check, the remove refusal and NULL meaning none are those of any property that refers to a dim.
Groups go¶
The groups section then has nothing left and goes. Group, Schema.groups, Schema.group_columns, Schema.groups_of, Schema.group_key, add_group, remove_group and the groups/ directory go, and dims: [X] resolves a dim and nothing else. A manifest with groups is refused as any unknown key is.
add and remove take a multi-dim's tuples as any frame DuckDB scans, with the struct column named for the multi-dim. That column is what record.dims["connection"] returns, so a read goes back in as it stands. A frame may instead have a column per coordinate, and a list of mappings from coordinate to label becomes that frame first. Every form then takes the same path and type checks. datarecord refuses a frame with both the struct column and a coordinate column as ambiguous:
w.add("connection", other.dims["connection"]) # another record's tuples, as read
w.add("connection", [{"entity": "g1", "bus": "north"}], role="output")
w.add("connection", pd.DataFrame({"entity": ["g2"], "bus": ["south"]}))
w.remove("connection", [{"entity": "g1", "bus": "north"}])
A property is a keyword, as for any dim, or a further column of the frame. datarecord refuses a tuple that contains a label missing from its coordinate's dim.
Rationale¶
A tuple set that is a dim in dims and a relation elsewhere has two kinds, and the caller picks one at each use. Declared under dimensions, it has one kind, with a table for its properties, and add and remove treat it as any dim.
dims/ already has one table per dim, keyed by that dim's column and restated whole. A multi-dim's table is keyed by its struct column, so a reader of the format looks in one place.
What it costs: every manifest with groups is rewritten, and every record re-exported, as groups/ moves to dims/. Every public name with group in it goes, and callers of add_group and remove_group call add and remove.
Usage¶
w.set("efficiency", 0.9, entity="g1", bus="north") # selector: the coordinates
w.set("connection.role", "input", entity="g1", bus="north")
w.set("connection.bus", "south", entity="g1") # refused: a coordinate is the key
Implementation notes¶
Dimension.over, and the group struct-column machinery keyed on the dim.dims/<multi-dim>.parquet, withgroups/andresolved/groups/dropped.addandremovetake tuples,Schema.resolveresolves a coordinate, and the group names go.- The PyPSA tool declares
connectionunderdimensions, and fixtures are re-exported. - Pages, usage pages and changelog.
Checks: the connection, broadcast, partial and cascade tests pass with connection under dimensions. The word group appears in no public name, and on no live page outside the DEPs. A PyPSA network round-trips with connection a multi-dim and role its property. datarecord refuses a zone that is not a live label on set, and refuses removing a zone that a value still refers to.
Rejected ideas¶
- A relation usable as a dim in
dims: [connection]: a tuple set becomes a relation or a dim by where it is used. - No multi-dims at all, with
dims: [entity, bus]and validity by a condition: this loses the struct column, the whole-or-NULL broadcast and the table forrole. - A surrogate port dim with properties
{one_of: entity}and{one_of: bus}: a value is written against a port label, and the tuple stops being the address.
Open issues¶
- Whether a multi-dim may be
overanother multi-dim. Nothing needs it yet, so datarecord refuses it until something does. It is on the open questions page. - Whether a functional mapping over two coordinates justifies a dim.
plant_periodexists only to havezone, and its labels must be written before the property can be. It is on the open questions page. - Whether a multi-dim in
partialmay be categorised, now that the open question on a group's categories becomes one on multi-dims. This DEP keeps a multi-dim uncategorised, and a later DEP may lift that.