DEP 2: groups, one construct for classification and sparsity¶
| DEP | 2 |
|---|---|
| Title | groups, one construct for classification and sparsity |
| Author | Jonas Hörsch |
| Status | Superseded (2026-10-26, implemented 2026-09-01) |
| Created | 2026-08-27 |
| Version | v0.1.0a1 |
| Superseded-By | DEP 23 (its groups, which become multi-dims) |
The design landed in the data model and the record format. Where they and this DEP differ, follow those pages.
Abstract¶
A group with an optional into replaces both the group and the on mapping of DEP 1. Every group is one file, groups/<name>.parquet. into declares that each tuple of over has one label of a declared dim. The group is then checkable, and a reader can aggregate along it with by=.
Motivation¶
The schema has two ways to declare "which tuples over dims exist". You state it directly with a Group, or with Dimension(on=...) for the case where each label appears once. The two are stored, addressed, aggregated and validated by different code, for the difference of one uniqueness constraint.
A modeller pays for this when writing maths. Summing along a mapping is GROUP BY country, while to sum along a group you must say which coordinate survives. GAMS, Calliope and PyPSA all use one construct for both, so the split is datarecord's and not the domain's.
Specification¶
groupsabsorbsDimension.on.Dimension.on,Dimension.mappingandSchema.mappings_onare deleted.overdeclares the coordinates, as a list where coordinate names equal dim names, or as a dict where they do not. The list is sugar for the dict.intois the name of a declared dim. Each tuple ofoverhas exactly one label ofinto, checked on write.intomay sit over several coordinates, such asover: [bus, scenario].- Without
into, a group is a set of sparse tuples, is not a dim and never appears as a column. - A group may share its name with a dim. The rule against the two colliding is dropped.
- In an attribute's
dims,Xis the dimXif one is declared, and otherwise the groupXexpanded to its coordinates.intois not consulted. - Every group's rows are
groups/<name>.parquet, keyed by its coordinates. Thedims/<group>/<Type>.parquetsplit and the mapping column on the classified dim's axis file are gone, andSchema.group_columnsno longer leads withentity_type. - A reader collapses coordinates in its maths language with
over=<group>[coords], whose brackets list what goes. The bracket is required on everyover=over a group of several coordinates.by=<group>keeps the group'sintodim and collapses all ofover, and exists only whereintois declared. - A NULL group coordinate still means the group's rows, and a group in
dimsstill expands to its coordinates. The record does not project across levels.withinand traits are unchanged.
Rationale¶
into is the declaration that was missing. Without it, a group can only happen to be single-valued, and a duplicate row is an error the schema has no name for. GAMS and Calliope both leave functional lookups undeclared, so every sum silently counts a duplicated label twice. datarecord's schema is meant to make the shape of data checkable, so datarecord should close that gap.
into must be a declared dim because a functional group needs the axis file. It gives the labels an order and a place for a per-country CO2 budget. The argument for the axis file in DEP 1 stands unchanged.
Sharing a name is the natural spelling for a group with into. dims: [country] resolves to the dim and by=country to the group, so a rule against sharing would force a name like country_map. datarecord resolves dims first when it addresses an attribute, so the case without into is not ambiguous either. The cost is that a shadowed group without into loses its only spelling in dims without an error.
group stays the word, because country groups buses too. GROUP BY, PyPSA's group_constraints and Calliope's group overload it, so prose about aggregation says which is meant.
One file per group gives the no-denormalisation property of DEP 1 for free: a layer that restates a group restates one file. The <Type> split put a Link's and a Line's connection rows in two files with identical keys. It had no type to split on for corridor or contract.
A relation between two entities, such as a contract between a seller and a buyer, gets a file, an aggregation spelling and an optional into. over: [entity], into: parent declares a hierarchy of components, which Dimension.on could not express.
by= and over= are two keywords because they mean opposite things. A bare over=country read as "keep country" would give one keyword two senses. The bracket is required because a functional group needs it wherever a coordinate must survive that by= would collapse: over=country[bus] on over: [bus, scenario] keeps a per-scenario country total. Inference from the enclosing scope also fails under nesting. In sum(sum(x, over=connection), over=siting), the inner sum would collapse bus, which the outer sum needs, and return the right shape with the wrong number.
The change needs a rename across the tree and a format migration with no reader of both layouts. Reading a bus's country becomes a join instead of a column read. WorkingRecord's kind= and the CONNECTION constant in mutable.py stop naming one group specially.
Usage¶
groups:
country:
over: [bus]
into: country
connection:
over: [entity, bus]
corridor:
over: { from: bus, to: bus }
attributes:
efficiency:
dims: [connection, timestep] # entity | bus | timestep
co2_budget:
dims: [country] # the dim
sum(x, over=connection[entity]) keeps bus, and sum(x, by=country) keeps country.
Implementation notes¶
intois sugar resolved at parse:Group(over=["bus"], into="country")returns["bus", "country"]fromcoordinates. The file layout, the fold's key anddimsexpansion readcoordinatesand never branch oninto.- The model keeps
intofor the uniqueness constraint, whose key iscoordinatesminusinto, forby=, and to write the manifest back as authored. - A validator beside the
overcheck refuses anintothat is no declared dim. Otherwisedims: [country]would expand silently tobus | country. The collision check goes.
Rejected ideas¶
- A group whose
intois a label set of its own: it loses the axis file. - Keeping the rule against a group sharing a dim's name: a lint naming the shadowed group is the better answer.
- Restricting
intoto one coordinate: needed only if a functional group were a column on an axis file, which is not kept. - Overloading
over=for "keep": one keyword would have two senses. - Inferring the bracket from the enclosing scope: wrong under nesting.
Open issues¶
- Whether a shadowed group warrants a lint at load.
- Whether a group's rows are checked against the labels of its coordinate dims, as GAMS does for a relation.
- Whether every aggregation a reader needs factors into a declared dependency plus a reader's operation. This is untested against a second reader.