DEP 11: deletion cascades through tombstones, not through filters¶
| DEP | 11 |
|---|---|
| Title | deletion cascades through tombstones, not through filters |
| Author | Jonas Hörsch |
| Status | Implemented (2026-10-13) |
| Requires | DEP 10 |
| Created | 2026-10-13 |
| Version | Unreleased |
| Superseded-By | DEP 20 (its removal by tombstone, and dead labels as winning tombstones) |
The design landed in layers and the fold. Where they and this DEP differ, follow those pages.
Abstract¶
A label is removed by a tombstone and only by a tombstone. Removing one tombstones every group tuple that contains it, and a layer that leaves such a tuple alive is refused. A read infers nothing from a dead label that a row refers to. This replaces, for group tuples, the rule of DEP 10 that the tuples stand, dangling.
Motivation¶
Removing an entity drops its values in four places, each covering part of the problem:
| Where | What it drops | Keyed on |
|---|---|---|
fold_inputs |
parent owner-map rows hit by this layer's tombstones on entity, groups, partial dims |
owner-key columns, and entity fields through entity_coordinates |
_living (read) |
value rows that set a removed entity on a coordinate outside the owner key | the resolved entity axis, through entity_coordinates |
WorkingRecord._collapsed_inputs |
staged rows hit by this layer's own entity and tuple tombstones | _tombstoned and entity_coordinates |
resolve_groups |
only the group's own tombstones | none |
After remove("Generator", ["g1"]), the value x(g1, bus0) is gone, but the tuple (g1, bus0) still stands in record.groups["connection"]. Every path also goes through entity_coordinates and Schema.entity_fields, which hard-code one dim, so removing a bus label removes nothing inside a struct.
Specification¶
- Removing a label writes the cascade.
WorkingRecord.remove(ctype, names)tombstones the entities and every group tuple with a field equal to one, read from the resolved groups. The fold applies a group's own tombstones to it, soresolve_groupsneeds no filter. write_layerrefuses a dangling tuple. A layer that leaves alive a tuple containing a label dead after the layer, or adds a tuple containing one, is refused. A layer written by hand carries its own cascade.- The fold drops keyed values.
fold_inputsdrops each parent row with a tombstoned label on a coordinate in the owner key. That key isentityand the dims and groups inpartial. A re-added label starts fresh. Onentity,fold_inputschecks the top-levelentityonly, and the tuple tombstones of rule 1 cover a group's entity fields. - The read drops unkeyed values against dead labels. For each coordinate outside the owner key, the read drops each value row with a label for which the fold keeps a tombstone. For a group, those are its dead tuples. The read skips this step when there are no such labels.
partial stays the granularity of an override. It is not transitive: a group that draws on a dim in partial need not be in it.
A materialised node has its dead labels in dead/<coord>.parquet, beside dims/ and groups/. It does so for each dim and group outside partial, and for each dim a group draws on. Rule 4 reads the base's dead labels plus those of the layers above. The resolved axes and groups stay tombstone-free, and fold_axis returns the dead labels as a by-product.
_living's filter, the entity-field anti-joins in fold_inputs and _collapsed_inputs, and every literal entity in the cascade go. An anti-join is a query that keeps the rows with no match in another table. entity_coordinates and Schema.entity_fields stay for Fold.flags(ctype) and for set's _keyword_entities.
Rationale¶
Dead labels must survive materialisation. Suppose layer 1 sets x(g1, bus0) with connection outside partial, layer 2 removes g1, and layer 3 is materialised. A read at layer 4 takes x's owner, layer 1, from the cached map. The dead labels of the layers above the base do not include (g1, bus0), so the value would come back. The fold cannot cut it earlier, because along an unkeyed coordinate the owner map has one row per attribute, not per label.
Re-adding a label revives its unkeyed values. Suppose layer 3 re-adds g1 and (g1, bus0). The owner map still lists layer 1 for x, so x(g1, bus0) returns, as does every unkeyed attribute over the tuple. That is what a coordinate outside partial means, To have a re-added label start fresh, you declare the group in partial.
The design has three costs:
write_layerdepends on the ancestry. Rule 2 is a fold plus a semi-join (a query that keeps the rows that do have a match) per group field.- Removing a bus removes its connections, and PyPSA's
buildreports the component'sbusas missing. - The resolved cache grows by a dead-labels file per dim and group outside
partial.
Implementation notes¶
- Rule 1 is
WorkingRecord.remove. A dim label other than an entity is removed only by a layer written directly. - Rule 2 runs in
write_layerfor a layer that writes any axis or group, folding the parent's sources and the staged layer. It checks only the tuples the layer adds and those still containing a label it removes.write_layerdoes not refuse a later write because an ancestor left a tuple dangling. - Rule 4's anti-join is
_living, anddead/contains the coordinatesresolve.mortalreturns.
Rejected ideas¶
- A semi-join against the live labels, which the cache already has. It needs no new file, but it runs on every read, where the anti-join against a usually empty set is skipped.
Open issues¶
- Whether a nested dim follows rule 1.
fold_axiskeys a nested dim by(period, timestep), so a parent's tombstone does not apply to its children. Removingperiod=2030would tombstone(2030, t)for everyt. Not built, becauseWorkingRecordremoves no dim label and rule 2 checks no field over a nested dim.