DEP 4: staging as a layer, one fold, two sources¶
| DEP | 4 |
|---|---|
| Title | staging as a layer, one fold, two sources |
| Author | Jonas Hörsch |
| Status | Implemented (2026-09-01) |
| Created | 2026-09-01 |
| Version | v0.1.0a1 |
| Superseded-By | DEP 20 (its per-label axis fold, and partial as how many labels an axis layer carries) |
The design landed in the fold and WorkingRecord. Where they and this DEP differ, follow those pages.
Abstract¶
The fold reads each layer through a LayerSource, so parquet layers, a plain directory and a WorkingRecord's staged edits fold the same way. The staged edits become the last source. The fold is materialised up to the last frozen source and stays a relation after it. pending goes, and result attributes are declared in the schema.
Motivation¶
The editing page says a set of pending edits is a layer, an unwritten one. The code does not deliver that. mutable.py reimplements the algebra of layered/resolve.py three times over, and each copy drifted from its twin:
- The overlay:
fold_inputsdrops each parent row whose key the layer also has, then adds the layer's rows._overlay,_entity_unionand_group_uniondo the same against staged rows. That makes five sites for one rule. - The flags: the fold computes
variesandbroadcastin its ownershipGROUP BY, and_flags_armcomputes them again over the staging tables. - The collapse:
_latest_percollapses staged rows by_seq, and the fold collapses layers by depth.
Specification¶
- A
LayerSourcehands over one layer's own rows as the layer would be written. It haslayer_id,frozen, and one member per file:axes(),axis(dim),entity_type(name),group(name),attribute(name, kind)andall_attributes(kind). A member returnsNonewhere the layer wrote nothing.KindisLiteral["inputs", "outputs"]. entityis an axis, read throughaxis("entity").all_attributesexists only forinputs/andoutputs/, because their files share one key, and returns the union unprojected.attribute(name)reads one file with its own columns, since a padded column is ambiguous under the broadcast rule.axes()lists a directory once, replacing a probe per declared dim.- Everything the fold does to rows stays in the fold: padding, broadcast expansion, the ownership aggregate and
order_key._seqcollapsing and tombstones stay behind the source. ParquetLayer,DirectorySourceandStagedSourceimplement the protocol.NodeCachefolds a list of sources, root first. AWorkingRecordissources + [StagedSource(self)].- The fold is materialised up to the last frozen source, and everything after it stays a relation. Nothing needs invalidating.
NodeCache.dimsbecomes a plain property that caches the frozen prefix's axes. - A staged axis edit merges per column, so two
setcalls on one axis do not blank each other. - As rows are staged,
_complete_owned_wholecarries the parent's rows that complete each key an edit named along the dims not inpartial. A staged row that leaves such a dim NULL is excluded.staged_only()then contains what gets written. WorkingRecord.pendingis removed.Schemadeclares result attributes beside inputs.outputsstill lists what a layer contains, asattributesdoes.long_columns_forloses its fallback, andvalue_hintretires with_value_dtypeand_scalar_dtype. A result row may still carry a component that the record never declared.- A base that is neither layered nor a directory is out of scope, and
_base_revisionrefuses it.
Rationale¶
A written layer's extent along every dim outside partial is already complete. The fold drops each parent row whose key the layer also has, and loses nothing. Without that completion, the fold would drop the base's whole series for one staged point. Completing as rows are staged removes the difference, so the algebra needs no second copy. The completion runs per insert, because it is scoped by the keys an edit named.
Removing pending is cheaper than marking carried rows in every staging table. After completion, a one-value set on a thousand-snapshot attribute would report a thousand pending rows. To see what the record contains, you read it.
Stating the cache rule over frozen, not as a mode, means a second staged layer needs no new code. Truncating at a materialised ancestor becomes building fewer sources.
The design has these costs:
- A protocol in
layered/thatmutable.pyanddirectory.pyimplement structurally.mutable.pykeeps importing fromlayered/only inside function bodies, to avoid a cycle. - A tool must declare a result before attaching it, though PyPSA's
SubNetworkexists only after a solve. The tool declares its result vocabulary up front or amends the schema at solve time, andSchema.compatible_withdecides whether old layers survive. setreads the base the first time it touches an attribute with a dim outsidepartial.- A read with pending edits re-folds the staged step every time.
NodeCache's invariant becomes "nothing up to the lastfrozensource can go stale".
The change deletes about 250 lines from mutable.py, and the _collapsed_* helpers move into StagedSource. It has no format change of its own.
Implementation notes¶
Three changes landed first, each in its own PR:
componentsbecomesentity_typeacross the API and on disk.- Axis discovery by glob in
resolve_dimsandfold_axis. order_keybecomes a struct(depth, row). A layer's numbering no longer depends on its parent's values, which removes a subquery that would re-run on every read with a live tail. A key restated deeper keeps its first(depth, row). DuckDB orders structs by field, soORDER BY order_keyworks unchanged, and a child of a materialised parent takesmax(order_key.depth) + 1. This one is a format change.
Steps 0 to 4 added the identity spike and extended the both fixture to read one record four ways. They moved the completion out of commit, declared results and added LayerSource. Steps 5 to 7 landed in two commits. The build added two things:
ResolvedLayer, a third source for a materialised ancestor, whose axes and owner map come fromresolved/and whose own rows still come from its layer._axis_layermerges per column on an axis inpartialtoo, because the fold keeps a label's whole row from the last layer that wrote it.
Step 8, sharing the flags aggregate with DirectoryRecord, was dropped. With _flags_arm gone, the two aggregates share three bool_or lines, and a helper's parameters would outweigh them.
Two things stay unfixed by choice. attributes_of re-executes under a live tail, bounded by the map. commit still needs a revision, so a WorkingRecord over a DirectorySource is readable but cannot commit.
Over both bases, the tests check that an unstaged WorkingRecord reads like its base and a staged one like its commit. A second set after a read is visible. One more test asserts that the grouping grain of the fold's flags is invisible after attributes_of unions them.
Measured over a 20-layer unmaterialised ancestry of the ac_dc network, best of five:
| Operation | Time |
|---|---|
| deep read, nothing staged | 2.4 ms |
| deep read, one staged edit | 61 ms |
set, first touch of an attribute |
166 ms |
set, again on the same attribute |
16 ms |
set, again, all dims in partial |
10 ms |
directory flags, scanned |
~0 ms |
directory flags, through the fold |
12 ms |
Of the three performance claims, two hold. A read after a set takes one more fold step, but against a cached base, so 61 ms against 2.4 ms. In a repeat set, the completion takes about 5 ms. A WorkingRecord over a directory folds flags where a bare one scans and memoises them. A repeat set first measured 188 ms, because its validation rebuilt name-to-type from member frames. Reading the owner map instead brought it from 157 ms to 10 ms. Building StagedSource's collapses on 50k staged rows over 500 entities and 20 periods takes about 1.4 ms to plan and 4.3 ms to run. tests/test_scaling.py covers none of this.
Rejected ideas¶
- Marking carried rows apart from edited ones to keep
pending: a second row class in every staging table. - A combined read across axes or groups: each folds on its own key.
- A raw-rows member on the source:
_seqis staging state, and the fold must not see it. - A shared
overlay(older, newer, on)helper: no second caller remains. - Synthesising a source from a framework's object: it rebuilds the layer format, and
all_attributeswould materialise the whole record per read.
Open issues¶
- Memoising
StagedSource's collapses on the highest_seqstaged, which is local to the source. - Whether
materialisecould run over a staging area, freezing the tail into the prefix. - Whether the commit path collapses further. DEP 5 made
flattened()equal toselfin every field butentity_type.staged_only()stays open, because an adapter deletes about as much as it adds without a second caller. - Whether a
DirectoryRecordgains a map. DEP 5 resolves this the other way: it deletesDirectoryRecord, and a directory reads through the fold more cheaply.