DEP 26: the fold reads each layer under its own schema¶
| DEP | 26 |
|---|---|
| Title | the fold reads each layer under its own schema |
| Author | Jonas Hörsch |
| Status | Implemented (2026-11-03) |
| Requires | DEP 25 |
| Created | 2026-10-30 |
The design landed in resolution, schema, store, editing and open questions. Where they and this DEP differ, follow those pages.
Abstract¶
The fold reads each layer under the schema in that layer's manifest: its partial and its attributes' dims. A revision's owner map then depends on its ancestry alone, so its fold table and its materialised map stay valid whatever a descendant declares. Where a deeper layer's schema is wider, the fold first extends the map folded so far to it.
Motivation¶
Before this DEP, every layer of an ancestry folds under one schema, the record's current one. Adding a dim to partial changes the owner map's key, so no materialised map or fold table keyed before the change stays usable. Every revision below the change then refolds its ancestry from the root, which materialisation exists to avoid. The workspace keeps one fold table per key for the same revision.
Widening an attribute's dims by a dim already in partial keeps the key, so a map folded before the change stays in use. Its rows for that attribute have varies and broadcast both false along the dim. The read's join then finds no stored key, and the attribute reads empty (#117).
With DEP 25, each layer has the full schema it was written under, and compatible_with holds along every ancestry. So partial and each attribute's dims only grow from a layer to its child, and the fold can read each layer as it was written.
Specification¶
Each layer under its own schema¶
- A layer's owner-map rows are keyed by its own schema's
partialdims, and its flags follow its own attributes'dims. Every source in a fold has its own layer's schema. - A revision's owner map is keyed by that revision's schema, and its read resolves under it.
- The fold table is
owner_map_<id>, with no key in its name. A materialisedowner_map.parquetalways stays usable. read_owner_map,materialisedandsources_to_readlose their schema parameter, andLayerSource.materialisedits schema argument.
Extending the map to a wider schema¶
Where a layer's schema is wider than the schema of its parent's map M, the fold extends M before folding the layer onto it. A map folded live and a materialised seed extend the same way. Two changes need an extension, whether or not d is new to the schema:
| Change in the layer's schema | Extension of M |
|---|---|
d added to an attribute's dims |
the field d in both flag structs, with broadcast.d set on the attribute's rows, which are NULL along d |
d joins partial |
a NULL key column for rows not varying along d, and a split of the others |
The first extension marks in the flags that the rows broadcast along d, so the read joins them as broadcast and flags() reports them. No map then lacks a flag field that its revision's schema declares. Two consecutive layers with one schema extend nothing.
The split divides a row with varies.d, an overlap row included, by the labels that its assigned layer wrote along d at that key:
- One join of
M's varying rows against the files of the layers they are assigned to gives one row per label, keyed as stored and NULL-safe on the old key. Where the layer also wrote the key NULL alongd, it gives one more row, NULL there. A multi-dim's labels are its tuples. - The fold recomputes each split row's flags per new key, with the aggregate a layer's own rows use.
- The fold drops each split row with a label of
dremoved atM's revision. The read hid those rows there, and oncedis a key column, no read will. - While
dis outsidepartial, the fold keeps the labels each restatement removed alongd, seeded from a materialised base, minus the parent's axis. Usually there are none, and the fold skips this drop. - From the layer on, the fold prunes
dby the layer's removals as it does anypartialdim. - Where
bool_or(varies.d)overMis false, as for a dim new to the schema, the projection alone is the extension.
No other compatible change affects the fold. To refuse a crossing along a dim outside partial, the fold judges the layer's rows against the extended map, under the layer's schema.
Deleting a value along a dim outside partial¶
A layer that restates a key along a dim outside partial and omits a label its parent had there deletes that value. A later widening of partial no longer brings it back. A WorkingRecord edit fills the untouched coordinates of the key. datarecord makes each WorkingRecord edit carry a label's whole extent, but a layer written another way need not.
Rationale¶
A layer's rows have the meaning that its own schema gives them. Only folding a layer under another schema forces datarecord to key a cache by schema. Read per layer, a revision's map is a function of write-once layers, so a cache is valid forever, with no key to check.
The split is needed because a NULL key column does not mean the same for every row. Take a root under partial = {entity} writing (gas, low) 1 and (gas, high) 2, and a child under {entity, scenario} writing (gas, high) 3. Extended with NULL, the root's map row becomes (gas, NULL), and its varies.scenario is still true. The read joins it only to the root's rows stored NULL along scenario, of which there are none, so (gas, low) is lost. Split, the owner map assigns (gas, low) and (gas, high) to the root, and the child's row covers the second. The read gives (gas, low) 1 and (gas, high) 3.
Recomputed flags keep each split row joined to the file rows its own key matches. A split row that inherited both varies.d and broadcast.d would also join the layer's row NULL along d, and return its label twice.
It costs:
- A fold reads one schema per layer. DEP 25's cache contains it, so a warm read opens no file for it.
- A read under a wider
partialadds a projection, and a split join that reads the attribute files' key columns. It runs once per revision, cached with its fold, instead of a refold of the ancestry. - Adding a long axis such as
snapshottopartialgrows the map to a row per label where rows vary along it, as a refold does.
Usage¶
Nothing a user writes changes. After a descendant widens the schema, datarecord reads a materialised revision as it is, so rev.materialise() before a widening is no longer wasted.
rev.materialise()
child = store.commit(source, parent=rev) # source declares scenario in partial
child.record.attributes["p_max_pu"] # child folded onto rev's map, extended
Implementation notes¶
- Each source has its layer's schema from its manifest, and
_ownand_overkey and flag by it. _fold_ontoextends the map wherever the schema widens, with the split and the drop of rows with removed labels._as_storedand_ownerslose their defaults for a missing flag field.- The fold table becomes
owner_map_<id>, and the schema parameter and the key check leaveread_owner_map,materialisedandsources_to_read.
Once built, each revision reads the same before and after a descendant widens the schema, with and without materialised maps. The example in the rationale reads (gas, low) 1 and (gas, high) 3. No workspace holds two fold tables for one revision.
Rejected ideas¶
- A NULL key column for every widening: it loses the rows varying along a dim that joins
partial, as in the rationale's example. - Refolding the ancestry under the widest schema: it invalidates every cache below the widening, and reads a layer under a foreign
partial. - Split rows inheriting their parent row's flags: a row that had a key both ways would return its label twice.
Open issues¶
- Whether
WorkingRecordgets an explicit way to delete one value along a dim outsidepartial. Only a hand-written layer can. - Whether a layer may remove a dim
dfrompartial. It would restate, as resolved, every key of its ownpartialat which a parent map row has a label ofd. The fold would projectdout of the parent's map, dropping every row with a label ofd. It would refuse a layer leaving one uncovered.compatible_withwould accept the smallerpartial, and a helper would build such a layer from a revision.