DEP 8: one interface over two objects: a source returns its own rows, and a fold returns the resolved rows¶
| DEP | 8 |
|---|---|
| Title | one interface over two objects: a source returns its own rows, and a fold returns the resolved rows |
| Author | Jonas Hörsch |
| Status | Superseded (2026-10-21, implemented 2026-09-04) |
| Created | 2026-09-02 |
| Version | v0.1.0a1 |
| Superseded-By | DEP 19 (its fold that is LayerData) |
The design landed in the fold, the RecordLike protocol, layers and the package. Where they and this DEP differ, follow those pages.
Abstract¶
axis(dim) means "the rows of the object I hold" wherever it is written. A LayerSource returns the rows of its own layer, and a fold returns the rows of everything folded into it. ResolvedLayer is deleted and NodeCache becomes Resolver. A materialised node becomes a Fold that the fold finds among its sources. write_record takes LayerData, the interface a source and a Resolver share.
Motivation¶
ResolvedLayer is a persisted NodeCache: Revision.materialise() writes the fold of a node to resolved/, and ResolvedLayer reads it back. It is typed as a LayerSource and sits in the fold's own input list. So a fold result appears as a fold input. The seed paths of resolve_dims and _fold_map read its folded axes and owner map, while source_for reads its own layer's rows through the same object.
DEP 6 needs axis to be layer-local on every source, and its sketch adds resolved_axis beside map_uri to get there. That hides the dual role instead of removing it.
WorkingRecord already keeps the two views as two objects: the Record over base and staged rows, and the StagedSource. But staged_only() returns a _Written, a third spelling of the staged layer's rows, because write_record's input is spelled as neither interface.
Specification¶
NodeCacheis renamedResolver, andstablebecomesfrozen, true when every source is frozen.Dimsis renamedCoords.Fold, indatarecord/layered/fold.py, hasschema, the folded axes, groups and owner map, and the readsattributes(),owners(attribute)andflags(entity_type=None).flagscovers the whole record whenentity_typeisNone.Fold.read(revision_id, con)readsresolved/and folds nothing.LayerSource.materialised(con, schema)returns the node'sFoldorNone._base_and_above(sources, con)scans all but the last source, deepest first, and the first hit is the base. The last source is never the base: a node resolves from its own layer, never its own cache, somaterialisenever reads its half-written cache back.- The base seeds each fold as depth 0, below every layer. So per key the fold still keeps the row of a layer above, and that layer's tombstones still remove a base key.
ResolvedLayeris deleted.sourcescontains only plainLayerSources,source_forreturns an own-rows source and never the base, andaxisis layer-local on every source.LayerData, inrecord.py, isschema,frozen, and the pairsaxes/axis,entity_types/entity_type,groups/groupandattributes/attribute. Its enumerators returnIterable[str], becauseResolver.attributesis ordered and a source's is a set.LayerSourceandResolverboth satisfy it structurally.layer_id,uriandall_attributesstay source-only.with_source,source_for,owners,flags,foldandrevision_idstay onResolver.- A source gains
entity_types,groupsandattributesas plain directory listings. Every source gets the schema its caller already has, andResolver.schemais a required field.declaredand the fallback toread_schema(con)go. Resolverrenamesentity_type_frametoentity_typeand routesattribute(name, kind)to private_relationand_outputs, which return an empty relation and notNone.group_framestays as the presentational readRecorduses.write_recordtakesLayerData | RecordLike. ARecordLikeproducer goes through_RecordLikeAsLayerDatainwrite.py.commitpassessources[-1]forNewChildand theResolverforDirectory._Written,staged_only()and the_staged_*frame builders are deleted, except_staged_groups.write.pyworks on DuckDB relations throughout.- An enumerator returns the empty set exactly when
entity_type_dim is None. A declaredStringaxis still lists its member files.
Rationale¶
What axis returns depends on the object you call it on, not on a resolved_ prefix or a mode flag. A file-backed source then needs no branch for the configuration it reads. The name Resolver describes the job, where NodeCache described an implementation detail, and Coords has groups and the broadcast algebra as well as axes.
frozen and materialised mean different things. frozen means that a source's rows cannot change under a reader, and gates the in-memory cache per connection. materialised is a filesystem fact set only by .materialise(), and gates the base that shortens the fold. Every materialised node is frozen, and almost no frozen node is materialised.
flags sits on Fold because scoping to a type needs both the owner map and the entity axis. LayerData contains only the members that mean the same for a layer and a fold. A method that only one of them has shows that the two are different types.
The writer needs exactly LayerData: the schema, plus per kind the keys an object has and the rows of each. The read never needs the enumerators, because the fold learns keys from the schema and the owner map. The writer must list what a layer contains, where the schema declares more. The Directory commit writes the resolved record, which is a fold, through the same interface.
The change reshapes the source list, and sources_to_read, resolve_coords and source_for move together. It also moves write_record's contract.
Implementation notes¶
It landed as three commits on refactor/staging-as-a-layer: cd2ce92, 006c159 and 7f244a9.
- The read-path split:
materialised, the base from the deepest materialised source,ResolvedLayerdeleted, and theResolverrename.Foldlives in its own module to break an import cycle withsources.py. - The
Foldobject. Folding a liveFoldneedsCoords' broadcast algebra, soFold.computeis aresolve.pyconcern andResolver.foldwrapsdimsand_map("inputs"). No cache moved. The liveFoldhasentity_types={}, because it is never a base. LayerDataand the write path.
test_resolved_reads_same_as_unresolved landed first, green before and after. It folds one node through its base and one from the root, and compares the owner map, the entity axis and every attribute. def axis in sources.py is only on _FileLayer, and sources.py changes only by deletion. On a materialised untyped node, entity_types() is the empty set on both the base Fold and the head's ParquetLayer.
Rejected ideas¶
resolved_axisandresolved_axesbesidemap_uri: a third folded-answer name that hides the dual role.- A stored
basefield onResolver: the base is discovered from the sources. flagson a map-onlyOwnerMap: it would need the entities relation handed in from outside.all_attributesonLayerData, with a foldedResolver.all_attributes: no caller reads it on a fold. A layer's share of the owner map needs the resolved axes to expand a NULL, so the split falls between raw rows and expand-and-aggregate.
Open issues¶
- Whether
Record's public narwhals interface joins the shared vocabulary, or only the internal rows interface does. - Whether a later
Storerefactor makes aResolver.owner_map(kind)and a foldedall_attributesworth having.