DEP 5: one Record, over a NodeCache¶
| DEP | 5 |
|---|---|
| Title | one Record, over a NodeCache |
| Author | Jonas Hörsch |
| Status | Superseded (2026-10-21, implemented 2026-09-01) |
| Requires | DEP 4 |
| Created | 2026-09-01 |
| Version | v0.1.0a1 |
| Superseded-By | DEP 19 (its Record over a NodeCache) |
The design landed in the fold and the RecordLike protocol. Where they and this DEP differ, follow those pages.
Abstract¶
DirectoryRecord is deleted, because a plain directory is a DirectorySource and folds like any layer. LayeredRecord becomes the one class, Record, a narwhals view over a NodeCache. The protocol becomes RecordLike, and WorkingRecord subclasses Record.
Motivation¶
DEP 4 made a staging area a source of one fold. A plain directory is a source too. The read-path page lists four ways the directory and layered implementations differ, but on one directory all four give the same answer:
| Compared | Result |
|---|---|
dims, entity_types, groups, attributes, outputs |
identical |
flags, every declared type |
identical |
| member order, every declared type | identical |
| resolved rows per attribute | identical |
flags, first call |
6.6 ms scanned, 2.9 ms folded |
Over one source the fold is a scan. It computes flags in its ownership GROUP BY, where DirectoryRecord scans twice, and order_key over one source is file order. partial is inert over one layer. The table described an implementation that predated the fold reading anything but a tree.
Specification¶
NodeCachedoes not grow. It stays the DuckDB view over a list of sources.Recordis one frozen dataclass that has aNodeCacheand adapts its relations to narwhals frames.Record.over(*sources, con=...)builds one, andRecord.at(uri)reads a plain directory as its one layer.Recordis the class andRecordLikethe protocol. The 17 annotation andisinstancesites in reader positions takeRecordLike.WorkingRecordsubclassesRecord. ItsNodeCacheis its base's with aStagedSourceappended. It inherits every read member unchanged exceptoutputs, because results do not overlay.set,add,removeandcommitstay onWorkingRecordalone. ItsNodeCacheis fixed at construction: an edit changes the tables the last source reads, never which sources there are.Recordcaches its key sets only whereNodeCache.stableis true, so a staged new attribute stays visible.Record.atreads a standalone directory's ownmanifest.json, andNodeCachehas it asdeclared. AWorkingRecordover such a base reads under that schema.directory.pyand itsflagsaggregate are deleted, andLayeredRecordis renamed. There is no format change.
Rationale¶
tools/ builds against narwhals frames, so keeping NodeCache and Record apart keeps that interface. A framework object still satisfies the protocol structurally, so the protocol stays and takes the longer name, RecordLike, for "anything shaped like a record".
Two arguments for keeping DirectoryRecord fail. A one-source fold leaves no owner map on the connection, because a DirectorySource's id derives from its URI and nothing is created: five tables before, five after. And a directory copied out of any tree, with no revisions row, folds identically, because the fold needs a layer layout and not a node.
The change withdraws a design property: a directory is now read with resolution. If reading it as a one-layer record is wrong, it is wrong here. DirectoryRecord.flags memoised per type, so the fold is faster on the first call and the cache on the tenth.
Implementation notes¶
- The cached key sets, not
frozen=True, were the real obstacle.NodeCache.stableandRecord._stable_cachefix it without an override inWorkingRecord. overtakesconexplicitly, because aLayerSourcehas none.- A
LayerSource.own_schemamember was tried and dropped: a fold has at most one directory, always at the root. - The
bothfixture compares one directory read as a directory with the same rows read as a revision's layer. A foreign directory with no tree must read through the fold.test_tools.pymust pass unchanged. A test lists the membersWorkingRecorddefines beyondRecord, withoutputsthe one allowed read member. WorkingRecordalready has its base'sNodeCache, resolved once in__init__.NewChild()asks therevisionstable whether there is a node to branch from.
Open issues¶
- Whether
Revisionstill needsrecord, one construction away fromnode_cache. - Whether
WorkingRecord's base cache becomes a slice of its own, minus the last source.