DEP 6: a member file contains values, and nothing else¶
| DEP | 6 |
|---|---|
| Title | a member file contains values, and nothing else |
| Author | Jonas Hörsch |
| Status | Superseded (2026-10-13, implemented 2026-09-04) |
| Created | 2026-09-02 |
| Version | v0.1.0a1 |
| Superseded-By | DEP 12 |
The design landed in the record format and the schema. Where they and this DEP differ, follow those pages. DEP 12 makes the entity-type axis an Enum property of entity. A record with categories and one without then share one entity-axis layout with no member files.
Abstract¶
A member file, dims/entity_type/<Type>.parquet, has a type's attribute values and nothing else. Where the schema declares no entity-type axis, a non-varying attribute over entity is a column of dims/entity.parquet, and no member files exist. In every record, deleted leaves the member file and stays on the entity axis alone.
Motivation¶
A schema may omit the entity-type axis, which makes the type labels data. Values over entity alone still land in one member file per label, so a label the schema never declared becomes a filename:
dims/entity.parquet entity, entity_type, deleted
dims/entity_type/thing.parquet entity, p_nom
dims/entity_type/other.parquet entity, p_nom
The entity_type column there matches no declaration. A special case in _validate_frame admits it and a fallback in _axis_columns types it as String. Every member file also has deleted, so remove writes a tombstone twice, to the axis and to the member table.
Dropping the column alone does not fix this. entity_types(), entity_type_frame and attributes_of all read the label off the resolved entity axis, so a column of NULLs would make entity_types() return {None}. Deriving the label from filenames needs a listing per layer per fold, which the entity axis exists to avoid.
Specification¶
- Where the schema declares no entity-type axis, which is
entity_type_dim is None, a non-varying attribute overentityis a column ofdims/entity.parquet.dims/entity_type/does not exist, andattributes_on("entity")returns those attributes. - In that case the entity axis a source supplies is
(entity, deleted, *attrs), with noentity_typecolumn. The materialisedresolved/dims/entity.parquethas none either. entity_types()returns the empty set.entity_type(ctype)raises.attributes_of(ctype)scopes to the whole axis.- A producer with no types supplies one
dims["entity"]frame with the member columns merged in, and noentity_typesmapping. - A member file has
entityand its type's columns, and nodeleted.removewrites one row, on the axis. - A schema that declares the axis keeps the per-type layout unchanged.
_validate_frame's special case and theStringfallbacks in_axis_columnsgo.ResolvedLayer'saxisandaxesbecome layer-local first, as DEP 8 argues. This DEP depends on that split.
Rationale¶
Both changes follow from one claim: a member file has values, and the entity axis has everything about a component's existence. Every column of the format then matches a declaration, and entity_type appears only where it has a dtype. A record with no kinds has one axis file, not one file per label an add used. Record.entity_types stops reporting filenames as a vocabulary.
_FileLayer, ParquetLayer and DirectorySource need no change, because they read whatever columns a file has. A file-backed source reads both configurations the same way. The PyPSA tool always has types, so it is unaffected in practice.
The design adds a second layout for one construct, keyed on one schema property. It replaces the entity_type special case and does not add to it. attributes_on("entity") changes meaning, so every caller needs an audit. Every record without a type axis is re-exported. A materialised cache written under the old layout needs a rebuild.
Implementation notes¶
tests/test_untyped.pyasserts thatrecord.entity_typesis empty and reads values offdims["entity"].- A written untyped layer has no
dims/entity_type/directory. - An
entity_typecolumn on the entity axis of an untyped record is refused. - A materialised node of an untyped record reads the same as an unmaterialised one. This test lands before either change.
- The resolved entity axis and the materialised file have no
entity_typecolumn. Checking onlyentity_types() == set()would pass with a column of NULLs.
Open issues¶
- Whether a declared entity-type axis typed
String, with noEnum, is the same case. The build keys onentity_type_dim is None, so such a schema keeps the per-type layout. - Whether
attributes_fornarrows at all without types. - Whether a materialised cache has a schema version or is invalidated when the schema changes. This is still unsettled.