Schema¶
Schema
¶
Bases: _Declared
One record's schema.
Attributes:
| Name | Type | Description |
|---|---|---|
version |
int
|
Bumped by any change to the declarations. A reader meeting a version it was not written for should refuse rather than guess. |
dimensions |
dict[str, Dimension]
|
Every declared axis, keyed by dim name. |
properties |
dict[str, dict[str, PropertySpec]]
|
Subject dim -> property name -> spec: the columns of
|
attributes |
dict[str, AttributeSpec]
|
Attribute -> spec, record-wide. One attribute is one spec and
one |
partial |
Names | None
|
Which dims a layer may patch value by value. |
meta |
dict[str, Any]
|
A framework's own top-level data - network attributes, CRS, free-form metadata. Stored and never interpreted, since none of it describes the dimensioned data. |
Notes
digest
cached
property
¶
The sha256 hex of the schema's compact JSON, as its manifest writes it.
Equal for equal schemas and stable across processes, so it keys a
schema wherever the schema itself cannot. Computed once: the nested
mappings must not be mutated; derive a changed schema with
model_copy(update=...).
dims
property
¶
Every declared dim but the multi-dims, in declaration order.
Notes
multi_dims
property
¶
Every declared multi-dim, in declaration order, each carried in a row as one struct column named for it.
Notes
one_of_dims
property
¶
Every dim with one_of, in declaration order: those whose axis is the schema's list.
Notes
broadcast_dims
property
¶
The coordinates a NULL broadcasts over: every declared dim, the multi-dims last.
A NULL here means "every value of this dim", which the fold expands against the axis; a NULL struct means every tuple of the resolved multi-dim.
What the varies/broadcast structs have a field per, and what
expand_dims joins.
Notes
partial_dims
property
¶
The fold key's coordinates, in declaration order.
The dims a layer may patch per value (partial). The fold's
key is one fixed tuple over all attributes, so it carries every axis any layer may patch by value or
by row, not only those some currently declared attribute varies over. An
attribute not owned per one of them writes NULL there, the "NULL means
all values" rule - which also lets a schema declare an axis before any
attribute uses it.
long_columns
property
¶
The long schema's full column set.
The map's column set, which is uniform across attributes because the
map is one relation over all of them. An individual file carries only
its own attribute's columns (long_columns_for), and union_by_name
supplies NULL for the rest when the fold unions them here.
Notes
owner_map_key
property
¶
Owner-map key columns, compared NULL-safely when folding.
partial_dims, plus attribute: a layer may patch one value along a
partial dim without restating every other's.
A coordinate an attribute's own file does not carry reads as NULL, which is what makes the key one fixed tuple over attributes whose columns differ.
owner_map_columns
property
¶
The owner map's full column set.
Notes
model_copy
¶
A copy as pydantic makes it, without the digest of the original.
Source code in src/datarecord/schema/schema.py
one_of_properties
¶
Map each property that refers to a dim, as (subject, name), to that dim.
Notes
Source code in src/datarecord/schema/schema.py
over
¶
The coordinates of dim, each to the dim it draws on, in field order; empty for a dim that is no multi-dim.
Raises:
| Type | Description |
|---|---|
KeyError
|
If the schema declares no |
Notes
Source code in src/datarecord/schema/schema.py
coordinates_of
¶
The coordinate columns one attribute's rows carry, multi-dims first.
A multi-dim in dims is carried as one struct column named for it, so
dims={"connection", "snapshot"} gives ("connection", "snapshot").
Per attribute rather than schema-wide: one file per attribute means one column set per attribute, and an all-NULL column of a dim the attribute is not addressed by would claim a coordinate the value has none of.
Raises:
| Type | Description |
|---|---|
KeyError
|
If the schema does not declare |
Source code in src/datarecord/schema/schema.py
duck_type
¶
duck_type(
spec: Dimension | AttributeSpec | PropertySpec,
) -> DuckDBPyType
Return the DuckDB type of a declared dim, attribute or property.
A dim with one_of has the ENUM of them, a multi-dim the struct of
a field per coordinate, typed as the dim it draws on. A property that
refers to a dim has the type of that dim.
Notes
Source code in src/datarecord/schema/schema.py
check_multi_dim_columns
¶
check_multi_dim_columns(
multi_dims: Iterable[str],
rel: DuckDBPyRelation,
*,
carrier: str,
) -> None
Reject each of multi_dims carried other than as one struct column of exactly its coordinates.
Judged by the relation's columns and types, so no row is read. A carrier without a multi-dim's column passes.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
carrier
|
str
|
What carries the columns, named in the message. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
On a coordinate as a column of its own, or a multi-dim column not a struct of exactly the coordinates, naming the struct to pass instead. |
Notes
Source code in src/datarecord/schema/schema.py
refuse_partial_tuples
¶
Reject a struct of any of multi_dims in rel with some fields NULL and not all.
Reads the rows, unlike check_multi_dim_columns. A carrier without a
multi-dim's column passes.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
carrier
|
str
|
What carries the columns, named in the message. |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
Naming |
Notes
Source code in src/datarecord/schema/schema.py
long_columns_for
¶
Return one attribute's full long column set, in order.
An attribute carries the coordinates its dims name and no others, so a
per-snapshot weighting has no column for any other dim.
Raises:
| Type | Description |
|---|---|
KeyError
|
If the schema does not declare |
Notes
Source code in src/datarecord/schema/schema.py
categorizing
¶
Return the categorizing properties of dim, the ones default_by keys on.
A categorizing property is a property of dim that refers to a dim with
one_of. Only a dim in partial has any.
Notes
Source code in src/datarecord/schema/schema.py
require_categorizing
¶
Refuse a key that is not a categorizing property of subject.
Raises:
| Type | Description |
|---|---|
ValueError
|
Naming the keys and |
Notes
Source code in src/datarecord/schema/schema.py
require_one_of
¶
Refuse a label its dim does not list.
data maps column names to values. A name is checked where it is
subject and subject has one_of, or where it is a property of
subject that refers to a dim with one_of. Other names
and NULL values are skipped.
Raises:
| Type | Description |
|---|---|
ValueError
|
Naming |
Notes
Source code in src/datarecord/schema/schema.py
default_for
¶
The default of attribute or subject.name property name for a label of categories.
categories are keyed by each categorizing property's own name. The
first default_by key listing the label's category decides, else
default; a category left out matches none.
Raises:
| Type | Description |
|---|---|
KeyError
|
As |
Notes
Source code in src/datarecord/schema/schema.py
addresses
¶
Whether attribute reaches a label of dim at all.
True where its dims name dim, or a multi-dim one of whose
coordinates draws on dim.
Notes
Source code in src/datarecord/schema/schema.py
resolve
¶
resolve(
name: str,
) -> (
AttributeSpec
| tuple[str, PropertySpec]
| tuple[str, Dimension]
)
What name addresses: an attribute, or a property or coordinate as subject.name.
Read off the name alone: a plain name is never a property.
Returns:
| Type | Description |
|---|---|
AttributeSpec or tuple of str and PropertySpec or Dimension
|
The attribute's spec, a property's subject and spec, or a coordinate's multi-dim and the dim the coordinate draws on. |
Raises:
| Type | Description |
|---|---|
KeyError
|
If |
Notes
Source code in src/datarecord/schema/schema.py
coordinate_over
¶
The one coordinate of the multi-dim multi drawing on dim, if any.
Raises:
| Type | Description |
|---|---|
ValueError
|
If several coordinates of |
Source code in src/datarecord/schema/schema.py
owned_per
¶
Which dims a layer owns attribute per.
Derived rather than declared: AttributeSpec.dims says which axes the
attribute may vary over, partial_dims the fold key (the partial
dims), and ownership is their intersection. A
dim in dims but not the fold key - a non-partial value axis like
timestep - is owned whole, so a patch to one of its values restates the
attribute's entire extent along it (_owned_whole).
Source code in src/datarecord/schema/schema.py
multi_dims_of
¶
Which declared multi-dims address attribute, in declaration order.
Notes
Source code in src/datarecord/schema/schema.py
column_type
¶
Return the declared DuckDB type of column, or None if the schema declares none.
Covers the structural columns the format fixes, the declared dims,
the properties of subject - columns of its
axis file, read first - and the owner map's two flag structs, whose fields
follow the schema's dims.
No dim is structural: each is declared, and typed from that
declaration. So the ENUM of a dim with one_of pins them
everywhere the column is built. An attribute is no column but a long row's value,
typed by value_type.
A schema declaring no dims at all is "no manifest yet" rather
than a record to fold, and DuckDB has no empty struct - so the flag
columns are undeclared there, and a caller building an empty relation
falls back to VARCHAR for a map that will never hold a row.
Source code in src/datarecord/schema/schema.py
declared
¶
declared(
column: str,
subject: str | None = None,
attribute: str | None = None,
) -> tuple[str, DuckDBPyType, str] | None
Look up the declaration of column.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
subject
|
str | None
|
The dim whose file holds the column. A column named like
one of its properties is that property, named |
None
|
attribute
|
str | None
|
The attribute whose long rows hold the column. Its |
None
|
Returns:
| Type | Description |
|---|---|
tuple of str, DuckDBPyType and str, or None
|
The declared name, the DuckDB type, and the dtype as an error
message shows it; None where the schema declares no type for
|
Notes
Source code in src/datarecord/schema/schema.py
require_takes
¶
require_takes(
rel: DuckDBPyRelation,
what: str,
subject: str | None = None,
attribute: str | None = None,
) -> DuckDBPyRelation
Check that each column of rel has a type that fits its declaration, and return rel, with its all-NULL columns cast to their declared types.
A column type fits where takes says so. A column
that fails this but holds only NULLs passes and is cast to its declared
type, because dataframe libraries guess the type of an empty or
all-None column. Only such columns have their rows read. subject and
attribute are as in declared.
Raises:
| Type | Description |
|---|---|
ValueError
|
Naming |
Source code in src/datarecord/schema/schema.py
value_type
¶
Return the DuckDB type of the value column of attribute.
One attribute is one attributes/<attr>.parquet with one value
column, so the dtype is the attribute's alone.
Notes
Source code in src/datarecord/schema/schema.py
compatible_with
¶
compatible_with(other: Schema) -> list[str]
Why layers written under other would not read under self.
Empty when the change is compatible: old layers stay readable and only
version moves. The compatible changes are those where NULL already
means what the new schema needs it to mean, so the broadcast rule absorbs
them without touching a row.
Returns:
| Type | Description |
|---|---|
list of str
|
One reason per incompatibility, empty if there are none. |
Notes
Source code in src/datarecord/schema/schema.py
1148 1149 1150 1151 1152 1153 1154 1155 1156 1157 1158 1159 1160 1161 1162 1163 1164 1165 1166 1167 1168 1169 1170 1171 1172 1173 1174 1175 1176 1177 1178 1179 1180 1181 1182 1183 1184 1185 1186 1187 1188 1189 1190 1191 1192 1193 1194 1195 1196 1197 1198 1199 1200 1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 | |
Dimension
¶
Bases: _Declared
One axis attribute data may vary over: its shape, not its data.
Not which dims an attribute varies over (AttributeSpec.dims), nor the
patch granularity (Schema.partial), nor order - an axis is ordered by its
file's row order, undeclared.
A dim declaring over is a multi-dim: its labels are tuples of other dims'
labels, carried in a row as one struct column named for it, a field per
coordinate (Schema.multi_dim_type). Coordinates rather than dims because
two of them may draw on one dim: a corridor between two buses is
(from, to).
Attributes:
| Name | Type | Description |
|---|---|---|
over |
dict[str, str] | None
|
Coordinate name -> the dim it draws its labels from, or None for a dim
that is no multi-dim. A list is sugar for the dict with identical keys
and values. Declared without |
dtype |
DimensionDtype | None
|
The axis labels' type, a mathspec |
one_of |
tuple[str, ...] | None
|
The dim's labels, in order, or None for labels a layer writes. A
dim with |
unit |
str | None
|
What this axis's labels measure, if anything - |
description |
str | None
|
What the axis is, in prose. Never interpreted. |
AttributeSpec
¶
Bases: _ValueSpec
What shape one attribute's data may take.
Attributes:
| Name | Type | Description |
|---|---|---|
dtype |
ParameterDtype
|
The value column's type, a mathspec |
dims |
Names
|
Dims this attribute may vary over; a subset
of those declared. Whatever they are, its values are long rows of
|
default |
Any | None
|
The value a coordinate no row covers takes. |
default_by |
dict[str, dict[str, Any]]
|
A categorizing property, as |
breakpoints |
bool
|
Whether it may carry a piecewise-linear curve. |
unit |
str | None
|
What the values measure - |
description |
str | None
|
What the attribute is, in prose. Never interpreted. |
Notes
PropertySpec
¶
Bases: _ValueSpec
One value per label of a dim, describing it.
A property is keyed by its subject (Schema.properties) and stored as a
column of dims/{dim}.parquet.
Its name is its subject's alone, so it is addressed as subject.name
(Schema.resolve).
A property declares exactly one of dtype and one_of. A property of a
dim in partial that refers to a dim with
one_of categorizes its subject (Schema.categorizing).
Attributes:
| Name | Type | Description |
|---|---|---|
dtype |
ParameterDtype | None
|
The column's type, a mathspec |
one_of |
str | None
|
The dim the property refers to. Its values have that dim's type, and a stored value that is not NULL must be a live label of the dim. |
default |
Any | None
|
The value a label no row sets takes. |
default_by |
dict[str, dict[str, Any]]
|
A categorizing property of a dim the subject reaches, as
|
unit |
str | None
|
What the values measure; |
description |
str | None
|
What the property is, in prose. Never interpreted. |